Best (most performant) way of sharing NAS storage to Windows 11 VM

[Clint84]

BACKGROUND:
I have a Windows 11 VM that runs my cameras VMS (Video Management Software). My server is in a cluster with 3 other identical machines running other production VMs and they are using their internal NVME SSDs in a CEPH cluster for High Availability. I have a separate server running TrueNAS Scale (baremetal) and the Proxmox backup Server (VM) for daily backups. The server does not have internal space for one or more 3.5" drives to provide the necessary amount of storage so I opted for utilizing some of my NAS pool. The NAS pool is otherwise used for bulk/long term storage of office files and for VM Backups that run overnight.

My current configuration has the OS drive of my VM using my CEPH storage so that the VM can operate in HA mode. I created a iSCSI Block Share target on the NAS. I connected the Proxmox Cluster to the iSCSI share, and then added a virtual SCSI disk to the VM for 6TB. If my server fails and the VM moves, it can reconnect to the iSCSI share without issue since the share is available across all servers in the cluster. The NAS and the Proxmox cluster are networked via a 10Gb switch with SFP+ DAC cables. Total bandwidth of all cameras to the recording VMS is only about 150 Mb/s.

ISSUES:
Sometimes when reviewing the recordings on the VMS in playback (which is done on a client software on a different workstation due to needing gpu/igpu power to decode the high resolution streams), it becomes very slow or unresponsive and takes a while for the video streams to load. Accessing the VM direcly via Proxmox web console or Remote Desktop shows it operates smoothly and quickly within the operating system itself, indicating that the VM itself is not bogging down (although due to lack of graphics performance cannot test viewing the playback off the shared drive). A reboot of the VM solves the issue every time. It seems like it slowly builds up some sort of resistance in accessing the data over time. I would just do scheduled reboots, but it takes several minutes for all the services to come back online and be able to view the video streams again; and we have 24hr monitoring of cameras so they can't keep going down all the time. The issue does not center around the times when backups are being written overnight, most access to recordings is done during daytime hours when the NAS storage is relatively unaccessed except for the recording for the VMS.

FIXES?
Would using an NFS share to the Proxmox cluster be a better option/more performant over the iSCSI?
Would pointing to the iSCSI target from within the Win11 VM instead of Proxmox be better?
Would adding an extra disk shelf to the NAS with an external HBA and creating a new pool specifically for the VMS to record to, then share it in one of the aforementioned ways work best?

I tried searching online for comparisons between whether iSCSI shares are better or worse than NFS and usually find people recommending iSCSI for performance. I couldn't really find any information on comparing peformance/stability of an iSCSI direct to the Windows VM vs routing through the Proxmox host and presenting as a drive to the VM.

 

[bbgeek17]

Would using an NFS share to the Proxmox cluster be a better option/more performant over the iSCSI?

Under normal circumstances, given your numbers, it is likely that you would not see perceivable difference. You may even notice lower performance, as you would need to add another layer (qcow or raw file).

Would pointing to the iSCSI target from within the Win11 VM instead of Proxmox be better?

There is no scientific evidence that direct Windows 11 iSCSI connection would function more stable than Hypervisor one in your case. That said, there may be specific variables in your environment (hypervisor CPU, VM CPU/Cores, NUMA assignments, network card type, PCI location, etc) which can affect your experience one way or another.

Would adding an extra disk shelf to the NAS with an external HBA and creating a new pool specifically for the VMS to record to, then share it in one of the aforementioned ways work best?

Again, since you have no numerical evidence of "resistance build up in storage" , there is no way to advise on positive or improved outcome with above change.

If this was my environment, I would start with using direct CIFS connection from Windows to NAS.

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

You havent really established this is a "storage" issue at all. your problem description,

Sometimes when reviewing the recordings on the VMS in playback (which is done on a client software on a different workstation due to needing gpu/igpu power to decode the high resolution streams), it becomes very slow or unresponsive and takes a while for the video streams to load.

suggests the issue is with how your software interacts with the decoding and not so much with the storage; it COULD be the storage but you havent posted any diagnostic data to suggest it, namely-
what are your network statistics at the time?
what does the io pressure look like at the hypervisor level? (you can even just post screenshots for these two items)
from inside the guest- any data to suggest network issues?
from the storage- any data to suggest disk issues or share side issues?

 

[Clint84]

You havent really established this is a "storage" issue at all. your problem description,


suggests the issue is with how your software interacts with the decoding and not so much with the storage; it COULD be the storage but you havent posted any diagnostic data to suggest it, namely-
what are your network statistics at the time?
what does the io pressure look like at the hypervisor level? (you can even just post screenshots for these two items)
from inside the guest- any data to suggest network issues?
from the storage- any data to suggest disk issues or share side issues?

True, I haven't been able to confirm it is in fact storage, that just seem to be the most likely factor that made sense since the navigating VMs operating system was smooth and responsive and only when retrieving data stored on the remote drive did I have any issues. I mainly wanted to identify if my configuration was less than ideal and possibly the cause. It does sound like I need to delve deeper into the network metrics and maybe the recording/client software as well. Being a networked drive obviously lends to network issues as well, although I overestimated the traffic for the cameras by doing some napkin math based on the bitrate of my cameras instead of looking at the actual traffic metrics. Turns out it is quite a bit less network traffic on the cameras. The last time the issue occured was several days ago and I didn't have time available to troubleshoot, I had to reboot and retrieve footage quickly. I only just had time today to start attempting to find a solution. Was hoping I made a glaring faux paux with my storage configuration that could easily be remedied. I will attach the current information for while it is running "normal". Usually my issue crops up within a week or two so I will be able to create a reference between when it is normal and slow. I will add in the data during a problem time to see if a conclusion can be drawn. I will also try to simply restart the software service for the recording software instead of the entire VM to rule out the software slowing it down.

Again, since you have no numerical evidence of "resistance build up in storage" , there is no way to advise on positive or improved outcome with above change.

Forgive my verbage, I am more of a hardware technician and I do encounter times where a circuit slowly builds a resistance leading to a failure or unexpected operation, that is then cured by power cycling the hardware and discharging the energy in the components. I drew from this analogy to try and express how it seemed to me, although I fully understand that the storage is not building a resistance. I guess it would translate more to the VM seems to, over time, have more difficulty retrieving data from the storage.

Usually when I am reviewing footage I can drag the timeline around to skip over minutes and it only takes a second or two to resume playback at the newly chosen time. The process is usually referred to as "scrubbing through footage". It allows me to spot changes in the view quickly to narrow down when an event occurred since the recordings are continuous and not event driven. When the problem is present, it can take 30+ seconds for it to load a chosen time and everytime I move the timeline to jump ahead by even a few minutes of video, it will take 30+ seconds to resume, turning a 10 minute task into a 30 minute or more task.

 

[alexskysilk]

graphs dont show any likelihood of networking issues/starvation. if your storage isnt reporting any issues, thats not likely to be your problem.