Any tips/suggestions to improve disk performance/response time? (or general performance)

[zenowl77]

mostly it is really good, but it is a little slow here and there especially under load, that is where it becomes the most noticeable very quickly, access time tends to get slow and laggy.

benchmarks are a bit degraded from expected speeds, for example my 10TB HGST Enterprise drive seems to get about 40-50% throughput (100-120mb/s at the most where as host can see 220-250) with access times averaging up to 30-70ms inside the VM and both NVME and HDD can end up at 500-600ms response time under load.

NVME is LVM, 10TB is XFS (tried ZFS but it was much slower, used a lot of ram and seemed overkill on a single disk), both set to no cache, IO thread, Async IO: Threads for VM, using VirtIO SCSI controller, only one VM and 2 LXCs running. Guest is windows 10 Pro 22H2 Workstation

i am mostly playing around with AI text/image generation, video editing/conversion, gaming here and there and other fairly demanding things in the main VM so high access time becomes pretty noticeable rather quickly.

any tips, suggestions or thoughts on this would be greatly appreciated. even little things to give a slight boost/edge in any way. (even possibly any cheap hardware i can add)

hugepages enabled, vGPU enabled,

hardware:
CPU: i7-7820X
RAM: 96GB DDR4
mobo: Gigabyte X299 UD4
NVME: WD Blue SN550 1TB
HDD: 2x HGST Ultrastar He10 (huh721010ale604) 10TB 7200rpm 1x 12TB version
GPU (1): GTX 1060 3GB 
GPU (2): intel Arc A310 4GB
GPU (3): Tesla P4 8GB

 

[waltar]

The first problem might be that you talked to less about your configurations which you done or what you are running exactly to understand to give you a response to.

 

[dumitru_mardari]

how do you measure your speed? you'll generally find different speeds from tool to tool. fio will give you something else than crystaldiskmark and that's where you might be surprised getting a 50% difference from bare metal speeds. try something like kdiskmark on linux.

 

[zenowl77]

The first problem might be that you talked to less about your configurations which you done or what you are running exactly to understand to give you a response to.

what other information would you like? this is mostly just a general performance loss that seems to happen in windows VMs where it feels laggy and the more stress it is under the worse it becomes.

how do you measure your speed? you'll generally find different speeds from tool to tool. fio will give you something else than crystaldiskmark and that's where you might be surprised getting a 50% difference from bare metal speeds. try something like kdiskmark on linux.

i have used Crystaldiskmark, ATTO disk benchmark, Anvil benchmark and a few others, but mostly as a way to test what is occuring and give some kind of qunatification in real data to what i am feeling happening within my use of the VM. stress to the VM leads to noticable performance losses and degradation of the usage experience very quickly. even if it is the only VM running, tasks just feel extremely laggy and sluggish very quickly if it is not doing one thing at a time and the thing is not too intense. (which it should have all the hardware it needs to handle a few tasks at once and handle stressful tasks at least decently enough)

 

[dumitru_mardari]

Cannot relate to your experiences. I have 1 tb nvme with 5000MB per sec read and write. Even when i stress test it, everything works fine. I did passthrough the nvme as pcie device, so the drive is completely dedicated to one VM.

 

[zenowl77]

Cannot relate to your experiences. I have 1 tb nvme with 5000MB per sec read and write. Even when i stress test it, everything works fine. I did passthrough the nvme as pcie device, so the drive is completely dedicated to one VM.

yeah i think that would fix the issue i am experiencing if i passed the whole device, but it is also the device proxmox boots from, so cannot do that in this case

i was reading the other day i guess this issue is created with the scsi driver when it is VM >scsi driver > host > nvme driver and the latency goes sky high

i am not sure what to do there just yet at least to eliminate that without passthrough, will have to experiment and test different configurations

 

[dumitru_mardari]

You can buy an 120gb ssd and install proxmox there. Proxmox by itself doesnt need many resources to run and 120gb ssd are like 15 euros rn. But also, you dont absolutely need to passthrough whole nvme. You can use as well nvme as virtual storage with virtio scsi single controller and reach 5000MB/sec reads as well. At least i was able to reach this performance with my nvme. Therefore there is little doubt from my side that the controller type or proxmox configuration can be at fault.