NVME Seems Capped Inside Ubuntu VM

V

vexter0944

New Member
Apr 14, 2023
6
0
1
Hello,

Been running Proxmox on a new setup for a couple of weeks now. Screen shot below for version etc - if you need more stats on Proxmox - let me know.

I have a I5-10400 on a https://www.amazon.com/gp/product/B08WC8QDTC?th=1 (MPG Z590 Gaming Force Gaming Motherboard) and 64gb of ram. I have 2 nvm's installed in this server - 1 is for the proxmox OS and 1 is for all the VMs to use for disks. I have a Ubuntu 22.04 server running Docker on this Proxmox server and it's working fine - but the file transfer seems capped at about 400-500 Megabytes/sec. I forgot to mention this proxmox server has a 10gbe card that is connected to a Brocade switch. Using Iperf, I can test from my Truenas server (also on 10gb) and hit the expected throughput from 800-900 Megabytes/Sec. So this is not a network issue so far as I can tell.

Should I passthrough the nvme or am I missing some settings/drivers? I'm puzzled. Any help/ideas? Happy to run any tests needed etc I'm happy to try and run them.

1681504572765.png
 
For others who come along later they'll find the following information useful, if you provide it:
The hardware layout for the virtual machine (you can screenshot this)
The disks layout for the node (you can screenshot this)
The storage layout for the datacentre (you can screenshot this)

Knowledge is power ;) It'll provide all the interesting bits regarding the disks and the layout.
What brand/type are the NVMe drives?
 
_gabriel said:
vDisk Cache is disabled by default for safety in case of power outage.
You need to set to "Write back" to except Windows bare metal performance.
Click to expand...
Thanks! That helped some - still capped - but better - see the screenshot. I'm not trying to force as correlation, but you can see the disk speed and network speed are almost identical....hence why I feel like trasferring from my truenas (10gb) to this VM is capped by disk speed when iperf can hit 8-9gb between the two no problem. Thanks for the help - if I can get you more info etc, let me know.

1681576618835.png
 
One last note - which could be helpful - I did a mount to the nfs4 share on my truenas from proxmox directly and copied the same file - so not from the VM - and it shows similar. the test at around 11:30 is from the earlier post where I copied the same file to the guest vm. The spike at the end at about 12:06 is from me copying the file from a nfs share ssh'd into the proxmox host. Quite similar results....maybe something at the proxmox os level more than the VM?

1681578620891.png
 
Yes, the write amplification from TrueNAS can slow things down majorly. I've been wrestling with a system on my end, but I have been aiming for data safety instead of speed, which ends up at the opposite end of what you are trying to achieve.
I certainly got the data safety...

As a suggestion, I see Hard Disk (SCSI0) is set as a HDD, you could always set that to SSD and tick discard; this allows the client trim to function on hosts where thin provisioning is enabled, gaining some space back on the host; the disk does need a unique ID for this to be able to be set as an automatic task in TrueNAS, but it can also be done manually via the shell in TrueNAS. It might not be relevant for your use case, tho, but is worth knowing about if you create a storage group using ZFS, or LVM-Thin, on the host itself in the future.

Given you're setting up everything for speed, is this system backed by a robust UPS? And, if so, is proxmox itself set up to shut down automatically when the UPS kicks in and is running out of power? I bring that up with all seriousness, as ZFS can die quite horribly if there are too many bad shutdowns; I know, I've been dealing with it this end.

Now, here are some other things you can use for monitoring:
Install iotop onto the proxmox host, and set it running; it's a cli utility, so ssh into the box and observe it from a separate window. You'll be able to see your disk read/write at semi-useful update speeds.
In the TrueNAS shell, you can also run zpool iostat -v. This is a nice command: https://openzfs.github.io/openzfs-docs/man/8/zpool-iostat.8.html

I'm sure those will help you a great deal in getting a better idea of where all the performance bottlenecks are, as well as show you the write amplification.

Have fun :)
 
_gabriel said:
you need to isolate disk using fio to benchmark your disk.
storage speed of truenas can be the bottleneck too.
Click to expand...
@_gabriel - I looked around for some fio commands to run - there's a lot of them to pick and choose from, do you have a link or a command I could run to return a test for this thread?
 
Nuke Bloodaxe said:
Yes, the write amplification from TrueNAS can slow things down majorly. I've been wrestling with a system on my end, but I have been aiming for data safety instead of speed, which ends up at the opposite end of what you are trying to achieve.
I certainly got the data safety...

As a suggestion, I see Hard Disk (SCSI0) is set as a HDD, you could always set that to SSD and tick discard; this allows the client trim to function on hosts where thin provisioning is enabled, gaining some space back on the host; the disk does need a unique ID for this to be able to be set as an automatic task in TrueNAS, but it can also be done manually via the shell in TrueNAS. It might not be relevant for your use case, tho, but is worth knowing about if you create a storage group using ZFS, or LVM-Thin, on the host itself in the future.

Given you're setting up everything for speed, is this system backed by a robust UPS? And, if so, is proxmox itself set up to shut down automatically when the UPS kicks in and is running out of power? I bring that up with all seriousness, as ZFS can die quite horribly if there are too many bad shutdowns; I know, I've been dealing with it this end.

Now, here are some other things you can use for monitoring:
Install iotop onto the proxmox host, and set it running; it's a cli utility, so ssh into the box and observe it from a separate window. You'll be able to see your disk read/write at semi-useful update speeds.
In the TrueNAS shell, you can also run zpool iostat -v. This is a nice command: https://openzfs.github.io/openzfs-docs/man/8/zpool-iostat.8.html

I'm sure those will help you a great deal in getting a better idea of where all the performance bottlenecks are, as well as show you the write amplification.

Have fun :)
Click to expand...
thanks @Nuke Bloodaxe for the detailed response! I'm not sure if I made it clear or not, my Truenas server is separate from this Proxmox server entirely in my homelab - just stating to frameout more info is all. I am certainly open to trying SSD as the disk type - so might give that a go today. I do have a proper UPS but do not have it set to turn off Proxmox at this time (new box have gotten around to setting that up yet). I'll try the iotop and zpool iostat - those sound like a good starting place.

Aprpeciate your time and input!
 
You must log in or register to reply here.
Top Bottom