Proxmox VM Slow Disk Writes

[Vedeyn]

Good Afternoon,

I recently built a new server and installed Proxmox on it. Unfortunately I noticed that I am seeing some pretty low disk write speeds inside of VMs running on a 512GB Intel 660p NVMe SSD. Proxmox itself is not installed on this SSD, it is only used for VM OS disks. I've added this SSD to Proxmox as type LVM.

Proxmox Version:
pve-manager/6.1-3/37248ce6 (running kernel: 5.3.10-1-pve)


Speeds outside of a VM look great:
root@proxmox1:~# sync; dd if=/dev/zero of=/mnt/tmp/tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.447218 s, 2.4 GB/s

Speeds inside of a VM are not however:
root@nas:~# sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.24745 s, 130 MB/s

This disk is added to the VM using virtio scsi, discard and SSD emulation enabled. This same VM also has a disk residing on a RAID10 HDD array:
root@nas:~# sync; dd if=/dev/zero of=/home/user/NAS/tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.69699 s, 633 MB/s

Speeds here are much better. This also uses virtio scsi but no discard or SSD emulation.


Any help would be appreciated.

Thanks,
Vedeyn

 

[lps90]

Same happens with containers.
Slow disc write speeds

 

[Vladimir Bulgaru]

Hey!

This might help you: https://forum.proxmox.com/threads/unusually-slow-io-on-a-pcie-ssd.54833/

 

[lps90]

Hey!

This might help you: https://forum.proxmox.com/threads/unusually-slow-io-on-a-pcie-ssd.54833/

Any solution for Lxc containers?

 

[Vedeyn]

Hey!

This might help you: https://forum.proxmox.com/threads/unusually-slow-io-on-a-pcie-ssd.54833/

Thanks, I'll look into some potential driver issues. I'm running all Ubuntu VMs so not quite the same case but maybe worth looking into.

I did some additional testing and found something that is rather odd. I added an LV to my SSD, formatted as EXT4, mounted it in proxmox, and then added that directory to the storage pool. I added a disk to a VM using that storage pool and the speeds are fantastic, well over 1GB/s. Anyone have any idea why that might be? Another curious thing to note is that I do not see any of this behavior on my 2nd proxmox node that has only a sata SSD in it so it just seems to be the Intel 660p that is having the issue. I have no other NVMe SSDs I can troubleshoot with unfortunately.

 

[Vedeyn]

Another update:

I created and added an LVM-Thin pool to proxmox, added a disk to my Ubuntu VM using the LVM-Thin storage, and tested using the same settings as before.. and I got great speeds (over 1GB/s). So this seems to be isolated to disks on the Intel 660p using LVM.

Anyone have thoughts on why that might be?