[SOLVED] 10GbE IOPS different on host vs VM

jdancer

Well-Known Member
May 26, 2019
153
29
48
54
I have a standalone Dell R620 ZFS host with 10GbE networking.

I get the full network IOPS when doing a wget of a file on the R620.

However, when I run the same wget inside a VM, I do NOT get the same network IOPS. Maybe I get about 60-75% network IOPS.

Anything on the host and/or VM settings I can change to get full network IOPS?

I have done the following optimizations:

Set write cache enable (WCE) to 1 on SAS drives
Set VM cache to none
Use VirtIO-single SCSI controller and enable IO thread and discard option
Set VM CPU type to 'host'
Set Linux VMs IO scheduler to none/noop (I do NOT have any Windows VMs, 100% Linux VMs)

EDIT: Fixed the issue by setting Multiqueue value to VM number of vCPUs.
 
Last edited:
Sounds good.

You could try to enable multiqueue for your virtio NIC. You are using virtio and not the slow E1000 NIC right?
Also keep in mind that IO to ZFS on the host is different than IO to of guests.
Coping files to a ZFS pool on the host will use datasets and everything is done on file level. Writing/reading to a filesystem in a VM is writing on file level to an additional filesystem like ext4, this will write on blocklevel to a zvol.
So these are completely different workloads even when copying the same files.

Host:
Physical Disks <- vdevs <- dataset

Guest:
Physical Disks <- vdevs <- zvol <- virtio <- guests filesystem
 
Last edited:
Sounds good.

You could try to enable multiqueue for your virtio NIC. You are using virtio and not the slow E1000 NIC right?
Also keep in mind that IO to ZFS on the host is different than IO to of guests.
Coping files to a ZFS pool on the host will use datasets and everything is done on file level. Writing/reading to a filesystem in a VM is writing on file level to an additional filesystem like ext4, this will write on blocklevel to a zvol.
So these are completely different workloads even when copying the same files.

Host:
Physical Disks <- vdevs <- dataset

Guest:
Physical Disks <- vdevs <- zvol <- virtio <- guests filesystem
Yes, Im using VirtIO for VM networking.

What value should I use the VM's VirtIO Multiqueue? I think I read it should match the number of cores of the CPU?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!