[SOLVED] 10GbE IOPS different on host vs VM

[jdancer]

I have a standalone Dell R620 ZFS host with 10GbE networking.

I get the full network IOPS when doing a wget of a file on the R620.

However, when I run the same wget inside a VM, I do NOT get the same network IOPS. Maybe I get about 60-75% network IOPS.

Anything on the host and/or VM settings I can change to get full network IOPS?

I have done the following optimizations:

Set write cache enable (WCE) to 1 on SAS drives
Set VM cache to none
Use VirtIO-single SCSI controller and enable IO thread and discard option
Set VM CPU type to 'host'
Set Linux VMs IO scheduler to none/noop (I do NOT have any Windows VMs, 100% Linux VMs)

EDIT: Fixed the issue by setting Multiqueue value to VM number of vCPUs.

 

[Dunuin]

Sounds good.

You could try to enable multiqueue for your virtio NIC. You are using virtio and not the slow E1000 NIC right?
Also keep in mind that IO to ZFS on the host is different than IO to of guests.
Coping files to a ZFS pool on the host will use datasets and everything is done on file level. Writing/reading to a filesystem in a VM is writing on file level to an additional filesystem like ext4, this will write on blocklevel to a zvol.
So these are completely different workloads even when copying the same files.

Host:
Physical Disks <- vdevs <- dataset

Guest:
Physical Disks <- vdevs <- zvol <- virtio <- guests filesystem

 

[spirit]

don't use wget to benchmark. (you'll test both storage/network at the same, with a single queue/thread). what do you want to bench ? network or storage ?

if network : use iperf
if storage: use fio

 

[jdancer]

Sounds good.

You could try to enable multiqueue for your virtio NIC. You are using virtio and not the slow E1000 NIC right?
Also keep in mind that IO to ZFS on the host is different than IO to of guests.
Coping files to a ZFS pool on the host will use datasets and everything is done on file level. Writing/reading to a filesystem in a VM is writing on file level to an additional filesystem like ext4, this will write on blocklevel to a zvol.
So these are completely different workloads even when copying the same files.

Host:
Physical Disks <- vdevs <- dataset

Guest:
Physical Disks <- vdevs <- zvol <- virtio <- guests filesystem

Yes, Im using VirtIO for VM networking.

What value should I use the VM's VirtIO Multiqueue? I think I read it should match the number of cores of the CPU?

 

[Dunuin]

Yes, Im using VirtIO for VM networking.

What value should I use the VM's VirtIO Multiqueue? I think I read it should match the number of cores of the CPU?

Jup, number of vCPUs.

 

[jdancer]

Thanks. That fixed the network IOPS issue.