Random disk IO is 10 times slower in VM compared to host

[Gareth Blades]

We are currently evaluating Proxmox with the intention to switch from Vmware ESXI. We have been using sysbench within a VM to test performance. All is fine with the exception of randon disk reads and writes which are significantly slower.

Machine specs:-
Lenovo SR630
RAID bus controller: Broadcom / LSI MegaRAID Tri-Mode SAS3508
4* Lenovo 960GB SSDs configured as raid-5. These are read intensive drives rated at 5 full drive writes per day for 5 years.

Testing commands :-
sysbench fileio --file-total-size=4G --file-test-mode=rndrw prepare
sysbench fileio --file-total-size=4G --file-test-mode=rndrw run

Vmware ESXI 8.0 :-
File operations:
reads/s: 8109.14
writes/s: 5406.03
fsyncs/s: 17308.89
Throughput:
read, MiB/s: 126.71
written, MiB/s: 84.47

Proxmox 8.2.2 host :-
File operations:
reads/s: 7155.38
writes/s: 4770.25
fsyncs/s: 15269.00
Throughput:
read, MiB/s: 111.80
written, MiB/s: 74.54

Ubuntu 24.04.1 LTS VM :-
File operations:
reads/s: 420.16
writes/s: 280.04
fsyncs/s: 903.12
Throughput:
read, MiB/s: 6.56
written, MiB/s: 4.38

The VM is configured to use Virtio SCSI Single as the SCSI controller.
I have tried different VM disk cache and Async IO options with little difference. Best the read MiB/s goes up to about 8.

Does anyone know why the random disk IO would be so much worse within the VM compared to the host itself.?

For most things it won't be a problem but for our bigger and busier databases it certainly will be so I would like to get this improved if possible.

Thanks

 

[dakralex]

Welcome to the Proxmox forum, Gareth Blades!

The sudden drop in performance in the Ubuntu VM seems indeed quite odd. Could you post the full Ubuntu VM config (with qm config <vmid>) and information about how the storage of the VM is setup (e.g. output of cat /etc/pve/storage.cfg)? I'm assuming that the SSDs are setup with the hardware RAID controller, is the SAS3508 setup with any cache?

 

[Gareth Blades]

qm config

agent: 1,fstrim_cloned_disks=1
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
ide2: none,media=cdrom
memory: 2048
meta: creation-qemu=8.1.5,ctime=1729519937
name: offpmoxtest
net0: virtio=BC:24:11:38:74:58,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,aio=threads,format=raw,iothread=1,size=50G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=83a0cc73-e6ff-468f-878a-ad77b5122218
sockets: 1
vmgenid: cc15543d-058f-46b3-a16a-727add82b068

/etc/pve/storage.cfg on the proxmox server. I assume that is what you are after. The file doesn't exist on the ubuntu vm

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

Yes the disks are setup with hardware raid. There is a built in cache on the raid controller together with battery backup. This is the raid configuration


Thanks

 

[Gareth Blades]

I've tried with the default io_uring async io and all the 'Cache' settings and also the default 'No Cache' and the different Async io options.
I have SSD Emulation enabled on this host but on the other proxmox install running an older proxmox 8.1 this wasn't emabled as I have the same issue on that system aswell.

 

[Gareth Blades]

Any more thoughts on this?
I've tried it on another server which is the same make and model with the same raid controller. It is running an older proxmox 8.1.4 installation.

Sysbench in host os (proxmox 8.1.4)
Throughput:
read, MiB/s: 111.16
written, MiB/s: 74.11

Sysbench in unbuntu 20.04 vm
Throughput:
read, MiB/s: 23.02
written, MiB/s: 15.35

It's not quite as bad as the other machine but still over 4* slower.