Slow write performance in Rocky/RHEL VMs

L

Leo Karian

New Member
Aug 19, 2024
6
2
3
Hello everyone,
at my work we are migrating from ESXi to Proxmox. After having migrated about 25 VMs with Rocky and having created about 10 more VMs, I have realized that the writing speed is being affected by something I don't know. In the ESXi VMs everything works correctly.

I have created several test VMs and I have noticed that in VMs with Rocky Linux and RHEL distributions I have this problem, but the same does not happen with Fedora or Debian.
To do the tests I have configured all the VMs with the same configuration and the writing is done directly on the local disk of the host.

For the tests I have done I've used iperf3, both to measure the network and to create a file (using the -F option). Also with dd I get slow write performance.

After several weeks of testing with different configurations, the result is always the same and I don't know what else to look for.
I haven't found similar problems in the forum or on google.

These are the configuration I have and the tests I have done:

Server bare-metal: PowerEdge R750xs
Proxmox: pve-manager/8.3.3/f157a38b211595d6 (running kernel: 6.8.12-2-pve)

VMs configuration:
acpi: 1
agent: 1
boot: order=scsi0
cores: 4
cpu: host
hotplug: 0
machine: q35
memory: 4096
meta: creation-qemu=9.0.2,ctime=1737742842
net0: virtio=BC:24:11:4C:24:36,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local:XXX/vm-XXX-disk-0.qcow2,discard=on,iothread=1,size=30G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=xxxxxxxx-79f6-496f-be66-xxxxxxx4723f
sockets: 1
vmgenid: xxxxxxxx-6793-43aa-9dc6-xxxxx26abf3b
vmstatestorage: local


Kernel versions:
Debian12: Linux debian12-test 6.1.0-30-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.124-1 (2025-01-12) x86_64 GNU/Linux
Fedora29: Linux fedora29-test 4.18.16-300.fc29.x86_64 #1 SMP Sat Oct 20 23:24:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Fedora33: Linux fedora33-test 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Rocky8: Linux rocky8-test 4.18.0-553.el8_10.x86_64 #1 SMP Fri May 24 13:05:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Rocky9: Linux rocky9-test 5.14.0-503.14.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 15 12:04:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Redhat8: Linux redhat8-test 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

Tests:

Debian 12
Virtio Drivers:
root@debian12-test:~# lsinitramfs /boot/initrd.img-$(uname -r) | grep virtio
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/block/virtio_blk.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/net/virtio_net.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/scsi/virtio_scsi.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_mmio.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci_legacy_dev.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci_modern_dev.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_ring.ko

File WritingTest:
root@debian12-test:~# iperf3 -s -p 5003 -F /tmp/iperf.tmp
-----------------------------------------------------------
Server listening on 5003 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.200.38, port 33472
[ 5] local 192.168.200.202 port 5003 connected to 192.168.200.38 port 33476
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 112 MBytes 941 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec
...
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
[ 5] 10.00-10.00 sec 304 KBytes 720 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
Sent 1.10 GByte / 1.10 GByte (100%) of /tmp/iperf.tmp
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver


Fedora 29 and Fedora 33
Virtio Drivers: are same modules for both.
Fedora29 kernel version: $kernel=4.18.16-300.fc29.x86_64
Fedora33 kernel version: $kernel=5.8.15-301.fc33.x86_64
[root@fedoraXX-test ~]# lsinitrd /boot/initramfs-$(uname -r).img | grep virtio
usr/lib/modules/$kernel/kernel/drivers/block/virtio_blk.ko.xz
usr/lib/modules/$kernel/kernel/drivers/char/virtio_console.ko.xz
usr/lib/modules/$kernel/kernel/drivers/net/virtio_net.ko.xz
usr/lib/modules/$kernel/kernel/drivers/scsi/virtio_scsi.ko.xz

File WritingTest: are similar for both
[root@fedoraXX-test ~]# iperf3 -s -p 5003 -F /tmp/iperf.tmp
-----------------------------------------------------------
Server listening on 5003
-----------------------------------------------------------
Accepted connection from 192.168.200.38, port 34452
[ 5] local 192.168.200.198 port 5003 connected to 192.168.200.38 port 34460
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 108 MBytes 905 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec
...
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
[ 5] 10.00-10.04 sec 4.48 MBytes 923 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
Sent 1.10 GByte / 1.10 GByte (100%) of /tmp/iperf.tmp
[ 5] 0.00-10.04 sec 1.10 GBytes 938 Mbits/sec receiver


Rocky Linux 8
Virtio Drivers:
[root@rocky8-test ~]# lsinitrd /boot/initramfs-$(uname -r).img | grep virtio
-rw-r--r-- 1 root root 9228 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/block/virtio_blk.ko.xz
-rw-r--r-- 1 root root 15488 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/char/virtio_console.ko.xz
drwxr-xr-x 2 root root 0 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/virtio
-rw-r--r-- 1 root root 32056 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/virtio/virtio-gpu.ko.xz
-rw-r--r-- 1 root root 27628 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/net/virtio_net.ko.xz
-rw-r--r-- 1 root root 8812 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xz
drwxr-xr-x 2 root root 0 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio
-rw-r--r-- 1 root root 15292 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio/virtio_mem.ko.xz
-rw-r--r-- 1 root root 4564 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio/virtio_vdpa.ko.xz
-rw-r--r-- 1 root root 13196 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/fs/fuse/virtiofs.ko.xz

[root@rocky8-test ~]# grep -i virtio /boot/config-$(uname -r)
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
# CONFIG_BT_VIRTIO is not set
CONFIG_VIRTIO_BLK=m
# CONFIG_VIRTIO_BLK_SCSI is not set
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_DRM_VIRTIO_GPU=m
CONFIG_SND_VIRTIO=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI_LIB=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_VDPA=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_MEM=m
CONFIG_VIRTIO_INPUT=m
# CONFIG_VIRTIO_MMIO is not set
# CONFIG_VIRTIO_IOMMU is not set
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=m
# CONFIG_CRYPTO_DEV_VIRTIO is not set

File WritingTest:
[root@rocky8-test ~]# iperf3 -s -p 5003 -F /tmp/iperf.tmp
-----------------------------------------------------------
Server listening on 5003
-----------------------------------------------------------
Accepted connection from 192.168.200.38, port 47280
[ 5] local 192.168.200.200 port 5003 connected to 192.168.200.38 port 47288
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 46.0 MBytes 383 Mbits/sec
[ 5] 1.01-2.00 sec 51.2 MBytes 431 Mbits/sec
[ 5] 2.00-3.01 sec 46.2 MBytes 387 Mbits/sec
[ 5] 3.01-4.00 sec 49.6 MBytes 418 Mbits/sec
[ 5] 4.00-5.01 sec 50.0 MBytes 419 Mbits/sec
[ 5] 5.01-6.00 sec 49.2 MBytes 414 Mbits/sec
[ 5] 6.00-7.00 sec 60.2 MBytes 507 Mbits/sec
[ 5] 7.00-8.00 sec 64.0 MBytes 536 Mbits/sec
[ 5] 8.00-9.00 sec 66.2 MBytes 556 Mbits/sec
[ 5] 9.00-10.00 sec 69.4 MBytes 582 Mbits/sec
[ 5] 10.00-10.04 sec 3.00 MBytes 662 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
Sent 555 MByte / 555 MByte (100%) of /tmp/iperf.tmp
[ 5] 0.00-10.04 sec 555 MBytes 464 Mbits/sec receiver


Rocky Linux 9
Virtio Drivers:
[root@rocky9-test ~]# lsinitrd /boot/initramfs-$(uname -r).img | grep virtio
virtiofs
-rwxr-xr-x 1 root root 208 Jun 19 2022 usr/lib/dracut/hooks/cmdline/95-parse-virtiofs.sh
-rwxr-xr-x 1 root root 637 Jun 19 2022 usr/lib/dracut/hooks/pre-mount/99-mount-virtiofs.sh
-rw-r--r-- 1 root root 16016 Nov 2 23:24 usr/lib/modules/5.14.0-503.14.1.el9_5.x86_64/kernel/drivers/block/virtio_blk.ko.xz
-rw-r--r-- 1 root root 18580 Nov 2 23:24 usr/lib/modules/5.14.0-503.14.1.el9_5.x86_64/kernel/drivers/char/virtio_console.ko.xz
-rw-r--r-- 1 root root 39656 Nov 2 23:24 usr/lib/modules/5.14.0-503.14.1.el9_5.x86_64/kernel/drivers/net/virtio_net.ko.xz
-rw-r--r-- 1 root root 11468 Nov 2 23:24 usr/lib/modules/5.14.0-503.14.1.el9_5.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xz
drwxr-xr-x 2 root root 0 Nov 2 23:24 usr/lib/modules/5.14.0-503.14.1.el9_5.x86_64/kernel/drivers/virtio
-rw-r--r-- 1 root root 19836 Nov 2 23:24 usr/lib/modules/5.14.0-503.14.1.el9_5.x86_64/kernel/drivers/virtio/virtio_mem.ko.xz

[root@rocky9-test ~]# grep -i virtio /boot/config-$(uname -r)
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
# CONFIG_BT_VIRTIO is not set
CONFIG_VIRTIO_BLK=m
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=y
# CONFIG_I2C_VIRTIO is not set
# CONFIG_GPIO_VIRTIO is not set
CONFIG_DRM_VIRTIO_GPU=m
CONFIG_DRM_VIRTIO_GPU_KMS=y
CONFIG_SND_VIRTIO=m
CONFIG_VIRTIO_ANCHOR=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI_LIB=y
CONFIG_VIRTIO_PCI_LIB_LEGACY=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_VDPA=m
# CONFIG_VIRTIO_PMEM is not set
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_MEM=m
CONFIG_VIRTIO_INPUT=m
# CONFIG_VIRTIO_MMIO is not set
CONFIG_VIRTIO_DMA_SHARED_BUFFER=m
# CONFIG_XEN_VIRTIO is not set
CONFIG_VIRTIO_IOMMU=y
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=m
# CONFIG_CRYPTO_DEV_VIRTIO is not set

File WritingTest:
[root@rocky9-test ~]# iperf3 -s -p 5003 -F /tmp/iperf.tmp
-----------------------------------------------------------
Server listening on 5003
-----------------------------------------------------------
Accepted connection from 192.168.200.38, port 56994
[ 5] local 192.168.200.201 port 5003 connected to 192.168.200.38 port 57008
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 77.5 MBytes 649 Mbits/sec
[ 5] 1.00-2.00 sec 70.9 MBytes 594 Mbits/sec
[ 5] 2.00-3.00 sec 76.8 MBytes 643 Mbits/sec
[ 5] 3.00-4.00 sec 79.4 MBytes 666 Mbits/sec
[ 5] 4.00-5.00 sec 70.4 MBytes 589 Mbits/sec
[ 5] 5.00-6.00 sec 78.9 MBytes 663 Mbits/sec
[ 5] 6.00-7.00 sec 60.6 MBytes 508 Mbits/sec
[ 5] 7.00-8.00 sec 55.9 MBytes 469 Mbits/sec
[ 5] 8.00-9.00 sec 61.2 MBytes 514 Mbits/sec
[ 5] 9.00-10.00 sec 47.9 MBytes 402 Mbits/sec
[ 5] 10.00-10.04 sec 1.75 MBytes 362 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
Sent 681 MByte / 681 MByte (100%) of /tmp/iperf.tmp
[ 5] 0.00-10.04 sec 681 MBytes 569 Mbits/sec receiver


Red Hat 8
Virtio Drivers:
[root@redhat8-test ~]# lsinitrd /boot/initramfs-$(uname -r).img | grep virtio
-rw-r--r-- 1 root root 8996 Sep 20 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/block/virtio_blk.ko.xz
-rw-r--r-- 1 root root 14856 Sep 20 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/char/virtio_console.ko.xz
-rw-r--r-- 1 root root 25644 Sep 20 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/net/virtio_net.ko.xz
-rw-r--r-- 1 root root 8624 Sep 20 2021 usr/lib/modules/4.18.0-348.el8.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xz

[root@redhat8-test ~]# grep -i virtio /boot/config-$(uname -r)
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
CONFIG_VIRTIO_BLK=m
# CONFIG_VIRTIO_BLK_SCSI is not set
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_DRM_VIRTIO_GPU=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI_LIB=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_VDPA=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_INPUT=m
# CONFIG_VIRTIO_MMIO is not set
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=m
# CONFIG_CRYPTO_DEV_VIRTIO is not set

File WritingTest:
[root@redhat8-test ~]# iperf3 -s -p 5003 -F /tmp/iperf.tmp
-----------------------------------------------------------
Server listening on 5003
-----------------------------------------------------------
Accepted connection from 192.168.200.38, port 44124
[ 5] local 192.168.200.203 port 5003 connected to 192.168.200.38 port 44136
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 55.0 MBytes 460 Mbits/sec
[ 5] 1.00-2.00 sec 55.1 MBytes 463 Mbits/sec
[ 5] 2.00-3.00 sec 48.0 MBytes 402 Mbits/sec
[ 5] 3.00-4.00 sec 55.8 MBytes 468 Mbits/sec
[ 5] 4.00-5.00 sec 57.6 MBytes 484 Mbits/sec
[ 5] 5.00-6.00 sec 57.6 MBytes 483 Mbits/sec
[ 5] 6.00-7.01 sec 55.9 MBytes 468 Mbits/sec
[ 5] 7.01-8.00 sec 56.9 MBytes 478 Mbits/sec
[ 5] 8.00-9.00 sec 51.1 MBytes 428 Mbits/sec
[ 5] 9.00-10.01 sec 48.6 MBytes 406 Mbits/sec
[ 5] 10.01-10.04 sec 1.12 MBytes 295 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
Sent 543 MByte / 543 MByte (100%) of /tmp/iperf.tmp
[ 5] 0.00-10.04 sec 543 MBytes 454 Mbits/sec receiver


I suspect that it must be a problem with the virtio scsi drivers, which, according to what I have read, are included in the kernel. But this topic is a bit beyond my knowledge.

I would like to thank you in advance for any clues you can give me.

Regards
 
I tested same with same like results with rocky8 vm for "iperf3 -F /path/file" ... but when do *NOT* use " -F" iperf3 shows 14,5 Gbit/s and dd for a 10GB 1,3GB/s which both show 10Gbit limit for host and vm to image nfs storage.
So looks like a (EDIT: inside vm RH/Rocky/Fedora "-F") iperf3 binary to kernel interface problem ...
 
Last edited:
  • Like
Reactions: Leo Karian
waltar said:
I tested same with same like results with rocky8 vm for "iperf3 -F /path/file" ... but when do *NOT* use " -F" iperf3 shows 14,5 Gbit/s and dd for a 10GB 1,3GB/s which both show 10Gbit limit for host and vm to image nfs storage.
So looks like a iperf3 binary to kernel interface problem ...
Click to expand...
Personally I test thing after thing. For example iperf3 -F test network bandwidth + file access + potentially VM OS cache , VM FS cache, host OS cache, RAID card cache, disk cache, ...
For disks I use : https://wiki.liberasys.com/doku.php?id=diskwipetests:disks_wipe_and_tests&s[]=test
For bandwidth, iperf should be good, without "-F option".
 
  • Like
Reactions: Leo Karian
Thanks waltar and ghusson for your replies.
Indeed, when the -F option is not used the speed I get is the highest that can be obtained with 1GB. That is why I have discarded the network as the problem.

To make a more visual test, I have created another VM in ESXi. Now, in this test, both are Rocky Linux 8.10, created from scratch, with the default settings.

First I ran a few tests with dd on both VMs, and while they are on different bare-metal and speeds are different, disk writing should not be a problem, as both are far faster than the network.

[root@rocky8-test ~]# dd if=/dev/zero of=/tmp/test.tmp bs=1G count=5 oflag=dsync
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.13815 s, 875 MB/s
[root@rocky8-test ~]# dd if=/dev/zero of=/tmp/test.tmp bs=1G count=5 oflag=dsync
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 5.81296 s, 924 MB/s
[root@rocky8-test ~]# dd if=/dev/zero of=/tmp/test.tmp bs=1G count=5 oflag=dsync
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.87045 s, 781 MB/s

[root@rocky8-test-esxi ~]# dd if=/dev/zero of=/tmp/test.tmp bs=1G count=5 oflag=dsync
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 3.93095 s, 1.4 GB/s
[root@rocky8-test-esxi ~]# dd if=/dev/zero of=/tmp/test.tmp bs=1G count=5 oflag=dsync
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 4.25501 s, 1.3 GB/s
[root@rocky8-test-esxi ~]# dd if=/dev/zero of=/tmp/test.tmp bs=1G count=5 oflag=dsync
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 4.16146 s, 1.3 GB/s


I understand from both of your comments that iperf3 is not a good tool to test with the -F option, although the results confuse me a bit, as both VMs have the same OS. The tests with iperf3, from one VM to the other, and vice versa, are these:

[root@rocky8-test-esxi ~]# iperf3 -s -p 5003 -F /tmp/iperf.tmp
-----------------------------------------------------------
Server listening on 5003
-----------------------------------------------------------
Accepted connection from 192.168.200.200, port 50810
[ 5] local 192.168.200.205 port 5003 connected to 192.168.200.200 port 50814
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 104 MBytes 870 Mbits/sec
[ 5] 1.00-2.00 sec 109 MBytes 915 Mbits/sec
[ 5] 2.00-3.00 sec 111 MBytes 930 Mbits/sec
[ 5] 3.00-4.00 sec 106 MBytes 887 Mbits/sec
[ 5] 4.00-5.00 sec 109 MBytes 914 Mbits/sec
[ 5] 5.00-6.00 sec 107 MBytes 899 Mbits/sec
[ 5] 6.00-7.00 sec 106 MBytes 886 Mbits/sec
[ 5] 7.00-8.00 sec 106 MBytes 887 Mbits/sec
[ 5] 8.00-9.00 sec 108 MBytes 909 Mbits/sec
[ 5] 9.00-10.00 sec 110 MBytes 925 Mbits/sec
[ 5] 10.00-10.06 sec 6.12 MBytes 901 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
Sent 1.06 GByte / 1.06 GByte (100%) of /tmp/iperf.tmp
[ 5] 0.00-10.06 sec 1.06 GBytes 902 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5003
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
[root@rocky8-test-esxi ~]# iperf3 -c 192.168.200.200 -p 5003
Connecting to host 192.168.200.200, port 5003
[ 5] local 192.168.200.205 port 49796 connected to 192.168.200.200 port 5003
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 72.0 MBytes 604 Mbits/sec 0 445 KBytes
[ 5] 1.00-2.00 sec 63.9 MBytes 536 Mbits/sec 0 445 KBytes
[ 5] 2.00-3.00 sec 60.7 MBytes 509 Mbits/sec 0 445 KBytes
[ 5] 3.00-4.00 sec 60.3 MBytes 506 Mbits/sec 1 445 KBytes
[ 5] 4.00-5.00 sec 59.3 MBytes 497 Mbits/sec 0 445 KBytes
[ 5] 5.00-6.00 sec 58.9 MBytes 494 Mbits/sec 0 445 KBytes
[ 5] 6.00-7.00 sec 56.0 MBytes 470 Mbits/sec 0 445 KBytes
[ 5] 7.00-8.00 sec 52.9 MBytes 444 Mbits/sec 0 445 KBytes
[ 5] 8.00-9.00 sec 56.3 MBytes 472 Mbits/sec 0 445 KBytes
[ 5] 9.00-10.00 sec 51.3 MBytes 430 Mbits/sec 0 445 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 592 MBytes 496 Mbits/sec 1 sender
[ 5] 0.00-10.04 sec 587 MBytes 491 Mbits/sec receiver

iperf Done.

Second test is the problem, from ESXi VM to Proxmox VM.

Then, following ghusson's recommendation, I installed fio on Rocky 8 and Debian 12, which are on the same bare-metal, same setup, and got these results:

root@debian12-test:~# DATEHOUR=$(date "+%Y%m%d-%H%M%S")
DISKNAME="iscsi"
fio --size=5G --sync=0 --fsync=100 --direct=1 --verify=0 --zero_buffers --invalidate=1 --ioengine=libaio --rw=randrw --rwmixread=50 --bssplit=8k/50:16k/50 --numjobs=50 --iodepth=64 --runtime=120 --time_based --group_reporting --gtod_reduce=1 --filename=/tmp/fio_pool --name "/tmp/fio_test_${DISKNAME}-${DATEHOUR}"
/tmp/fio_test_iscsi-20250131-131820: (g=0): rw=randrw, bs=(R) 8192B-16.0KiB, (W) 8192B-16.0KiB, (T) 8192B-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.33
Starting 50 processes
/tmp/fio_test_iscsi-20250131-131820: Laying out IO file (1 file / 5120MiB)
Jobs: 48 (f=0): [f(2),E(1),f(1),E(1),f(45)][100.0%][r=1231MiB/s,w=1233MiB/s][r=107k,w=107k IOPS][eta 00m:00s]
/tmp/fio_test_iscsi-20250131-131820: (groupid=0, jobs=50): err= 0: pid=879: Fri Jan 31 13:20:25 2025
read: IOPS=108k, BW=1153MiB/s (1209MB/s)(135GiB/120025msec)
bw ( MiB/s): min= 533, max= 2032, per=100.00%, avg=1153.41, stdev= 5.14, samples=11950
iops : min=50777, max=185545, avg=108371.77, stdev=474.16, samples=11950
write: IOPS=108k, BW=1153MiB/s (1209MB/s)(135GiB/120025msec); 0 zone resets
bw ( MiB/s): min= 531, max= 2023, per=100.00%, avg=1153.56, stdev= 5.14, samples=11950
iops : min=50502, max=184904, avg=108388.23, stdev=474.62, samples=11950
cpu : usr=0.37%, sys=2.38%, ctx=11749156, majf=0, minf=575
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=101.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=12994731,12997103,0,259842 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=1153MiB/s (1209MB/s), 1153MiB/s-1153MiB/s (1209MB/s-1209MB/s), io=135GiB (145GB), run=120025-120025msec
WRITE: bw=1153MiB/s (1209MB/s), 1153MiB/s-1153MiB/s (1209MB/s-1209MB/s), io=135GiB (145GB), run=120025-120025msec

Disk stats (read/write):
dm-0: ios=12971098/13053006, merge=0/0, ticks=47268684/47510168, in_queue=94778852, util=100.00%, aggrios=12995032/13061068, aggrmerge=2/16164, aggrticks=46461798/46592104, aggrin_queue=93114263, aggrutil=56.92%
sda: ios=12995032/13061068, merge=2/16164, ticks=46461798/46592104, in_queue=93114263, util=56.92%

[root@rocky8-test ~]# fio --size=5G --sync=0 --fsync=100 --direct=1 --verify=0 --zero_buffers --invalidate=1 --ioengine=libaio --rw=randrw --rwmixread=50 --bssplit=8k/50:16k/50 --numjobs=50 --iodepth=64 --runtime=120 --time_based --group_reporting --gtod_reduce=1 --filename=/tmp/fio_pool --name "/tmp/fio_test_${DISKNAME}-${DATEHOUR}"
/tmp/fio_test_iscsi-20250131-130639: (g=0): rw=randrw, bs=(R) 8192B-16.0KiB, (W) 8192B-16.0KiB, (T) 8192B-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 50 processes
/tmp/fio_test_iscsi-20250131-130639: Laying out IO file (1 file / 5120MiB)
Jobs: 50 (f=50): [m(50)][100.0%][r=292MiB/s,w=295MiB/s][r=26.6k,w=26.8k IOPS][eta 00m:00s]
/tmp/fio_test_iscsi-20250131-130639: (groupid=0, jobs=50): err= 0: pid=2513: Fri Jan 31 13:08:46 2025
read: IOPS=28.9k, BW=328MiB/s (344MB/s)(38.4GiB/120041msec)
bw ( KiB/s): min=130808, max=688966, per=100.00%, avg=335788.05, stdev=1920.66, samples=11950
iops : min=11277, max=58571, avg=28928.44, stdev=162.36, samples=11950
write: IOPS=28.9k, BW=328MiB/s (344MB/s)(38.4GiB/120041msec); 0 zone resets
bw ( KiB/s): min=136029, max=693837, per=100.00%, avg=335982.26, stdev=1913.47, samples=11950
iops : min=11743, max=58651, avg=28933.79, stdev=161.45, samples=11950
cpu : usr=0.35%, sys=1.48%, ctx=2058635, majf=0, minf=393
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=101.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3471682,3472403,0,69575 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=38.4GiB (41.2GB), run=120041-120041msec
WRITE: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=38.4GiB (41.3GB), run=120041-120041msec

Disk stats (read/write):
dm-0: ios=3465297/3594948, merge=0/0, ticks=1554905/1597514, in_queue=3152419, util=100.00%, aggrios=3471678/3601374, aggrmerge=4/112, aggrticks=1554000/1593081, aggrin_queue=3147080, aggrutil=100.00%
sda: ios=3471678/3601374, merge=4/112, ticks=1554000/1593081, in_queue=3147080, util=100.00%

I don't understand why there is such a difference between the two tests.
 
Yes, that's correct!
Following this link (https://www.tencentcloud.com/document/product/213/9929) I got virtio info:

[root@rocky8-test ~]# grep -i virtio /boot/config-$(uname -r)
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
# CONFIG_BT_VIRTIO is not set
CONFIG_VIRTIO_BLK=m
# CONFIG_VIRTIO_BLK_SCSI is not set
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_DRM_VIRTIO_GPU=m
CONFIG_SND_VIRTIO=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI_LIB=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_VDPA=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_MEM=m
CONFIG_VIRTIO_INPUT=m
# CONFIG_VIRTIO_MMIO is not set
# CONFIG_VIRTIO_IOMMU is not set
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=m
# CONFIG_CRYPTO_DEV_VIRTIO is not set

[root@rocky8-test ~]# lsinitrd /boot/initramfs-$(uname -r).img | grep virtio
-rw-r--r-- 1 root root 9228 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/block/virtio_blk.ko.xz
-rw-r--r-- 1 root root 15488 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/char/virtio_console.ko.xz
drwxr-xr-x 2 root root 0 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/virtio
-rw-r--r-- 1 root root 32056 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/virtio/virtio-gpu.ko.xz
-rw-r--r-- 1 root root 27628 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/net/virtio_net.ko.xz
-rw-r--r-- 1 root root 8812 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xz
drwxr-xr-x 2 root root 0 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio
-rw-r--r-- 1 root root 15292 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio/virtio_mem.ko.xz
-rw-r--r-- 1 root root 4564 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio/virtio_vdpa.ko.xz
-rw-r--r-- 1 root root 13196 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/fs/fuse/virtiofs.ko.xz

root@debian12-test:~# grep -i virtio /boot/config-$(uname -r)
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
# CONFIG_BT_VIRTIO is not set
CONFIG_NET_9P_VIRTIO=m
CONFIG_VIRTIO_BLK=m
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=m
# CONFIG_I2C_VIRTIO is not set
# CONFIG_GPIO_VIRTIO is not set
CONFIG_DRM_VIRTIO_GPU=m
# CONFIG_SND_VIRTIO is not set
CONFIG_VIRTIO_ANCHOR=y
CONFIG_VIRTIO=m
CONFIG_VIRTIO_PCI_LIB=m
CONFIG_VIRTIO_PCI_LIB_LEGACY=m
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_PMEM=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_MEM=m
CONFIG_VIRTIO_INPUT=m
CONFIG_VIRTIO_MMIO=m
# CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set
CONFIG_VIRTIO_DMA_SHARED_BUFFER=m
# CONFIG_XEN_VIRTIO is not set
# CONFIG_VIRTIO_IOMMU is not set
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=m
CONFIG_CRYPTO_DEV_VIRTIO=m

root@debian12-test:~# lsinitramfs /boot/initrd.img-$(uname -r) | grep virtio
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/block/virtio_blk.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/net/virtio_net.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/scsi/virtio_scsi.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_mmio.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci_legacy_dev.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci_modern_dev.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_ring.ko

I don't have much knowledge of modules in Linux, although I don't think they are the same virtio drivers since the kernels are different, but I suspect there is something to do with these drivers and the correct configuration of the VM in Proxmox or, there is simply some error with the drivers.
 
  • Like
Reactions: waltar
Obviously the debian modules work much better ... the kernel from rh8 is much older (but have couple of backports) than the debian one.
What about selinux overhead, is this still activ on rocky8 vm ?
Nevertheless as seen a debian vm is a better choice for an i/o app if possible yet.
 
  • Like
Reactions: Leo Karian
waltar, I have disabled selinux and the behavior is the same, so I can rule that out.
I also think that rh8 kernel is older than debian 12, but with fedora 29 they are almost the same and yet the speed is different.
On the other hand, I still can't understand why, if with fio I can get speeds of 340 MB/s, when I transfer data over the network, it only uses half the bandwidth (60 MB/s) if the disk write is quite high and the network-only tests are fine.

bbgeek17, I hadn't thought of that. I'll look into and try to update with that repository.

Many thanks to both of you!
I'll do the tests and let you know again.
 
  • Like
Reactions: Kingneutron
bbgeek17 said:
With RHEL based system one could use ELRepo kernel for more recent kernel version
Click to expand...
Never heard about that ... and would never do on production system under subscription ... but anyway, hey, nothing against a test in a vm
which just is for testing or low level service on rocky ... maybe a good deal ... I will try that - thanks.
 
Leo Karian said:
scsi0: local:XXX/vm-XXX-disk-0.qcow2,discard=on,iothread=1,size=30G,ssd=1
scsihw: virtio-scsi-single
Click to expand...

Change the VM config:
Code: 
scsi0: local:XXX/vm-XXX-disk-0.qcow2,size=30G,ssd=1,aio=native,cache=none,detect-zeros=off,discard=off
scsihw: virtio-scsi-pci

Is there any improvement ?
 
Last edited:
  • Like
Reactions: Leo Karian
Leo Karian said:
Yes, that's correct!
Following this link (https://www.tencentcloud.com/document/product/213/9929) I got virtio info:

[root@rocky8-test ~]# grep -i virtio /boot/config-$(uname -r)
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
# CONFIG_BT_VIRTIO is not set
CONFIG_VIRTIO_BLK=m
# CONFIG_VIRTIO_BLK_SCSI is not set
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_DRM_VIRTIO_GPU=m
CONFIG_SND_VIRTIO=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI_LIB=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_VDPA=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_MEM=m
CONFIG_VIRTIO_INPUT=m
# CONFIG_VIRTIO_MMIO is not set
# CONFIG_VIRTIO_IOMMU is not set
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=m
# CONFIG_CRYPTO_DEV_VIRTIO is not set

[root@rocky8-test ~]# lsinitrd /boot/initramfs-$(uname -r).img | grep virtio
-rw-r--r-- 1 root root 9228 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/block/virtio_blk.ko.xz
-rw-r--r-- 1 root root 15488 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/char/virtio_console.ko.xz
drwxr-xr-x 2 root root 0 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/virtio
-rw-r--r-- 1 root root 32056 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/virtio/virtio-gpu.ko.xz
-rw-r--r-- 1 root root 27628 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/net/virtio_net.ko.xz
-rw-r--r-- 1 root root 8812 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xz
drwxr-xr-x 2 root root 0 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio
-rw-r--r-- 1 root root 15292 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio/virtio_mem.ko.xz
-rw-r--r-- 1 root root 4564 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/virtio/virtio_vdpa.ko.xz
-rw-r--r-- 1 root root 13196 Jan 15 2024 usr/lib/modules/4.18.0-553.el8_10.x86_64/kernel/fs/fuse/virtiofs.ko.xz

root@debian12-test:~# grep -i virtio /boot/config-$(uname -r)
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
# CONFIG_BT_VIRTIO is not set
CONFIG_NET_9P_VIRTIO=m
CONFIG_VIRTIO_BLK=m
CONFIG_SCSI_VIRTIO=m
CONFIG_VIRTIO_NET=m
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM_VIRTIO=m
# CONFIG_I2C_VIRTIO is not set
# CONFIG_GPIO_VIRTIO is not set
CONFIG_DRM_VIRTIO_GPU=m
# CONFIG_SND_VIRTIO is not set
CONFIG_VIRTIO_ANCHOR=y
CONFIG_VIRTIO=m
CONFIG_VIRTIO_PCI_LIB=m
CONFIG_VIRTIO_PCI_LIB_LEGACY=m
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_PMEM=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_MEM=m
CONFIG_VIRTIO_INPUT=m
CONFIG_VIRTIO_MMIO=m
# CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set
CONFIG_VIRTIO_DMA_SHARED_BUFFER=m
# CONFIG_XEN_VIRTIO is not set
# CONFIG_VIRTIO_IOMMU is not set
# CONFIG_RPMSG_VIRTIO is not set
CONFIG_VIRTIO_FS=m
CONFIG_CRYPTO_DEV_VIRTIO=m

root@debian12-test:~# lsinitramfs /boot/initrd.img-$(uname -r) | grep virtio
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/block/virtio_blk.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/net/virtio_net.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/scsi/virtio_scsi.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_mmio.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci_legacy_dev.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_pci_modern_dev.ko
usr/lib/modules/6.1.0-30-amd64/kernel/drivers/virtio/virtio_ring.ko

I don't have much knowledge of modules in Linux, although I don't think they are the same virtio drivers since the kernels are different, but I suspect there is something to do with these drivers and the correct configuration of the VM in Proxmox or, there is simply some error with the drivers.
Click to expand...
Wohooo...
4.18 kernel is from 2018 and you want it to work optimally with recent kernel VIRTIO drivers support.
I don't think it is a good expectation.
Do not use virtio drivers, take full devices visualization drivers and try again with both OS, we will see ?
 
  • Like
Reactions: Leo Karian
emunt6 said:
Change the VM config:
Code: 
scsi0: local:XXX/vm-XXX-disk-0.qcow2,size=30G,ssd=1,aio=native,cache=none,detect-zeros=off,discard=off
scsihw: virtio-scsi-pci

Is there any improvement ?
Click to expand...
When I config the VM with those parameters I get error: "detect-zeros: property is not defined in schema and the schema does not allow additional properties"
But in any case, without that it's the same, there are no improvements.
I tried many VM configurations, especially changing Async IO, without any better results.

bbgeek17 said:
With RHEL based system one could use ELRepo kernel for a more up to date kernel version
Click to expand...
I installed several kernels from ElRepo, and more or less all with the same results:
FIO command (60 secs): fio --size=5G --sync=0 --fsync=100 --direct=1 --verify=0 --zero_buffers --invalidate=1 --ioengine=libaio --rw=randrw --rwmixread=50 --bssplit=8k/50:16k/50 --numjobs=50 --iodepth=64 --runtime=60 --time_based --group_reporting --gtod_reduce=1 --filename=/tmp/fio_pool --name "/tmp/fio_test_${DISKNAME}-${DATEHOUR}"


#Linux rocky9-test 5.14.0-503.14.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 15 12:04:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Run status group 0 (all jobs):
READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=26.3GiB (28.3GB), run=60022-60022msec
WRITE: bw=450MiB/s (471MB/s), 450MiB/s-450MiB/s (471MB/s-471MB/s), io=26.3GiB (28.3GB), run=60022-60022msec

#Linux rocky9-test 5.14.0-503.22.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 22 13:59:07 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Run status group 0 (all jobs):
READ: bw=451MiB/s (473MB/s), 451MiB/s-451MiB/s (473MB/s-473MB/s), io=26.5GiB (28.4GB), run=60030-60030msec
WRITE: bw=451MiB/s (473MB/s), 451MiB/s-451MiB/s (473MB/s-473MB/s), io=26.5GiB (28.4GB), run=60030-60030msec

ELRepo - kernel-ml 6.12
#Linux rocky9-test 6.12.11-1.el9.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jan 23 14:11:10 EST 2025 x86_64 x86_64 x86_64 GNU/Linux
Run status group 0 (all jobs):
READ: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s), io=28.2GiB (30.3GB), run=60032-60032msec
WRITE: bw=482MiB/s (505MB/s), 482MiB/s-482MiB/s (505MB/s-505MB/s), io=28.2GiB (30.3GB), run=60032-60032msec

ELRepo - kernel-ml 6.13
#Linux rocky9-test 6.13.1-1.el9.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Feb 1 15:44:51 EST 2025 x86_64 x86_64 x86_64 GNU/Linux
Run status group 0 (all jobs):
READ: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s), io=30.5GiB (32.7GB), run=60018-60018msec
WRITE: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s), io=30.5GiB (32.7GB), run=60018-60018msec

ELRepo - kernel-lt 6.1
#Linux rocky9-test 6.1.128-1.el9.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Feb 1 15:04:00 EST 2025 x86_64 x86_64 x86_64 GNU/Linux
Run status group 0 (all jobs):
READ: bw=477MiB/s (500MB/s), 477MiB/s-477MiB/s (500MB/s-500MB/s), io=27.9GiB (30.0GB), run=60039-60039msec
WRITE: bw=477MiB/s (500MB/s), 477MiB/s-477MiB/s (500MB/s-500MB/s), io=27.9GiB (30.0GB), run=60039-60039msec

_gabriel said:
perhaps XFS vs ext4 ?
testing in a CT should exclude kernel.
Click to expand...
In containers, using the PVE kernel, it is even slower:
Rocky 8 Container
#Linux infra-Rocky8-CT-Test 6.8.12-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z) x86_64 x86_64 x86_64 GNU/Linux
Run status group 0 (all jobs):
READ: bw=31.8MiB/s (33.4MB/s), 31.8MiB/s-31.8MiB/s (33.4MB/s-33.4MB/s), io=1918MiB (2011MB), run=60264-60264msec
WRITE: bw=31.8MiB/s (33.4MB/s), 31.8MiB/s-31.8MiB/s (33.4MB/s-33.4MB/s), io=1918MiB (2012MB), run=60264-60264msec

Rocky 9 Container
#Linux infra-Rocky9-CT-Test 6.8.12-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z) x86_64 x86_64 x86_64 GNU/Linux
Run status group 0 (all jobs):
READ: bw=34.4MiB/s (36.1MB/s), 34.4MiB/s-34.4MiB/s (36.1MB/s-36.1MB/s), io=2074MiB (2175MB), run=60311-60311msec
WRITE: bw=34.4MiB/s (36.1MB/s), 34.4MiB/s-34.4MiB/s (36.1MB/s-36.1MB/s), io=2075MiB (2176MB), run=60311-60311msec

I haven't tried EXT4 yet, I'll try that later, but I doubt it will improve.


ghusson said:
Wohooo...
4.18 kernel is from 2018 and you want it to work optimally with recent kernel VIRTIO drivers support.
I don't think it is a good expectation.
Do not use virtio drivers, take full devices visualization drivers and try again with both OS, we will see ?
Click to expand...
I'm going to try not using the Virtio drivers, I don't know how to do it, but I'll look into it.


Thanks to everyone for your answers and recommendations!
I'll keep you updated!
 
No, not the elrepo for debian, the self-compiled version from source ... which may take a while before testing ... :)
 
You must log in or register to reply here.
Top Bottom