Disk Speeds inside a VM.

[rofo69]

This is my issue.

My proxmox host has 2 SSD's. One is for the host itself, and the other is for the virtual Disks for the VMs and containers.
If I run hdparm or dd directly on the host, I get speeds on the VM SSD disk of around 370-390 MB/s, which is what I would expect.

If I go into a VM (doesnt matter if its windows or Ubuntu), the write speeds drop to 60-70MB/s with caching off, and around 110-130MB/s with write back enabled.
And with write back enabled, I get file transfers pausing for a few seconds when copying large files (6-8 GB).

So why is the write speed so slow when comparing performance from the host itself, and from within a VM ?

 

[waltar]

Select for disk interface virtio block.

 

[rofo69]

I am already set as follows:-

 

[waltar]

Best do io on second disk in vm but could be defined on first also, virtio block is selected after virtio scsi single :

 

[rofo69]

Where do I find the setting for virtIO Block if the VM has already been created ?

 

[_gabriel]

zfs with consumer grade ?

 

[rofo69]

As for virtio block, it seems thats being deprecated according to this thread:-

https://forum.proxmox.com/threads/changing-scsi-controller-disk-controller.147495/

 

[rofo69]

Also, if it was the file system type slowing it down, why does it race along at normal speeds when testing from the host, but its slow in the guest?
I have other VM's running, but not doing huge amounts (usually mostly idle), but its possible there is contention on the drive, but the speed seems ridiculously slow.

Same thing happens inside an unpriv LXC too, speeds are slow.

 

[waltar]

As mentioned create second disk for your vm or lxc, choose virtio scsi single and virtio block, use it as block inside for eg a db or do a filesystem onto and bench that second disk for io.

 

[rofo69]

@waltar - I created a second disk on a windows VM and mounted it as drive D, using the settings above. The speeds were exactly the same (very slow).

 

[waltar]

Windows is anyway slow by design ...

 

[rofo69]

Same happens in ubuntu and a debian LXC.
I cant work out why the ssd disk is normal speed on the host but much much slower in the guest.

 

[waltar]

host raid6 5x2T, hdparm -t 300MB/s, Debian vm 2nd disk 620MB/s. Anyway hdparm is not a good benchmark program.

 

[Monstrous1704]

Literally came here to post same thing, my setup follows basically every guide on here for best performance, was seeing about 10x drop in performance, I tweaked some stuff; turned off swap inside vms, WCE=1 for the SAS drives (Dell Enterprise SAS SSD's).

Running FIO on ext4 on Enterprise Dell 800gb SSD:

fio --filename=test --sync=1 --rw=randwrite --bs=4k --numjobs=1   --iodepth=4 --group_reporting --name=test --filesize=10G --runtime=300 --end_fsync=1 && rm test
Jobs: 1 (f=1): [w(1)][100.0%][w=10.5MiB/s][w=2686 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=805485: Tue Nov 19 17:24:47 2024
  write: IOPS=2651, BW=10.4MiB/s (10.9MB/s)(3107MiB/300001msec); 0 zone resets
    clat (usec): min=290, max=8581, avg=373.00, stdev=65.36
     lat (usec): min=290, max=8582, avg=373.52, stdev=65.40
    clat percentiles (usec):
     |  1.00th=[  310],  5.00th=[  318], 10.00th=[  322], 20.00th=[  334],
     | 30.00th=[  351], 40.00th=[  371], 50.00th=[  375], 60.00th=[  379],
     | 70.00th=[  388], 80.00th=[  396], 90.00th=[  408], 95.00th=[  429],
     | 99.00th=[  519], 99.50th=[  562], 99.90th=[  734], 99.95th=[ 1713],
     | 99.99th=[ 2114]
   bw (  KiB/s): min= 8608, max=11536, per=100.00%, avg=10613.51, stdev=320.74, samples=599
   iops        : min= 2152, max= 2884, avg=2653.30, stdev=80.17, samples=599
  lat (usec)   : 500=98.65%, 750=1.26%, 1000=0.01%
  lat (msec)   : 2=0.08%, 4=0.01%, 10=0.01%
  cpu          : usr=2.20%, sys=19.41%, ctx=1593348, majf=4, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,795426,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=10.4MiB/s (10.9MB/s), 10.4MiB/s-10.4MiB/s (10.9MB/s-10.9MB/s), io=3107MiB (3258MB), run=300001-300001msec

Disk stats (read/write):
  sdc: ios=0/2388596, merge=0/1649210, ticks=0/165880, in_queue=215757, util=56.29%

Running on VM using LVM-Thin in ubuntu (ext4), virtio scsi single, io thread, no cache, vm io scheduler = none.

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=4
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=1944KiB/s][w=486 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=12612: Wed Nov 20 00:28:53 2024
  write: IOPS=472, BW=1891KiB/s (1936kB/s)(554MiB/300003msec); 0 zone resets
    clat (usec): min=1210, max=10589, avg=2105.48, stdev=469.40
     lat (usec): min=1211, max=10590, avg=2106.37, stdev=469.41
    clat percentiles (usec):
     |  1.00th=[ 1418],  5.00th=[ 1549], 10.00th=[ 1647], 20.00th=[ 1745],
     | 30.00th=[ 1827], 40.00th=[ 1909], 50.00th=[ 1991], 60.00th=[ 2089],
     | 70.00th=[ 2180], 80.00th=[ 2343], 90.00th=[ 2933], 95.00th=[ 3064],
     | 99.00th=[ 3294], 99.50th=[ 3458], 99.90th=[ 4490], 99.95th=[ 4817],
     | 99.99th=[ 7242]
   bw (  KiB/s): min= 1636, max= 2344, per=100.00%, avg=1892.80, stdev=106.39, samples=599
   iops        : min=  409, max=  586, avg=472.97, stdev=26.64, samples=599
  lat (msec)   : 2=50.22%, 4=49.56%, 10=0.22%, 20=0.01%
  cpu          : usr=1.16%, sys=7.86%, ctx=283772, majf=1, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,141826,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=1891KiB/s (1936kB/s), 1891KiB/s-1891KiB/s (1936kB/s-1936kB/s), io=554MiB (581MB), run=300003-300003msec

Disk stats (read/write):
  sda: ios=0/427824, merge=0/296242, ticks=0/226079, in_queue=311400, util=100.00%

2686 -> 472 IOPS is quite the drop? About 5x slower...

Obviously 4k is the worst case but running 16/64k blocks runs about the same 5x drop between metal to virtual. As the other person stated, doesn't seem to matter if it's ubuntu/deb/fedora VM or LXC.

Also looking at hardware, barely tickling the CPU and Disk, below 1% IO Wait times. Am I missing something really obvious?





Complete shot in the dark, but did all the microcode patches for the older CPU's completely nerf VM performance? These are broadwell CPU's

 

[zvangi]

Im experiencing a similar thing here. I have few Proxmox VE installations with different NVMe and HDD's.

In one example Im getting 1 GB/s for I/O speed using bench.sh both in host and vm. Using Raid0 in this configuration (no zfs). These are 3.84 TB NVMe drives (They are listed as 2-3GB/s drives). These NVMe's should have much faster read or write speeds compared to what im getting especially in Raid0. Their model is SAMSUNG MZQLB3T8HALS-00007

In second example Im getting 5 or 6 GB/s I/O on host but only getting 1GB/s on vm using ubuntu. Im using VirtIO SCSI single on this second example and host is a zfs mirror I think. My drives are 2x990 PRO 2TB.

Any guess?

Thanks.

 

[Monstrous1704]

So I tried a virtio-blk for the primary disk, there was an improvement for sure, 472 -> 540 IOPS, nothing to snub your nose at, but miles away from the expected ~10% virt overhead drop on older hardware. (Metal is 2686)

 

[waltar]

With above fio cmd host 309iops, vm 231iops so it's about 75%, which is quiet good.
In general I would expect performance loss of a vm 10% for cpu, 20% for network and 50% on I/O, for a lxc slightly less loss, because of the additional layers and therefore increased latency. A bare metal installation is always the fastest and with virtualization you got some kind of ha but at decreased performance.

 

[waltar]

When choosing nfs (normally as used for all our vm/lxc) host 539iops, vm 420iops so it's about 80% even better while nfs results are still stable regardless the I/O size up to very big which is not really the case with local storage (vm small I/O data amount quiet good but with increased data and load becomes worse).

 

[rofo69]

There is definitely a relationship between the underlying speed of the ssd and the apparent speed in the VM.
Attaching a disk to a VM, backed by a sata SSD with a read/write speed of around 370 MB/s generates a speed inside the VM of around 60/70MB/s.
This is using the same settings as @Monstrous1704:-

Running on VM using LVM-Thin in ubuntu (ext4), virtio scsi single, io thread, no cache

So temporarily, I allow the proxmox host OS to be used as storage for VMs too and created and attached a disk to a VM. This is on a different SSD that runs at 490 MB/s on the host, and in proxmox that produced very different results. Initially it was very very slow, but towards the end of the file copy (8gig media file), it really shot up to around 300MB/s, so now I am more than confused. I'll do some more testing.

 

[Monstrous1704]

With above fio cmd host 309iops, vm 231iops so it's about 75%, which is quiet good.
In general I would expect performance loss of a vm 10% for cpu, 20% for network and 50% on I/O, for a lxc slightly less loss, because of the additional layers and therefore increased latency. A bare metal installation is always the fastest and with virtualization you got some kind of ha but at decreased performance.

I would love 50% IO performance!

Based on my numbers I'm lucky to get 25%, it was worse before I tweaked things. I'm kinda lost at this point, I had a unraid server running qcow images on consumer ssds which ran better than these higher end enterprise sas drives and better hardware!

I would love to be able to work out where/what the bottle neck is, so I at least know what to look for