Disk Speeds inside a VM.

[Monstrous1704]

- Changed from LVM-Thin to LVM, now getting 694 IOPS on the guest, So getting close to 25% of metal.
- Then moved the disk to EXT4 storage, now getting 767 IOPS on the guest, ~28% of metal.
- Also tried the disk on XFS storage, get about the same 766 on the guest.
- Last thing I tried, turned off mitigations on the pve-host (still enabled on guest), saw about 900 IOPS, still not 50% but double where I started...

So I guess what we'd all like to know, how the heck are people getting 80%-50% IO performance in their VM guests? What are we all doing wrong? As I see so many threads/tickets complaining about the same thing...

So far these seem to get me better performance:
- LVM not LVM-Thin (doesn't thin provision will use more disk, snapshots are larger?)
- Use Virtio-blk with IO Thread, Discard
- Use raw image (I think default for LVM anyway)
- Make sure using Host CPU, Make sure NUMA is selected and matching socket count (if your host has that)
- Make sure you set WCE=1 for your drives (make ent SAS drives tho!)
- Set mitigations=off - not for prod or any sort of system with untrusted actors, make your own assessment for risk.

 

[waltar]

- Use Virtio-blk with IO Thread, Discard

Don't use discard, trim weekly (or daily ifwant) by cron - that kill's performance !!

- Make sure using Host CPU, Make sure NUMA is selected and matching socket count (if your host has that)

We mostly use default, select just "host" if trying some nested virtualization simulations.

- LVM not LVM-Thin (doesn't thin provision will use more disk, snapshots are larger?)

We use xfs or nfs on xfs without lvm, test some zvol/zfs. Tune xfs on raidset and raid-ctrl, mkfs, mount, /proc and /sys, tune nfs mount if using.
Where is this setting ?

 

[Monstrous1704]

Don't use discard, trim weekly (or daily ifwant) by cron - that kill's performance !!

Interesting, just tried with discard off, made no difference outside of margin of error

 

[waltar]

Maybe no difference in vm/lxc but don't use on host where filesystem resides on.