[SOLVED] High IO on load

[kuzco201]

I've been running Proxmox about 2 years now and I'm always struggling with high IO issues. Due to conficting information online I've tried a ton of different settings and although it has been better then it was it's still not great. So I thought I would write this post and ask for some help.

I'm running Proxmox on an old Dell R610 with 2 Xeon E5649's, 56Gb registered ECC RAM and Proxmox is installed on 2 Samsung 1Tb QVO SSD's running in RAID 1 behind a Hardware RAID card. The VMs are also running on these SSDs. The system is using ZFS as FS and I added a 1 Tb PCIE caching ssd a year ago that helped a bit with the High IO. The caching ssd is partitioned in 200Gb to use as Log disk for ZFS and 800 Gb to use as Cache disk for ZFS.

All VMs are using QCOW2 images with a VIRTIO SCSI single and caching enabled. Most VMs are using either writethrough or writeback caching with a few using directsync because this was supposed to be safer, and I don't want to risk losing data on for example my email server. And all VMs have IO thread and SSD emulation enabled.

I've attached an image with the stats while building an application in my CI/CD (Jenkins/Sonarqube). The IO delay is usually around the 3-6% but can spike up to 8% on a daily basis and when running an actuall workload like the CI/CD it spikes up to 20-30% and when booting a few VMs at the same time it can totally freeze the system and spike upto 70-80%.

Any help or feedback would be helpfull since I'm no expert on this. Thanks in advance.

 

[Dunuin]

Search the forum for Samsung QVOs. The SSDs are basically crap and shouldn't be used in a server because QLC NAND got a terrible write performance as soon as the write cache (RAM and SLC) gets full. Other people complaining about bad performance are happy with great performance after replacing the QVOs with something else that uses TLC or MLC NAND. For sequential writes even 5400 RPM HDDs are faster than a smaller Samsung QVO.

 

[kuzco201]

Search the forum for Samsung QVOs. The SSDs are basically crap and shouldn't be used in a server because QLC NAND got a terrible write performance as soon as the write cache (RAM and SLC) gets full. Other people complaining about bad performance are happy with great performance after replacing the QVOs with something else that uses TLC or MLC NAND. For sequential writes even 5400 RPM HDDs are faster than a smaller Samsung QVO.

Damn that's a shame bought them because they were affordable SSDs. Guess I'll just deal with it for now until I can swap them for something better. BTW are a few harddisks in raid 5 better than 2 SSDs in raid 1 performance wise?

 

[Dunuin]

Damn that's a shame bought them because they were affordable SSDs. Guess I'll just deal with it for now until I can swap them for something better. BTW are a few harddisks in raid 5 better than 2 SSDs in raid 1 performance wise?

HDDs, especially in raid5/6, are always terrible as a VM storage compared to SSDs. I only would use that as a cold storage and then bring it into your VMs by SMB/NFS.

With SSDs its basically "buy cheap and you buy twice". I also learned that the hard way and needed to replaced my 6 consumer SSDs with 8 second hand enterprise SSDs (same price but better performance and should be able to handle 30 times the writes before dying) because my consumer SSD life expectation would be reached in under a year. Consumer hardware just isn't build to handle server workloads.

 

[janssensm]

Yep, walked the same path and I'm now too running used enterprise ssd's that will last for years.
But you should check your config too.
Running ZFS with a raid card between FS and disks is not recommended, this is also statet in the pve admin guide.
And you seem to be running cow on cow with qcow2 on ZFS. It is possible, nice to have for toubleshooting and testing (snapshot hierarchy), but not for production, better to choose one or the other.

 

[Dunuin]

Yep, walked the same path and I'm now too running used enterprise ssd's that will last for years.
But you should check your config too.
Running ZFS with a raid card between FS and disks is not recommended, this is also statet in the pve admin guide.
And you seem to be running cow on cow with qcow2 on ZFS. It is possible, nice to have for toubleshooting and testing (snapshot hierarchy), but not for production, better to choose one or the other.

Jep, see here why ZFS shouldn't run ontop of HW raid cards: https://openzfs.github.io/openzfs-docs/Performance and Tuning/Hardware.html#hardware-raid-controllers

And a CoW filesystem ontop of CoW filesystem multiplies your overhead and you get terrible performance and SSD wear. I would stick with HW raid and LVM-thin without qcow2 or ditch that HW raid controller, get a HBA and then use ZFS with RAW block devices and not with qcow2 on file level.

 

[kuzco201]

Jep, see here why ZFS shouldn't run ontop of HW raid cards: https://openzfs.github.io/openzfs-docs/Performance and Tuning/Hardware.html#hardware-raid-controllers

And a CoW filesystem ontop of CoW filesystem multiplies your overhead and you get terrible performance and SSD wear. I would stick with HW raid and LVM-thin without qcow2 or ditch that HW raid controller, get a HBA and then use ZFS with RAW block devices and not with qcow2 on file level.

Ah I see, heard it before but didn't think it would impact performance that much. Then I have 2 follow up questions. Wouldn't not using qcow2 prevent me from using snapshots on running vms as a backup method? And would running the RAID card in IT mode and using only ZFS cause more RAM usage? Because you know ZFS is RAM hungry, or wouldn't it make a difference because I'm using ZFS either way. Now I see qcow partly stands for Copy on write just like ZFS makes sense. But changing back to EXT4 and LVM-thin would require a complete re-install of Proxmox right?

Edit: Now that I think of it so would changing the HW card to IT mode and using ZFS because I would have to delete the current RAID and configure ZFS.

 

[Dunuin]

Ah I see, heard it before but didn't think it would impact performance that much. Then I have 2 follow up questions. Wouldn't not using qcow2 prevent me from using snapshots on running vms as a backup method?

ZFS supports snapshots and snapshot mode backups as well. You don't need qcow2 for that. And LVM-Thin supports snapshots too.

And would running the RAID card in IT mode and using only ZFS cause more RAM usage? Because you know ZFS is RAM hungry, or wouldn't it make a difference because I'm using ZFS either way.

ZFS uses alot of RAM but I don't see why running ZFS ontop of HW raid should use less RAM than just running ZFS ontop of a HBA (or IT mode raid controller).

Now I see qcow partly stands for Copy on write just like ZFS makes sense. But changing back to EXT4 and LVM-thin would require a complete re-install of Proxmox right?

You can move virtual disks between storages and PVE will convert between qcow2 and raw.

 

[kuzco201]

ZFS supports snapshots and snapshot mode backups as well. You don't need qcow2 for that. And LVM-Thin supports snapshots too.

ZFS uses alot of RAM but I don't see why running ZFS ontop of HW raid should use less RAM than just running ZFS ontop of a HBA (or IT mode raid controller).

You can move virtual disks between storages and PVE will convert between qcow2 and raw.

Ah oke, got a lot to think of thanks. So best thing I can do right now is just convert all the vm disks to RAW? I know I can just move between storages but I only have 1 storage pool and that's the ZFS and it's also the boot drive so I don't think I can just change it to LVM-thin.