Proxmox VE - IO and CPU Stalls

[niagaro]

Hi,
Brand new servers with PVE 5 installed. We started pushing more and more load to the servers and we are experiencing:
* IO and CPU stalls on our monitoring station (Centos 7) even when load reported by the machine is under 1
* IO stalls around 1am on our SQL Server (Windows 2019) which sometimes impact the log files and corrupts it and blocks the backups....
* IO Stalls on other servers where they need an hour to just do a simple reboot whether windows or linux.
Am still looking on the hardware side of things but:
* ZFS which is underlying the VM disks reports no errors, same for disk SMARTs

So, where can I gather more information in PVE about stalls and performance statistics (What is the default location for those log files)
And does anyone ever had that issue and knows how they fixed it?
Thanks

 

[wolfgang]

With our information, it is hard to say anything.
But known Problems what can do this are

• Use of a Raid Controller or a Controller with an extra Cache
• If the swap partition is on the rpool where the disk image lay on.
• Your disk is to slow for the load.
• Use of a slow write cache.

One thing If you say the IO load is at 1 in the VM.
But what is the IO delay on the Host?

 

[niagaro]

Hi,
Thanks for the information, I do have raid controllers but they are configured as passthrue and fed into the ZFS layer.
The swap partition is on an alternate disks on the servers. They are setup as follow:
2 disks - mirrored with the controller - OS
4 disks - 2 satas, 2 ssds, mirrored with ZFS into 2 pools: SATA Pool and SSD pool
2 sockets with 16 cores each, 128 GB memory and had 14 VMs (which is a medium amount of VMs)

All VMs use writethrough

IOs reported by atop are < 1ms and atop only report the disks to be busy 30 to 50% of the time with outliers at 120% and 5% once in a while

I've currently moved the load to an alternate server which has the same hardware and proxmox configuration to help figure out if the hardware itself might be an issue.

Also, can't seem to google this, but what are the fundamental differences between SATA driver and VirtIO driver? From the testing I did, SATA driver works better than virtio when underlying disks are SATA (50% more speed), but VirtIO works better on SSDs.

But does the SATA performance come at a cost of bad performance when a lot of VMs are present? Is it bad at paralleling requests from multiple VMs and thus, VirtIO should be used when you have lots of VMs regardless of the hard disk types?

 

[wolfgang]

I do have raid controllers but they are configured as passthrue and fed into the ZFS layer.

This is a problem in most cases.

Also, can't seem to google this, but what are the fundamental differences between SATA driver and VirtIO driver? From the testing I did, SATA driver works better than virtio when underlying disks are SATA (50% more speed), but VirtIO works better on SSDs.

If this is the case, then you have a somewhere in the IO stack a problem.
SATA is software emulated an Vitrtio paravitualiziert.
SATA is only for legacy or migration reasons available.

 

[sb-jw]

I do have raid controllers but they are configured as passthrue and fed into the ZFS layer.

What controller do you have? Normally it's better to flash the Controller in IT Mode. For example the H310 with IT FW has a queue Depth of 600 instead of 25.

More Infos: http://www.yellow-bricks.com/2014/06/09/queue-depth-matters/

I would recommend to use Hardware for ZFS and CEPH which you will find on the VMware VSAN HCL. If this hardware is working very well in Enterprise environments with an Shared Storage, they should work good with alternative solutions too. But yes, not all is perfect for use with CEPH or ZFS, but it's help to choice Hardware for such an setup.