VM slow after Proxmox PVE upgrade 6.4 > 7.2

[Ales_R]

Hello,
I have a PVE with 3 VM (1x linux; 1x Win2019 with active directory, DHCP, fileserver); 1x Win2019 with remote desktop + company ERP system). The PVE has 32GB of RAM and 2x 6TB HDD in ZFS RAID1. The performance has been stable for months with PVE 6.4.

I made an upgrade of PVE from 6.4 > 7.2 during weekend and we experience a significant decrease in performance of VM, especially the one with Win2019+RDS+ERP where the time to complete tasks in ERP increased 3x - 4x! Running a backup with 7-zip on this VM takes approximately the same time as before upgrade. Running a backup with 7-zip on VM with fileserver takes about 25% longer time as before upgrade.

I attached pictures of PVE load. the peeks in the graphs are daily VM backups. The upgrade took place 30.7.2022 so there is clearly visible significant increase in CPU usage and server load. When I checked the cpu usage of affected VM with ERP it increased from 7% to 21% during our company working ours. The ERP is terrible to work with now.
I already tried and it was without impact: turn on/off balooning, update windows drivers to latest Virtio,

Can anybody help me with it to solve this issue please?

 

[e100]

Did you find any solutions?

I am having similar issue, the only thing in common I've found is ZFS.
On all nodes SSD or HDD IO has increased since upgrading but the HDD nodes the increase is huge.
Where we use ZFS we use RAID10 with 10 or more disks.

All of my nodes have ZFS and some of them also have some LVM storage
On the nodes that are mixed, when backing up from LVM IO Wait is same as in Proxmox 6, when backing up from ZFS IO Wait is significantly higher than it was in Proxmox 6.

My thread about this is here: https://forum.proxmox.com/threads/high-io-wait-during-backups-after-upgrading-to-proxmox-7.113790/

 

[Ales_R]

Unfortunately, I havn't found any solution. Compared to your thread my "slowdown" occurs during all operations, not only backup. The worst is to work with our company ERP - time to complete tasks in ERP increased 3x - 4x (e.g. accessing document from 2 sec to 7 sec etc).

 

[e100]

@Ales_R I've not had chance to try but it was suggested to see if the proxmox 5.11 kernel helps. One of the threads I linked too in my post mentioned 5.11 works well.

So if you can, give that a try and report back your results. If 5.11 works ok that will help us track down the cause of the problem and get it resolved.

Also, I only noticed the issue during backups because that's the only time enough IO happens to cause the issue. If I log into one of my VMs and run IO benchmark tool like fio the io wait increases just like it does with the backup.

The problem is zfs IO is significantly slower for some reason.

 

[Ales_R]

How can I downgrade the kernel to 5.11? I do not have much possibilities to experiment because we use the Proxmox and VMs in production.

 

[RolandK]

@Ales_R , have a look at this thread https://forum.proxmox.com/threads/high-io-wait-during-backups-after-upgrading-to-proxmox-7.113790/

 

[Ales_R]

@RolandK, Thank you for the link. I have stopped watching this thread because I needed to solve it quickly - I bought new HW and migrated all VM finally. I am not experiencing these troubles on the new HW anymore.