Windows guest slow after pve 6 to 7 upgrade

[Ymac]

Hi!
I've been using PVE for about a year on my DL380G8, it has been running great with a few guests - one of them is a Windows Server 2019 (as a AD DS DC)
It worked great so far, but after I upgraded my host to PVE 7 successfully (following the official guide) - the windows guest became extremely sluggish and slow and shows very high CPU usage.
No modifications to the guest have been made, software nor virtual hardware.
For storage I use two SSD's in ZFS mirror.

I'm really struggling to debug this problem,
Thanks!

 

[oguz]

hi,

the windows guest became extremely sluggish and slow and shows very high CPU usage.

how does this look in windows task manager?

can you also post your VM configuration here? qm config VMID

 

[Ymac]

hi,


how does this look in windows task manager?

can you also post your VM configuration here? qm config VMID

root@REDACTED-pve:/home/REDACTED# qm config 106 agent: 1 bootdisk: scsi0 cores: 4 cpu: host,flags=+aes description: REDACTED ide0: local:iso/virtio-win-0.1.185.iso,media=cdrom,size=402812K memory: 8192 name: REDACTED net0: virtio=REDACTED,bridge=vmbr0,firewall=1 numa: 1 onboot: 1 ostype: win10 parent: Activated scsi0: local-zfs:vm-106-disk-0,discard=on,size=80G scsihw: virtio-scsi-pci smbios1: uuid=8f312a18-4ec9-4e59-be74-86878ad3d833 sockets: 2 vmgenid: bb8d2c28-539c-436b-a783-deb0ed2dc804
As for slowness - only logging in using RDP took me about two minutes.

This is after logging in

This is just idleing doing nothing. - keep in mind I allocated 8 cores and 8GB of ram which is plenty!
Disk latency is about 3-10ms which is fine

 

[oguz]

97.5% CPU used by system interrupts is really not normal. is this the only "process" using the CPU? if you watch the task manager for a while do you see anything else interesting?


are you having similar problems in other VMs as well or is this the only one with this issue?

also can you check htop on the host as well? what is the CPU load on the host while the VM is reporting such usage?

 

[Ymac]

This is "idleing" at 30% cpu (doing nothing)
When I RDP to it, it spikes to 90% CPU in the VM - and the host load avarage spikes as well to 3.00.
This is so weird because this setup worked flawlessly in the past.
The VM is VERY unresponsive and slow.

 

[Ymac]

Got another interesting screenshot, avarage VM cpu usage: (I upgraded on the 23rd of July)

 

[denos]

Possibly related (this comment and the one immediately after it):
https://forum.proxmox.com/threads/proxmox-ve-7-0-released.92007/post-402308

 

[nullify]

Can confirm I have similar issues, and there appear to be a few more threads on this subject.

https://forum.proxmox.com/threads/v...-proxmox-upgrade-from-pve6-4-to-pve7-0.92301/

Has anyone found a workaround?

 

[Ymac]

Can confirm I have similar issues, and there appear to be a few more threads on this subject.

https://forum.proxmox.com/threads/v...-proxmox-upgrade-from-pve6-4-to-pve7-0.92301/

Has anyone found a workaround?

Actually, I have a few days ago.
The “Core isolation: memory integrity” feature in “Windows Security” was the cause of the performance dip.
Once disabled the machine returned to work quickly and flawlessly.

I can speculate that one of the VirtuIO drivers doesn’t play well with VBS and HVCI and some interrupt handling isn’t performed efficiently (for example, if idle the CPU was ~20%, when quickly moving the mouse cursor it could spike up to 40%!)
HID input is an interrupt, so that’s my lead.

But I’m no kernel dev, so who knows.

 

[nullify]

The memory integrity feature was already disabled, so nothing I can do there.

I actually don't have too many issues with system interrupts, they show up occasionally but not constantly. CPU load is just 30% to 40% higher than expected all the time. But some tasks are worse than others, especially video decoding seems to be impacted.

Going back to kernel 5.11 helped a little, but it's far from satisfactory.

 

[nullify]

Okay, good news. I just updated to the recently released pve-kernel-5.15.

It completely fixed the issue for me, CPU load seems to have dropped by 50-75%. Seems like the scheduler changes that were introduced in 5.11 had some bugs.

It's a shame that PVE 7.1 released with 5.13, but you can get 5.15 from the standard repositories.
https://forum.proxmox.com/threads/opt-in-linux-kernel-5-15-for-proxmox-ve-7-x-available.100936/