Windows guest slow after pve 6 to 7 upgrade

Ymac

New Member
Jul 24, 2021
6
1
3
24
Hi!
I've been using PVE for about a year on my DL380G8, it has been running great with a few guests - one of them is a Windows Server 2019 (as a AD DS DC)
It worked great so far, but after I upgraded my host to PVE 7 successfully (following the official guide) - the windows guest became extremely sluggish and slow and shows very high CPU usage.
No modifications to the guest have been made, software nor virtual hardware.
For storage I use two SSD's in ZFS mirror.

I'm really struggling to debug this problem,
Thanks!
 
  • Like
Reactions: adrian.catalin
hi,

the windows guest became extremely sluggish and slow and shows very high CPU usage.
how does this look in windows task manager?

can you also post your VM configuration here? qm config VMID
 
  • Like
Reactions: Ymac
hi,


how does this look in windows task manager?

can you also post your VM configuration here? qm config VMID
root@REDACTED-pve:/home/REDACTED# qm config 106 agent: 1 bootdisk: scsi0 cores: 4 cpu: host,flags=+aes description: REDACTED ide0: local:iso/virtio-win-0.1.185.iso,media=cdrom,size=402812K memory: 8192 name: REDACTED net0: virtio=REDACTED,bridge=vmbr0,firewall=1 numa: 1 onboot: 1 ostype: win10 parent: Activated scsi0: local-zfs:vm-106-disk-0,discard=on,size=80G scsihw: virtio-scsi-pci smbios1: uuid=8f312a18-4ec9-4e59-be74-86878ad3d833 sockets: 2 vmgenid: bb8d2c28-539c-436b-a783-deb0ed2dc804
As for slowness - only logging in using RDP took me about two minutes.
1628258201046.png
This is after logging in
1628258271825.png
This is just idleing doing nothing. - keep in mind I allocated 8 cores and 8GB of ram which is plenty!
Disk latency is about 3-10ms which is fine
 
97.5% CPU used by system interrupts is really not normal. is this the only "process" using the CPU? if you watch the task manager for a while do you see anything else interesting?


are you having similar problems in other VMs as well or is this the only one with this issue?

also can you check htop on the host as well? what is the CPU load on the host while the VM is reporting such usage?
 
This is "idleing" at 30% cpu (doing nothing)
When I RDP to it, it spikes to 90% CPU in the VM - and the host load avarage spikes as well to 3.00.
This is so weird because this setup worked flawlessly in the past.
The VM is VERY unresponsive and slow.
1628844247198.png
 
Got another interesting screenshot, avarage VM cpu usage: (I upgraded on the 23rd of July)
1628960102915.png
 
Can confirm I have similar issues, and there appear to be a few more threads on this subject.

https://forum.proxmox.com/threads/v...-proxmox-upgrade-from-pve6-4-to-pve7-0.92301/

Has anyone found a workaround?
Actually, I have a few days ago.
The “Core isolation: memory integrity” feature in “Windows Security” was the cause of the performance dip.
Once disabled the machine returned to work quickly and flawlessly.

I can speculate that one of the VirtuIO drivers doesn’t play well with VBS and HVCI and some interrupt handling isn’t performed efficiently (for example, if idle the CPU was ~20%, when quickly moving the mouse cursor it could spike up to 40%!)
HID input is an interrupt, so that’s my lead.

But I’m no kernel dev, so who knows.
 
The memory integrity feature was already disabled, so nothing I can do there.

I actually don't have too many issues with system interrupts, they show up occasionally but not constantly. CPU load is just 30% to 40% higher than expected all the time. But some tasks are worse than others, especially video decoding seems to be impacted.

Going back to kernel 5.11 helped a little, but it's far from satisfactory.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!