Tbh, I am not sure here at all. I guessed, that you are on PVE 8.1 with the 6.5 kernel being older than:
6.5.11-5, but you are still on (an old version of) PVE 7.4.
The thing that lead me to my assumption, was this message inside your VM:
rcu_sched self-detected stall on CPU, which came up (again) recently with the (older/first versions of the) 6.5 kernel.
To begin with, I would recommend to update your PVE-host (and optimally your guests too) in any case.
For the PVE-host, you need to add the:
bullseye (
!)
pve-no-subscription repository: [1], if you do not have an active subscription, to your e.g.:
/etc/apt/sources.list and disable the:
pve-enterprise one by commenting it out (add:
# in front of it) in:
/etc/apt/sources.list.d/pve-enterprise.list.
Then run:
apt update followed by:
apt full-upgrade.
Afterwards, a reboot of the PVE-host is needed for the new kernel to take effect.
(
After your PVE 7 is up-to-date, you could also consider to upgrade to PVE 8: [2].)
Another observation is your IO-delay; it is very high. Is/was this ever the case or only recently? Is it ever that high or only when a/the guest(s) have the problem?
What storage type are you using? What exactly is the underlaying storage hardware?
The config of an affected VM could be useful:
qm config VMID (replace
VMID with the actual ID of the/an affected VM).
Unfortunately, I do not know how to troubleshoot this any further, sorry.
[1]
https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_7_x_repositories
[2]
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
