Hello everyone, I have a very irritating issue happening on my windows VM that was recently migrated from vSphere.
I am running PVE 8.3
VMs randomly freezes and stop responding.
I have attempted all the fixes that are out there, installed intel microcode, validated that VMware tools are removed, changed storage adapter.
this issue happens on different hosts so it is not pinned to hardware, on the logs I cannot see anything related to the VM before the freeze or any indicator that there is a HW issue.
following is the strace of a VM while frozen and configuration sample.
# strace -c -p $(cat /var/run/qemu-server/<VMID>.pid)
strace: Process 395308 attached
^Cstrace: Process 395308 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
79.29 0.231242 47 4877 ppoll
8.44 0.024618 2 8625 write
6.02 0.017564 3 4841 162 read
4.71 0.013738 3 4120 recvmsg
0.85 0.002483 6 403 12 futex
0.39 0.001139 2 415 ioctl
0.17 0.000509 6 80 sendmsg
0.05 0.000151 9 16 accept4
0.03 0.000102 6 16 close
0.02 0.000067 2 32 fcntl
0.01 0.000037 2 16 getsockname
------ ----------- ----------- --------- --------- ----------------
100.00 0.291650 12 23441 174 total
qm config <VMID>
agent: 1,freeze-fs-on-backup=0
bios: seabios
boot: order=scsi0
cores: 2
cpu: host
machine: pc-i440fx-8.1
memory: 16384
meta: creation-qemu=8.1.5,ctime=1730621104
name: arrl-vpebap1
net0: e1000e=00:50:56:bb:9c:05,bridge=VLAN503,firewall=1
numa: 1
ostype: win10
sata0: none,media=cdrom
scsi0: <NFS-Storage>:<VMID>/vm-<VMID>-disk-0.qcow2,iothread=1,size=170G
scsihw: virtio-scsi-single
smbios1: uuid=423b3c1e-bb21-c1c3-e425-1d431da0cb9b
sockets: 2
vmgenid: 385e4b1a-6ee9-4405-8294-8c52d40fa2d1
()
proxmox-ve: 8.3.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.3.2 (running version: 8.3.2/3e76eec21c4a14a7)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-5
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20241112.1~deb11u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1
I am running PVE 8.3
VMs randomly freezes and stop responding.
I have attempted all the fixes that are out there, installed intel microcode, validated that VMware tools are removed, changed storage adapter.
this issue happens on different hosts so it is not pinned to hardware, on the logs I cannot see anything related to the VM before the freeze or any indicator that there is a HW issue.
following is the strace of a VM while frozen and configuration sample.
# strace -c -p $(cat /var/run/qemu-server/<VMID>.pid)
strace: Process 395308 attached
^Cstrace: Process 395308 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
79.29 0.231242 47 4877 ppoll
8.44 0.024618 2 8625 write
6.02 0.017564 3 4841 162 read
4.71 0.013738 3 4120 recvmsg
0.85 0.002483 6 403 12 futex
0.39 0.001139 2 415 ioctl
0.17 0.000509 6 80 sendmsg
0.05 0.000151 9 16 accept4
0.03 0.000102 6 16 close
0.02 0.000067 2 32 fcntl
0.01 0.000037 2 16 getsockname
------ ----------- ----------- --------- --------- ----------------
100.00 0.291650 12 23441 174 total
qm config <VMID>
agent: 1,freeze-fs-on-backup=0
bios: seabios
boot: order=scsi0
cores: 2
cpu: host
machine: pc-i440fx-8.1
memory: 16384
meta: creation-qemu=8.1.5,ctime=1730621104
name: arrl-vpebap1
net0: e1000e=00:50:56:bb:9c:05,bridge=VLAN503,firewall=1
numa: 1
ostype: win10
sata0: none,media=cdrom
scsi0: <NFS-Storage>:<VMID>/vm-<VMID>-disk-0.qcow2,iothread=1,size=170G
scsihw: virtio-scsi-single
smbios1: uuid=423b3c1e-bb21-c1c3-e425-1d431da0cb9b
sockets: 2
vmgenid: 385e4b1a-6ee9-4405-8294-8c52d40fa2d1
()
proxmox-ve: 8.3.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.3.2 (running version: 8.3.2/3e76eec21c4a14a7)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-5
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20241112.1~deb11u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1