Freezing of VMs.

[nikst1m]

Hello!

After restoring VM from a backup (from PBS) / migration VM / cloning VM, all virtual machines on the host to which the restoration is being restored hang. It is possible to stop the virtual machine with the stop signal, but then it fails to start with the "timeout waiting on systemd" error. Proxmox VE 7.2-11. What could be the reason?

 

[Moayad]

Hi,

What could be the reason?

Hard to say without syslog. Can you please check the syslog/journalctl during the migration or cloning?

 

[nikst1m]

Hi,


Hard to say without syslog. Can you please check the syslog/journalctl during the migration or cloning?

Aug 15 22:09:02 v12 systemd[2816991]: Created slice User Application Slice.
Aug 15 22:09:02 v12 systemd[2816991]: Reached target Paths.
Aug 15 22:09:02 v12 systemd[2816991]: Reached target Timers.
Aug 15 22:09:02 v12 systemd[2816991]: Listening on GnuPG network certificate management daemon.
Aug 15 22:09:02 v12 systemd[2816991]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Aug 15 22:09:02 v12 systemd[2816991]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Aug 15 22:09:02 v12 systemd[2816991]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Aug 15 22:09:02 v12 systemd[2816991]: Listening on GnuPG cryptographic agent and passphrase cache.
Aug 15 22:09:02 v12 systemd[2816991]: Reached target Sockets.
Aug 15 22:09:02 v12 systemd[2816991]: Reached target Basic System.
Aug 15 22:09:02 v12 systemd[2816991]: Reached target Main User Target.
Aug 15 22:09:02 v12 systemd[2816991]: Startup finished in 145ms.
Aug 15 22:09:02 v12 systemd[1]: Started User Manager for UID 0.
Aug 15 22:09:02 v12 systemd[1]: Started Session 178 of user root.
Aug 15 22:09:02 v12 systemd[1]: session-178.scope: Succeeded.
Aug 15 22:09:02 v12 systemd[1]: Started Session 180 of user root.
Aug 15 22:09:03 v12 systemd[1]: session-180.scope: Succeeded.


Aug 15 22:11:34 v12 pvedaemon[2648488]: <root@pam> starting task UPID:v12:002B03FF:033EA467:64DBA3B6:qmshutdown:140:root@pam:
Aug 15 22:11:34 v12 pvedaemon[2819071]: shutdown VM 140: UPID:v12:002B03FF:033EA467:64DBA3B6:qmshutdown:140:root@pam:
Aug 15 22:11:37 v12 systemd[1]: Starting Daily apt download activities...
Aug 15 22:11:38 v12 systemd[1]: apt-daily.service: Succeeded.
Aug 15 22:11:38 v12 systemd[1]: Finished Daily apt download activities.
Aug 15 22:11:42 v12 pvedaemon[1374969]: VM 140 qmp command failed - VM 140 qmp command 'guest-ping' failed - got timeout
Aug 15 22:11:43 v12 pvestatd[1291]: status update time (6.752 seconds)
Aug 15 22:11:45 v12 QEMU[1998]: kvm: terminating on signal 15 from pid 954 (/usr/sbin/qmeventd)
Aug 15 22:11:45 v12 kernel: [544379.077390] fwbr140i0: port 2(tap140i0) entered disabled state
Aug 15 22:11:45 v12 kernel: [544379.116314] fwbr140i0: port 1(fwln140i0) entered disabled state
Aug 15 22:11:45 v12 kernel: [544379.116417] vmbr1: port 3(fwpr140p0) entered disabled state
Aug 15 22:11:45 v12 kernel: [544379.117003] device fwln140i0 left promiscuous mode
Aug 15 22:11:45 v12 kernel: [544379.117006] fwbr140i0: port 1(fwln140i0) entered disabled state
Aug 15 22:11:45 v12 kernel: [544379.155408] device fwpr140p0 left promiscuous mode
Aug 15 22:11:45 v12 kernel: [544379.155412] vmbr1: port 3(fwpr140p0) entered disabled state
Aug 15 22:11:46 v12 kernel: [544379.680700] fwbr140i1: port 2(tap140i1) entered disabled state
Aug 15 22:11:46 v12 kernel: [544379.732394] fwbr140i1: port 1(fwln140i1) entered disabled state
Aug 15 22:11:46 v12 kernel: [544379.732482] vmbr1: port 4(fwpr140p1) entered disabled state
Aug 15 22:11:46 v12 kernel: [544379.733089] device fwln140i1 left promiscuous mode
Aug 15 22:11:46 v12 kernel: [544379.733092] fwbr140i1: port 1(fwln140i1) entered disabled state
Aug 15 22:11:46 v12 kernel: [544379.767590] device fwpr140p1 left promiscuous mode
Aug 15 22:11:46 v12 kernel: [544379.767598] vmbr1: port 4(fwpr140p1) entered disabled state
Aug 15 22:11:46 v12 pvedaemon[2648488]: <root@pam> end task UPID:v12:002B03FF:033EA467:64DBA3B6:qmshutdown:140:root@pam: OK
Aug 15 22:11:46 v12 systemd[1]: 140.scope: Succeeded.
Aug 15 22:11:46 v12 systemd[1]: 140.scope: Consumed 12h 21min 53.448s CPU time.
Aug 15 22:17:44 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - got timeout
Aug 15 22:17:56 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:18:09 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:18:21 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:18:34 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:18:47 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:18:59 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:19:12 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:19:18 v12 pvestatd[1291]: status update time (12.609 seconds)
Aug 15 22:19:24 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:19:37 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:19:50 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:20:02 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries
Aug 15 22:20:15 v12 pvestatd[1291]: VM 156 qmp command failed - VM 156 qmp command 'query-proxmox-support' failed - unable to connect to VM 156 qmp socket - timeout after 31 retries

 

[Moayad]

Can you please post the output of `pveversion -v` as well?

 

[nikst1m]

Can you please post the output of `pveversion -v` as well?

proxmox-ve: 7.2-1 (running kernel: 5.15.60-2-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-helper: 7.2-13
pve-kernel-5.15: 7.2-12
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.60-2-pve: 5.15.60-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-3
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-4
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

 

[Moayad]

Hello,

Thank you for the output!

First, I would upgrade the Proxmox VE to the latest available version, since there are a lot of patches (apt update && apt dist-upgrade).

From the output of the syslog, I see that the VM mentioned is `156` - if yes, can you please also post the VM config (qm config 156)?

 

[nikst1m]

qm config 156:

agent: 1
boot: order=scsi0
cores: 4
memory: 8000
name: Zabbix
net0: virtio=221:29:22:10:F8,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
scsi0: local-lvm:vm-156-disk-0,size=50G
scsihw: pvscsi
smbios1: uuid=236fd088-44da-4b33-9559-90f2678c5880
sockets: 2
vmgenid: 286d0f83-612b-41e8-8c7a-b3d981e77c75



But in fact, all the virtual machines that were on the host hung.