qemu-server 8.4.1 fails to reboot Debian VM

B

bagginses

New Member
May 3, 2025
8
4
3
I have a problem where whenever I try to reboot a VM I get an error on startup "VM 101 qmp command failed - VM 101 qmp command 'guest-ping' failed - got timeout". I have tried this with several Debian VMs and gotten this error on all of them. This problem was supposedly fixed with qemu-server 8.4.1 (https://forum.proxmox.com/threads/qemu-server-8-3-14-prevents-vm-shutdown.168309) but I am still having this issue with 8.4.1.

I have verified that the guest-agent is enabled in options and that the qemu-guest-agent package is installed and running within the VMs.
Code: 
sudo systemctl status qemu-guest-agent
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static)
     Active: active (running) since Mon 2025-07-28 14:03:28 UTC; 31min ago
   Main PID: 353 (qemu-ga)
      Tasks: 2 (limit: 9479)
     Memory: 1.6M
        CPU: 112ms
     CGroup: /system.slice/qemu-guest-agent.service
             └─353 /usr/sbin/qemu-ga

Jul 28 14:33:22 dockge-2 qemu-ga[353]: info: guest-ping called
Jul 28 14:33:33 dockge-2 qemu-ga[353]: info: guest-ping called
Jul 28 14:33:44 dockge-2 qemu-ga[353]: info: guest-ping called

I have also checked that these pve services are running. Enabling these services is supposedly what fixed the issue for the user in the forum post attached above.
Code: 
systemctl status qmeventd.service pve-query-machine-capabilities.service
● qmeventd.service - PVE Qemu Event Daemon
     Loaded: loaded (/lib/systemd/system/qmeventd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-07-28 08:55:21 CDT; 38min ago
    Process: 4075406 ExecStart=/usr/sbin/qmeventd /var/run/qmeventd.sock (code=exited, status=0/SUCCESS)
   Main PID: 4075408 (qmeventd)
      Tasks: 1 (limit: 154357)
     Memory: 316.0K
        CPU: 3.007s
     CGroup: /system.slice/qmeventd.service
             └─4075408 /usr/sbin/qmeventd /var/run/qmeventd.sock

Jul 28 09:03:29 pve1 qmeventd[4081680]: Restarting VM 101
Jul 28 09:03:29 pve1 qm[4081680]: <root@pam> starting task UPID:pve1:003E4814:02C7BC1C:68878331:qmstart:101:root@pam:
Jul 28 09:03:29 pve1 qm[4081684]: start VM 101: UPID:pve1:003E4814:02C7BC1C:68878331:qmstart:101:root@pam:
Jul 28 09:03:31 pve1 qm[4081684]: VM 101 started with PID 4081733.
Jul 28 09:03:31 pve1 qm[4081680]: <root@pam> end task UPID:pve1:003E4814:02C7BC1C:68878331:qmstart:101:root@pam: OK
Jul 28 09:31:14 pve1 qmeventd[4075408]: read: Connection reset by peer
Jul 28 09:31:15 pve1 qmeventd[4102952]: Starting cleanup for 101
Jul 28 09:31:15 pve1 qmeventd[4102952]: trying to acquire lock...
Jul 28 09:31:15 pve1 qmeventd[4102952]:  OK
Jul 28 09:31:15 pve1 qmeventd[4102952]: Finished cleanup for 101

● pve-query-machine-capabilities.service - PVE Query Machine Capabilities
     Loaded: loaded (/lib/systemd/system/pve-query-machine-capabilities.service; enabled; preset: enabled)
     Active: active (exited) since Mon 2025-07-28 08:30:27 CDT; 1h 3min ago
   Main PID: 4033267 (code=exited, status=0/SUCCESS)
        CPU: 1ms

Jul 28 08:30:27 pve1 systemd[1]: Starting pve-query-machine-capabilities.service - PVE Query Machine Capabilities...
Jul 28 08:30:27 pve1 systemd[1]: Finished pve-query-machine-capabilities.service - PVE Query Machine Capabilities.

Even after this, rebooting a VM from the webui results in the error mentioned. The VM seems to start as I can see CPU and RAM usage spin up to normal amounts but I can't ssh in or even open the console from the pve interface. In the summary tab next to IPs it just says "Guest Agent not running". The only solution is to completely stop the VM and restart it. Any ideas what is causing this or potential solutions?

System log output during reboot attempt.
Code: 
journalctl -e
Jul 28 09:38:18 pve1 pvedaemon[4075367]: <root@pam> end task UPID:pve1:003EAAA4:02CABDFC:68878AE4:vncproxy:101:root@pam: OK
Jul 28 09:38:44 pve1 pvedaemon[4075368]: <root@pam> starting task UPID:pve1:003EB1B8:02CAF60F:68878B74:qmreboot:101:root@pam:
Jul 28 09:38:44 pve1 pvedaemon[4108728]: requesting reboot of VM 101: UPID:pve1:003EB1B8:02CAF60F:68878B74:qmreboot:101:root@pam:
Jul 28 09:38:47 pve1 kernel:  zd112: p1 p2 p3
Jul 28 09:38:47 pve1 kernel: tap101i0: left allmulticast mode
Jul 28 09:38:47 pve1 kernel: vmbr0: port 8(tap101i0) entered disabled state
Jul 28 09:38:47 pve1 qmeventd[4075408]: read: Connection reset by peer
Jul 28 09:38:47 pve1 virtiofsd[4103315]: Client disconnected, shutting down
Jul 28 09:38:47 pve1 virtiofsd[4103309]: Client disconnected, shutting down
Jul 28 09:38:47 pve1 pvedaemon[4075368]: <root@pam> end task UPID:pve1:003EB1B8:02CAF60F:68878B74:qmreboot:101:root@pam: OK
Jul 28 09:38:47 pve1 systemd[1]: 101.scope: Deactivated successfully.
Jul 28 09:38:47 pve1 systemd[1]: 101.scope: Consumed 59.468s CPU time.
Jul 28 09:38:48 pve1 qmeventd[4108822]: Starting cleanup for 101
Jul 28 09:38:48 pve1 qmeventd[4108822]: Finished cleanup for 101
Jul 28 09:38:48 pve1 qmeventd[4108822]: Restarting VM 101
Jul 28 09:38:48 pve1 qm[4108822]: <root@pam> starting task UPID:pve1:003EB220:02CAF7AF:68878B78:qmstart:101:root@pam:
Jul 28 09:38:48 pve1 qm[4108832]: start VM 101: UPID:pve1:003EB220:02CAF7AF:68878B78:qmstart:101:root@pam:
Jul 28 09:38:48 pve1 systemd[1]: Started 101.scope.
Jul 28 09:38:48 pve1 virtiofsd[4108842]: Waiting for vhost-user socket connection...
Jul 28 09:38:48 pve1 virtiofsd[4108847]: Waiting for vhost-user socket connection...
Jul 28 09:38:48 pve1 virtiofsd[4108842]: Client connected, servicing requests
Jul 28 09:38:48 pve1 virtiofsd[4108847]: Client connected, servicing requests
Jul 28 09:38:49 pve1 kernel: tap101i0: entered promiscuous mode
Jul 28 09:38:49 pve1 kernel: vmbr0: port 8(tap101i0) entered blocking state
Jul 28 09:38:49 pve1 kernel: vmbr0: port 8(tap101i0) entered disabled state
Jul 28 09:38:49 pve1 kernel: tap101i0: entered allmulticast mode
Jul 28 09:38:49 pve1 kernel: vmbr0: port 8(tap101i0) entered blocking state
Jul 28 09:38:49 pve1 kernel: vmbr0: port 8(tap101i0) entered forwarding state
Jul 28 09:38:49 pve1 qm[4108832]: VM 101 started with PID 4108854.
Jul 28 09:38:49 pve1 qm[4108822]: <root@pam> end task UPID:pve1:003EB220:02CAF7AF:68878B78:qmstart:101:root@pam: OK
Jul 28 09:38:56 pve1 pvedaemon[4075367]: VM 101 qmp command failed - VM 101 qmp command 'guest-ping' failed - got timeout
Jul 28 09:39:15 pve1 pvedaemon[4075368]: VM 101 qmp command failed - VM 101 qmp command 'guest-ping' failed - got timeout
Jul 28 09:39:23 pve1 pveproxy[4075403]: worker exit
Jul 28 09:39:23 pve1 pveproxy[4075401]: worker 4075403 finished
Jul 28 09:39:23 pve1 pveproxy[4075401]: starting 1 worker(s)
Jul 28 09:39:23 pve1 pveproxy[4075401]: worker 4109308 started
Jul 28 09:39:34 pve1 pvedaemon[4075368]: VM 101 qmp command failed - VM 101 qmp command 'guest-ping' failed - got timeout
 
the log looks like the reboot worked, but then afterwards there are a few instances where the guest-ping command fails.. this can happen during the startup of the VM, while the agent is not yet listening for connections for example. do those messages persist after the VM has fully booted?
 
fabian said:
the log looks like the reboot worked, but then afterwards there are a few instances where the guest-ping command fails.. this can happen during the startup of the VM, while the agent is not yet listening for connections for example. do those messages persist after the VM has fully booted?
Click to expand...
Yes those messages persist long after the VM has “booted”. I’m actually not sure if the VM fully boots and the guest-agent just isn’t working after a reboot. The strange thing is that rebooting from within the VM console works fine so it seems to be stemming from the guest agent whatever is happening. Until I force stop and start the VM I have no access to it via console or ssh.
 
Last edited:
please post the VM config and "pveversion -v"

after you reset the VM, is there anything in the VM system logs themselves? what OS are you using in the VM?
 
fabian said:
please post the VM config and "pveversion -v"

after you reset the VM, is there anything in the VM system logs themselves? what OS are you using in the VM?
Click to expand...
Here are the configs, I am running Debian 12 with linux kernel 6.1.0-37-amd64 and qemu-guest-agent version 1:7.2+dfsg-7+deb12u13.
Code: 
root@pve1:~# cat /etc/pve/qemu-server/101.conf
#<div align='center'>
#  <a href='https%3A//Helper-Scripts.com' target='_blank' rel='noopener noreferrer'>
#    <img src='https%3A//raw.githubusercontent.com/community-scripts/ProxmoxVE/main/misc/images/logo-81x112.png' alt='Logo' style='width%3A81px;height%3A112px;'/>
#  </a>
#
#  <h2 style='font-size%3A 24px; margin%3A 20px 0;'>Docker VM</h2>
#
#  <p style='margin%3A 16px 0;'>
#    <a href='https%3A//ko-fi.com/community_scripts' target='_blank' rel='noopener noreferrer'>
#      <img src='https%3A//img.shields.io/badge/&#x2615;-Buy us a coffee-blue' alt='spend Coffee' />
#    </a>
#  </p>
#
#  <span style='margin%3A 0 10px;'>
#    <i class="fa fa-github fa-fw" style="color%3A #f5f5f5;"></i>
#    <a href='https%3A//github.com/community-scripts/ProxmoxVE' target='_blank' rel='noopener noreferrer' style='text-decoration%3A none; color%3A #00617f;'>GitHub</a>
#  </span>
#  <span style='margin%3A 0 10px;'>
#    <i class="fa fa-comments fa-fw" style="color%3A #f5f5f5;"></i>
#    <a href='https%3A//github.com/community-scripts/ProxmoxVE/discussions' target='_blank' rel='noopener noreferrer' style='text-decoration%3A none; color%3A #00617f;'>Discussions</a>
#  </span>
#  <span style='margin%3A 0 10px;'>
#    <i class="fa fa-exclamation-circle fa-fw" style="color%3A #f5f5f5;"></i>
#    <a href='https%3A//github.com/community-scripts/ProxmoxVE/issues' target='_blank' rel='noopener noreferrer' style='text-decoration%3A none; color%3A #00617f;'>Issues</a>
#  </span>
#</div>
agent: enabled=1
bios: ovmf
boot: order=scsi0
cores: 6
efidisk0: flash:vm-101-disk-0,size=4M
localtime: 1
machine: q35
memory: 8192
meta: creation-qemu=9.2.0,ctime=1753678185
name: dockge-2
net0: virtio=02:1B:64:EE:D4:A1,bridge=vmbr0
onboot: 1
ostype: l26
scsi0: flash:vm-101-disk-1,discard=on,size=30G,ssd=1
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=93338215-31bb-44cb-9e16-8cd5795423a5
tablet: 0
tags: community-script
virtiofs0: photo
virtiofs1: immich
vmgenid: bb15fe33-fb52-4d7f-ab41-640ff5f84c06

Code: 
root@pve1:~# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-11-pve)
pve-manager: 8.4.5 (running version: 8.4.5/57892e8e686cb35b)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
ceph-fuse: 17.2.8-pve2
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
frr-pythontools: 10.2.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.2
libpve-cluster-perl: 8.1.2
libpve-common-perl: 8.3.2
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.3-1
proxmox-backup-file-restore: 3.4.3-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.12
pve-cluster: 8.1.2
pve-container: 5.3.0
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-4~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.8-pve1

As far as the VM logs, I noticed several lines pertaining to the network settings saying that the network configuration changed. Here is an example section of the logs, this is the only thing that stood out to me. These messages stopped seemingly once the VM was fully booted.
Code: 
Jul 31 15:37:23 dockge-2 kernel: Initializing XFRM netlink socket
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:24 dockge-2 kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter i>
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:24 dockge-2 systemd-networkd[298]: br-49fb225b0540: Link UP
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:24 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Another example section
Code: 
Jul 31 15:37:25 dockge-2 systemd[1]: Startup finished in 3.381s (kernel) + 5.384s (userspace) = 8.766s.
Jul 31 15:37:26 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:26 dockge-2 systemd-networkd[298]: br-94898c9735e4: Gained IPv6LL
Jul 31 15:37:26 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:26 dockge-2 systemd-networkd[298]: veth66cb67a: Gained IPv6LL
Jul 31 15:37:26 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:26 dockge-2 systemd-networkd[298]: vethcf6e587: Gained IPv6LL
Jul 31 15:37:26 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:27 dockge-2 systemd-networkd[298]: veth81e20db: Gained IPv6LL
Jul 31 15:37:27 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:27 dockge-2 systemd-networkd[298]: veth9f65fa1: Gained IPv6LL
Jul 31 15:37:27 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:27 dockge-2 systemd-networkd[298]: veth93bc74b: Gained IPv6LL
Jul 31 15:37:27 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:27 dockge-2 systemd-networkd[298]: br-49fb225b0540: Gained IPv6LL
Jul 31 15:37:27 dockge-2 systemd-timesyncd[347]: Network configuration changed, trying to establish connection.
Jul 31 15:37:31 dockge-2 qemu-ga[355]: info: guest-ping called
Jul 31 15:37:41 dockge-2 qemu-ga[355]: info: guest-ping called
Jul 31 15:37:51 dockge-2 systemd[1]: systemd-fsckd.service: Deactivated successfully.
Jul 31 15:37:52 dockge-2 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
 
I have some confusing results from an Ubuntu 24.04 server VM as well, I tested rebooting from webui and it worked once no problem. I did get the qemu-guest-agent got timeout error a couple times but it seemed to still start ok. However I tried rebooting it again and it now doesn't work, no console or ssh access again. Here's what's strange - I tried full stopping and starting the VM as I did with the Debian VM but it did not fix the problem. It required a full reboot of the node to fix the issue and get the Ubuntu VM working normal. Now that I am doing this I actually think I've had to do this before. What gives? I have a basically stock proxmox install with the latest version and I can't even get VMs to play nice with something as simple as a reboot? I should note that when I call a reboot from within the VM cli it seems to work just fine.
 
those messages sound benign (you are probably running some hypervisor inside that does network reconfiguration during booting).

it would be great if you could post a full journal until multi-user.target is reached.
 
not really.

maybe you can try the following:

- look at the current timestamp and write it down (let's call this A)
- trigger a failing reboot and write down the current timestamp (let's call this B)
- wait for the reboot sequence to (allegedly) finish and write down the current timestamp (let's call this C)
- stop the VM, write down the current timestamp (D) and cold start the VM
- wait for the VM to boot, and write down the current timestamp (E)

then, run "journalctl --since A --until E" inside the VM and post the full output and all the collected timestamps here.
 
You must log in or register to reply here.
Top Bottom