Backup of VM failed: unable to connect to qmeventd socket

[Mecanik]

Hi,

Proxmox 7.2.11

INFO: starting new backup job: vzdump 100 --compress zstd --mode snapshot --mailnotification always --notes-template '{{guestname}}' --quiet 1 --storage backups --prune-backups 'keep-last=1' INFO: Starting Backup of VM 100 (qemu) INFO: Backup started at 2022-11-13 22:30:04 INFO: status = running INFO: VM Name: ***** INFO: include disk 'scsi0' 'local:100/vm-100-disk-1.qcow2' 200G INFO: include disk 'efidisk0' 'local:100/vm-100-disk-0.qcow2' 528K INFO: include disk 'tpmstate0' 'local:100/vm-100-disk-2.raw' 4M INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: creating vzdump archive '/mnt/pve/backups/dump/vzdump-qemu-100-2022_11_13-22_30_02.vma.zst' ERROR: Backup of VM 100 failed - unable to connect to qmeventd socket (vmid: 100) - No such file or directory INFO: Failed at 2022-11-13 22:30:12 INFO: Backup job finished with errors TASK ERROR: job errors

Same thing affects Guest shutdown; if you press shutdown from Proxmox UI. Guest tools installed of course.

Thanks

 

[fiona]

Hi,
please post the output of pveversion -v and qm config 100.

EDIT: Please also check systemctl status qmeventd.service and /var/log/syslog from around the time the issue happens.

 

[Mecanik]

Hi,
please post the output of pveversion -v and qm config 100.

EDIT: Please also check systemctl status qmeventd.service and /var/log/syslog from around the time the issue happens.

proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.15.64-1-pve: 5.15.64-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

agent: 1
bios: ovmf
boot: order=scsi0;net0;ide0
cores: 4
cpu: kvm64,flags=+ibpb;+virt-ssbd;+amd-ssbd;+pdpe1gb;+hv-tlbflush;+aes
efidisk0: local:100/vm-100-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hotplug: disk,network,usb
ide0: cdrom,media=cdrom
machine: pc-i440fx-7.0
memory: 8192
meta: creation-qemu=7.0.0,ctime=1667806020
name: **************
net0: virtio=**************,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: local:100/vm-100-disk-1.qcow2,cache=writeback,discard=on,size=200G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=d9d0221c-130f-43d5-8358-882897898dd5
sockets: 1
tpmstate0: local:100/vm-100-disk-2.raw,size=16896,version=v2.0
vcpus: 4
vmgenid: c8f677f0-9f7d-4b4d-acd5-6615af94c4d2

 

[Mecanik]

Hi,
please post the output of pveversion -v and qm config 100.

EDIT: Please also check systemctl status qmeventd.service and /var/log/syslog from around the time the issue happens.

● qmeventd.service - PVE Qemu Event Daemon
     Loaded: loaded (/lib/systemd/system/qmeventd.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

Nov 08 13:32:50 host-5017329 systemd[1]: qmeventd.service: Failed to set invocation ID for unit: File exists
Nov 08 13:32:50 host-5017329 systemd[1]: Failed to start PVE Qemu Event Daemon.

syslog says nothing interesting, same error as in the subject.

 

[fiona]

● qmeventd.service - PVE Qemu Event Daemon
     Loaded: loaded (/lib/systemd/system/qmeventd.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

Nov 08 13:32:50 host-5017329 systemd[1]: qmeventd.service: Failed to set invocation ID for unit: File exists
Nov 08 13:32:50 host-5017329 systemd[1]: Failed to start PVE Qemu Event Daemon.

Well, seems like there is an issue with the service invocation. What is the output of

stat /run/systemd/units/invocation:qmeventd.service
ps aux | grep qmeventd

 

[Mecanik]

Well, seems like there is an issue with the service invocation. What is the output of

stat /run/systemd/units/invocation:qmeventd.service
ps aux | grep qmeventd

stat: cannot statx '/run/systemd/units/invocation:qmeventd.service': No such file or directory

The other command returns a lot of non-formatted data.

 

[fiona]

stat: cannot statx '/run/systemd/units/invocation:qmeventd.service': No such file or directory

Well, that's strange, as it was complaining before that the file (it should be this file AFAICT) already existed. Does running
systemctl daemon-reload and then systemctl start qmeventd.service help? Otherwise maybe try rebooting.

The other command returns a lot of non-formatted data.

Right, every VM's commandline will show up. You can try ps aux | grep qmeventd | grep -v /usr/bin/kvm and check if there is a qmeventd process running.

 

[Mecanik]

Well, that's strange, as it was complaining before that the file (it should be this file AFAICT) already existed. Does running
systemctl daemon-reload and then systemctl start qmeventd.service help? Otherwise maybe try rebooting.


Right, every VM's commandline will show up. You can try ps aux | grep qmeventd | grep -v /usr/bin/kvm and check if there is a qmeventd process running.

Thank you! It seems to have worked:

root@:~# systemctl daemon-reload
root@:~# systemctl start qmeventd.service
root@:~# systemctl status qmeventd.service
● qmeventd.service - PVE Qemu Event Daemon
     Loaded: loaded (/lib/systemd/system/qmeventd.service; enabled; vendor pres>
     Active: active (running) since Tue 2022-11-15 15:39:32 +08; 8s ago
    Process: 2452359 ExecStart=/usr/sbin/qmeventd /var/run/qmeventd.sock (code=>
   Main PID: 2452360 (qmeventd)
      Tasks: 1 (limit: 38352)
     Memory: 296.0K
        CPU: 1ms
     CGroup: /system.slice/qmeventd.service
             └─2452360 /usr/sbin/qmeventd /var/run/qmeventd.sock

Nov 15 15:39:32 host-5017329 systemd[1]: Starting PVE Qemu Event Daemon...
Nov 15 15:39:32 host-5017329 systemd[1]: Started PVE Qemu Event Daemon.

Unfortunately I cannot reboot the server, running production code...

I will test backups now

 

[Shadow Sysop]

I can verify the working solution, as I experienced the same issue as OP and Fiona's solution, systemctl daemon-reload and then systemctl start qmeventd.service, on the host fixed it for me.

 

[wireless]

Hi, someone has also found a strange hibernation behaviour with qmeventd active? When It on, there is no way to let hibernation work (I've opened a thread here cause situation seems a little more complicated). However, without qemeventd vm shutdown does not work.
Question 2: is there a way to make permanent active qmeventd after reboot? It is in failed status but if started it's ok.

 

[SaltyMango_]

Well, that's strange, as it was complaining before that the file (it should be this file AFAICT) already existed. Does running
systemctl daemon-reload and then systemctl start qmeventd.service help? Otherwise maybe try rebooting.


Right, every VM's commandline will show up. You can try ps aux | grep qmeventd | grep -v /usr/bin/kvm and check if there is a qmeventd process running.

Thanks! Starting the service worked for me.