Backup failed - VM not running; TASK ERROR: job errors

[whytf]

Hello,

I have a VM with qcow2 image but when i try to back it up i get this error below, i already tried qemu-img check with no errors, after that i tried to destroy and restore vm using an older backup that worked but with same error, only relevant thing that i remember is that i updated proxmox (package versions attached below) like 1 hour ago because backing up this vm always worked without a problem with this config.

Thanks for help.

INFO: starting new backup job: vzdump 980 --remove 0 --storage Nasko --notes-template '{{guestname}}' --mode stop --compress zstd --node Carmen
INFO: Starting Backup of VM 980 (qemu)
INFO: Backup started at 2022-06-09 15:33:49
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: storj
INFO: include disk 'scsi0' 'local:980/vm-980-disk-0.qcow2' 32G
INFO: exclude disk 'scsi2' '/dev/disk/by-id/ata-ST31000333AS_6TE07S11' (backup=no)
INFO: exclude disk 'scsi3' '/dev/disk/by-id/ata-ST1000LM014-1EJ164_W381Y0K1' (backup=no)
INFO: exclude disk 'scsi4' '/dev/disk/by-id/ata-TOSHIBA_MQ01ABD100_X4GKPKXAT' (backup=no)
INFO: exclude disk 'scsi5' '/dev/disk/by-id/ata-WDC_WD1003FZEX-00MK2A0_WD-WMC3F0813056' (backup=no)
INFO: creating vzdump archive '/mnt/pve/Nasko/dump/vzdump-qemu-980-2022_06_09-15_33_49.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '4459dc13-1b5a-4302-a588-36d6467f00cd'
INFO:   2% (913.0 MiB of 32.0 GiB) in 3s, read: 304.3 MiB/s, write: 140.4 MiB/s
INFO:   4% (1.4 GiB of 32.0 GiB) in 6s, read: 159.7 MiB/s, write: 133.8 MiB/s
INFO:   8% (2.8 GiB of 32.0 GiB) in 9s, read: 482.1 MiB/s, write: 116.8 MiB/s
INFO:  11% (3.8 GiB of 32.0 GiB) in 12s, read: 342.0 MiB/s, write: 124.4 MiB/s
INFO:  24% (7.8 GiB of 32.0 GiB) in 15s, read: 1.3 GiB/s, write: 39.4 MiB/s
INFO:  44% (14.1 GiB of 32.0 GiB) in 18s, read: 2.1 GiB/s, write: 5.9 MiB/s
INFO:  56% (18.2 GiB of 32.0 GiB) in 21s, read: 1.4 GiB/s, write: 91.7 MiB/s
INFO:  70% (22.7 GiB of 32.0 GiB) in 24s, read: 1.5 GiB/s, write: 71.7 MiB/s
INFO:  87% (28.1 GiB of 32.0 GiB) in 27s, read: 1.8 GiB/s, write: 41.2 MiB/s
ERROR: VM 980 not running
INFO: aborting backup job
ERROR: VM 980 not running
VM 980 not running
trying to acquire lock...
 OK
ERROR: Backup of VM 980 failed - VM 980 not running
INFO: Failed at 2022-06-09 15:34:20
INFO: Backup job finished with errors
TASK ERROR: job errors

VM config:

agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 4
cpu: host,flags=+pcid;+aes
ide2: none,media=cdrom
memory: 24576
meta: creation-qemu=6.2.0,ctime=1650378494
name: storj
net0: virtio=AE:8B:51:96:EE:AA,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local:980/vm-980-disk-0.qcow2,cache=directsync,discard=on,iothread=1,size=32G,ssd=1
scsi2: /dev/disk/by-id/ata-ST31000333AS_6TE07S11,size=976762584K,backup=0,cache=directsync,iothread=1
scsi3: /dev/disk/by-id/ata-ST1000LM014-1EJ164_W381Y0K1,size=976762584K,backup=0,cache=directsync,iothread=1
scsi4: /dev/disk/by-id/ata-TOSHIBA_MQ01ABD100_X4GKPKXAT,size=976762584K,backup=0,cache=directsync,iothread=1
scsi5: /dev/disk/by-id/ata-WDC_WD1003FZEX-00MK2A0_WD-WMC3F0813056,size=976762584K,backup=0,cache=directsync,iothread=1
scsihw: virtio-scsi-single
smbios1: uuid=dee0c5fe-cd30-4b0b-b0d4-0c6a4e5521d0
sockets: 1
vmgenid: 1ae3cf45-3638-49f7-b0d6-0add848d43da

Package versions:

proxmox-ve: 7.2-1 (running kernel: 5.15.35-2-pve)
pve-manager: 7.2-4 (running version: 7.2-4/ca9d43cc)
pve-kernel-5.15: 7.2-4
pve-kernel-helper: 7.2-4
pve-kernel-5.15.35-2-pve: 5.15.35-5
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-2
libpve-storage-perl: 7.2-4
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-9
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[sterzy]

Hi,
your issue might be related to the this bug report: https://bugzilla.proxmox.com/show_bug.cgi?id=4101

A fix is already being worked on and has been submitted to the mailing list: https://lists.proxmox.com/pipermail/pve-devel/2022-June/053243.html

For now you could try reverting to QEMU 6.2.0-8.

 

[whytf]

Hi,
your issue might be realted to the this bug report: https://bugzilla.proxmox.com/show_bug.cgi?id=4101

A fix is already being worked on and has been submitted to the mailing list: https://lists.proxmox.com/pipermail/pve-devel/2022-June/053243.html

For now you could try reverting to QEMU 6.2.0-8.

Thanks, i'll wait for a update as only one vm is affected.

 

[fabian]

fixed package (pve-qemu-kvm 6.2.0-10) available on pvetest, and pve-no-subscription shortly.

 

[Dunuin]

Also ran into that problem tonight. Thanks for the fast fix.

Should I delete the PBS backup snapshots created by that job, because they might be corrupted or should they be fine and the VMs just weren't able to start after the backup was done?

 

[fabian]

if the backup was finalized on PBS (it's not marked as in-progress) it should be fine, and the crash happened after the backup was done when removing the job inside Qemu.

 

[Dunuin]

@fabian:
VMs are still not starting after an backup with pve-qemu-kvm updated to 6.2.0-10. Will now do a reboot, maybe the VM processes were still using the pve-qemu-kvm 6.2.0-9. Don't remember if I stopped all VMs or rebooted the node after updating it.

Edit: After reboot backup was working again.

 

[fabian]

yes, such fixes always require a fresh qemu process, either via stop and start, or via live migration (if the issue doesn't affect the migration on the source side).