Backup of VM fails

[mps000]

Hi,

that's a different error message, so might be a different issue Or do you also have a huge disk? Please share the output of pveversion -v, qm config 108 and the full backup task log. Is the guest still responsive after such a failure?

Good point, I'll open another thread if necessary. Logs attached anyway.

Qemu agent is disabled because I wanted to see if that helped - it doesn't.

VM 108 becomes unresponsive to gui shell and remote ssh in. Won't even ping.

pveversion -v proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve) pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15) proxmox-kernel-helper: 8.0.9 pve-kernel-6.2: 8.0.5 pve-kernel-5.15: 7.4-4 proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4 proxmox-kernel-6.5: 6.5.11-4 proxmox-kernel-6.2.16-19-pve: 6.2.16-19 proxmox-kernel-6.2: 6.2.16-19 proxmox-kernel-6.2.16-15-pve: 6.2.16-15 proxmox-kernel-6.2.16-12-pve: 6.2.16-12 proxmox-kernel-6.2.16-10-pve: 6.2.16-10 pve-kernel-6.2.16-5-pve: 6.2.16-6 pve-kernel-5.15.108-1-pve: 5.15.108-1 pve-kernel-5.15.107-2-pve: 5.15.107-2 pve-kernel-5.15.102-1-pve: 5.15.102-1 pve-kernel-5.15.30-2-pve: 5.15.30-3 ceph-fuse: 16.2.11+ds-2 corosync: 3.1.7-pve3 criu: 3.17.1-2 glusterfs-client: 10.3-5 ifupdown2: 3.2.0-1+pmx7 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-4 libknet1: 1.28-pve1 libproxmox-acme-perl: 1.5.0 libproxmox-backup-qemu0: 1.4.0 libproxmox-rs-perl: 0.3.1 libpve-access-control: 8.0.7 libpve-apiclient-perl: 3.3.1 libpve-common-perl: 8.1.0 libpve-guest-common-perl: 5.0.6 libpve-http-server-perl: 5.0.5 libpve-network-perl: 0.9.4 libpve-rs-perl: 0.8.7 libpve-storage-perl: 8.0.5 libspice-server1: 0.15.1-1 lvm2: 2.03.16-2 lxc-pve: 5.0.2-4 lxcfs: 5.0.3-pve3 novnc-pve: 1.4.0-3 proxmox-backup-client: 3.0.4-1 proxmox-backup-file-restore: 3.0.4-1 proxmox-kernel-helper: 8.0.9 proxmox-mail-forward: 0.2.2 proxmox-mini-journalreader: 1.4.0 proxmox-offline-mirror-helper: 0.6.2 proxmox-widget-toolkit: 4.1.3 pve-cluster: 8.0.5 pve-container: 5.0.8 pve-docs: 8.1.3 pve-edk2-firmware: 4.2023.08-1 pve-firewall: 5.0.3 pve-firmware: 3.9-1 pve-ha-manager: 4.0.3 pve-i18n: 3.1.2 pve-qemu-kvm: 8.1.2-4 pve-xtermjs: 5.3.0-2 qemu-server: 8.0.10 smartmontools: 7.3-pve1 spiceterm: 3.3.0 swtpm: 0.8.0+pve1 vncterm: 1.8.0 zfsutils-linux: 2.2.0-pve3 root@proxmox3:~# qm config 108 agent: 0 bios: ovmf boot: order=scsi0 cores: 4 cpu: host efidisk0: microserver:108/vm-108-disk-1.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K memory: 4000 meta: creation-qemu=7.2.0,ctime=1686132496 name: Airflow net0: virtio=7E:74:B5:23:86:CA,bridge=vmbr0,firewall=1 numa: 1 ostype: l26 sata0: microserver:108/vm-108-cloudinit.qcow2,media=cdrom,size=4M scsi0: microserver:108/vm-108-disk-0.raw,aio=native,iothread=1,size=90G scsihw: virtio-scsi-single smbios1: uuid=4589adc2-bc40-468a-ac54-67e9c0aad37c sockets: 1 unused0: microserver:108/vm-108-disk-0.qcow2 vmgenid: de80651d-7eed-4013-beeb-e14990216635

Backup log
INFO: starting new backup job: vzdump 108 --remove 0 --mode snapshot --node proxmox3 --storage PMBackup --notes-template '{{guestname}}' INFO: Starting Backup of VM 108 (qemu) INFO: Backup started at 2023-11-24 10:25:47 INFO: status = running INFO: VM Name: Airflow INFO: include disk 'scsi0' 'microserver:108/vm-108-disk-0.raw' 90G INFO: include disk 'efidisk0' 'microserver:108/vm-108-disk-1.qcow2' 528K INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: skip unused drive 'microserver:108/vm-108-disk-0.qcow2' (not included into backup) INFO: creating Proxmox Backup Server archive 'vm/108/2023-11-24T10:25:47Z' INFO: enabling encryption INFO: started backup task 'd9ce12f9-860b-4b61-9c97-6c2dcdcbdaa7' INFO: resuming VM again ERROR: VM 108 qmp command 'cont' failed - got timeout INFO: aborting backup job

 

[fiona]

Don't see anything special in the configuration. What kind of storage is microserver?

VM 108 becomes unresponsive to gui shell and remote ssh in. Won't even ping.

In that case it might be an issue with the QEMU process itself. Please install debugger and debug symbols with
apt install gdb pve-qemu-kvm-dbgsym libproxmox-backup-qemu0-dbgsym and then, when the VM is stuck after the backup attempt, run

gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/108.pid) &> /tmp/backup-stuck-debug.log

to obtain the debug traces and share the resulting file.

 

[kode]

It's an AMD 16 core with 128GB memory and dual 10-gbit and full SSD on both sides. There is no high load on either cluster/backup server. I altered the time a couple of times already, the last backup now worked but that could have been a lucky shot as it sometimes did work.

I added the config file of the VM:

Spoiler: qm config

agent: 1,fstrim_cloned_disks=1
boot: order=scsi0
cores: 4
ide2: cephfs:iso/debian-11.2.0-amd64-netinst.iso,media=cdrom
memory: 24576
meta: creation-qemu=6.2.0,ctime=1651594341
name: *******
net0: virtio=66:8A:6F:EF:1E:0B,bridge=vmbr0,firewall=1,tag=****
numa: 0
ostype: l26
scsi0: rbd:vm-152-disk-0,discard=on,iothread=1,size=100G
scsi1: rbd:vm-152-disk-1,discard=on,iothread=1,size=50000G
scsihw: virtio-scsi-pci
smbios1: uuid=62c305e9-3a6b-48be-9522-8989768fc765
sockets: 1
vmgenid: 98ae1659-d43e-4998-85f0-89aed7868710

Is there any way to increase the backup time? I can't seem to find anything concerning that.

If the backup fails again I'll create a bug report.

 

[fiona]

Is there any way to increase the backup time? I can't seem to find anything concerning that.

Unfortunately not. We might consider setting it based on the disk-size or making it user-configurable. Or maybe there's a way to speed up the setup phase.

If the backup fails again I'll create a bug report.

Thanks!

 

[mps000]

Don't see anything special in the configuration. What kind of storage is microserver?


In that case it might be an issue with the QEMU process itself. Please install debugger and debug symbols with
apt install gdb pve-qemu-kvm-dbgsym libproxmox-backup-qemu0-dbgsym and then, when the VM is stuck after the backup attempt, run

gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/108.pid) &> /tmp/backup-stuck-debug.log

to obtain the debug traces and share the resulting file.

Hi Fiona. I have removed some unused discs in order to allow live migration and the error message / issue has changed so please regard this sub-thread as closed. Thanks for your help.