PVE7 / PBS2 - Backup Timeout (qmp command 'cont' failed - got timeout)

iprigger

iprigger

Renowned Member
Proxmox Subscriber
Sep 5, 2009
190
41
93
earth!
All,

I did update from pve6.4 to pve7.0 yesterday and seem to have some problems with backups...

Code: 
INFO: starting new backup job: vzdump 733 --mode snapshot --remove 0 --storage zz_PBSVVIRT --node torfajoekull
INFO: Starting Backup of VM 733 (qemu)
INFO: Backup started at 2021-08-27 09:58:30
INFO: status = running
INFO: VM Name: cirrus
INFO: include disk 'virtio0' 'esnx-a-core:733/vm-733-disk-0.raw' 12G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/733/2021-08-27T07:58:30Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'd92a4985-e257-4bed-8021-a3896f926dc5'
INFO: resuming VM again
ERROR: VM 733 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 733 failed - VM 733 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-08-27 09:58:37
INFO: Backup job finished with errors
TASK ERROR: job errors

we run on nfs storage and have the pbs server as a VM - which did not cause any troubles in the past but does now.

The error above can be resolved when deleting all old backups but comes back after the 4th or so backup...

It seems to be a bit worse with raw images - but on several tries qcow did fail, too.

any idea?
Tobias

P.S: Just one more info: The "timeout" is instant - so no real delay... looks a bit like something is amiss...
 
Last edited:
Moayad said:
Have you tried on other storage target?

Please post the VM config and the PVE version
Click to expand...
Hi,

I only have one pbs volume (4tb), so can't really test....

VM config
Code: 
agent: 1
boot: cdn
bootdisk: virtio0
cores: 4
cpu: kvm64,flags=+aes
cpuunits: 10240
ide2: none,media=cdrom
memory: 8192
name: cirrus
net0: virtio=76:D2:BA:A5:72:A3,bridge=vmbr0,tag=10
numa: 0
onboot: 1
ostype: l26
smbios1: uuid=8581516c-be13-42a6-9f95-c60b578fc41f
sockets: 1
tablet: 0
virtio0: esnx-a-core:733/vm-733-disk-0.raw,size=12G

PVE-Version:
Code: 
proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.4: 6.4-5
pve-kernel-5.11.22-3-pve: 5.11.22-6
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-4.15: 5.4-8
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.2.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-5
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.8-1
proxmox-backup-file-restore: 2.0.8-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

Thanks
Tobias
 
Hi All,

I have found some sort of mitigation to the problem... not a real solution but it seems to help:

1) I did still have some NFS Links on NFSv3 - which seems to be suboptimal with proxmox 7 (looks a bit like some changes in the network stack of the underpinnings do react different)

2) I throtteled the backup a bit in order to have less load during backup.

Seems to have helped...

Still would be cool if I could revert the config to what I did have in the past.

Tobias
 
Hi

I've exactly the same issue since I upgraded to 7. The backup fails every few days on random VMs and then some VMs change the root file system to read-only afterward. Really annoying...

Just for the documentation, here is my output of two different cases:
Code: 
INFO: VM Name: UServer23
INFO: include disk 'virtio0' 'local:103/vm-103-disk-1.qcow2' 50000M
INFO: include disk 'virtio1' 'local:103/vm-103-disk-0.qcow2' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox_daily/dump/vzdump-qemu-103-2021_09_21-00_11_55.vma.lzo'
INFO: started backup task 'd8cab9d3-a4fb-4482-b9ed-da4d4409026d'
INFO: resuming VM again
ERROR: VM 103 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 103 failed - VM 103 qmp command 'cont' failed - got timeout



INFO: VM Name: UServer22
INFO: include disk 'virtio0' 'local:106/vm-106-disk-1.qcow2' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox_daily/dump/vzdump-qemu-106-2021_09_24-00_44_45.vma.lzo'
INFO: started backup task 'd83e45d6-b044-4198-a31e-c5623f3589b4'
INFO: resuming VM again
ERROR: VM 106 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 106 failed - VM 106 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-09-24 00:44:51

I'm going to install the debug package and will let you know when I know more

Regards
Mathias
 
Hi @Moayad

It seems like I'm already using this version :-(

Code: 
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-4-pve: 5.11.22-8
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-4-pve: 4.13.13-35
pve-kernel-4.4.98-3-pve: 4.4.98-103
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-11
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-3
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

I'll now try to install the debug package

Regards
Mathias
 
schm said:
It seems like I'm already using this version :-(
Click to expand...
Have you stop and start the VM? you could also online migrate if the VM is productive. If that not helps, could you try to install the debug package and send the output of the backup task job and the journal during that backup job?
 
Alright, I installed qemu-server-dbgsym, hard restarted all VMs and after a couple of backup iterations, it failed again. Unfortunately, I don't see much more information...

Code: 
INFO: starting new backup job: vzdump --compress lzo --storage proxmox_daily --all 1 --quiet 1 --mode snapshot --mailnotification always
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2021-09-27 00:00:02
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: UClient1
INFO: include disk 'virtio0' 'local:101/vm-101-disk-1.qcow2' 50G
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/proxmox_daily/dump/vzdump-qemu-101-2021_09_27-00_00_02.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task 'b449d824-cae6-4910-adf0-6c6d70209ed3'
INFO:   0% (72.5 MiB of 50.0 GiB) in 3s, read: 24.2 MiB/s, write: 22.9 MiB/s
INFO:   1% (609.0 MiB of 50.0 GiB) in 9s, read: 89.4 MiB/s, write: 78.6 MiB/s
INFO:   2% (1.4 GiB of 50.0 GiB) in 16s, read: 122.2 MiB/s, write: 23.6 MiB/s
INFO:   5% (2.9 GiB of 50.0 GiB) in 19s, read: 495.4 MiB/s, write: 55.7 MiB/s
INFO:   6% (3.1 GiB of 50.0 GiB) in 22s, read: 90.0 MiB/s, write: 89.9 MiB/s
...
INFO:  96% (48.1 GiB of 50.0 GiB) in 11m 20s, read: 105.4 MiB/s, write: 104.9 MiB/s
INFO:  97% (48.6 GiB of 50.0 GiB) in 11m 25s, read: 103.4 MiB/s, write: 97.7 MiB/s
INFO:  98% (49.0 GiB of 50.0 GiB) in 11m 29s, read: 102.4 MiB/s, write: 101.3 MiB/s
INFO:  99% (49.6 GiB of 50.0 GiB) in 11m 35s, read: 99.6 MiB/s, write: 97.4 MiB/s
INFO: 100% (50.0 GiB of 50.0 GiB) in 11m 42s, read: 59.7 MiB/s, write: 59.0 MiB/s
INFO: backup is sparse: 3.90 GiB (7%) total zero data
INFO: transferred 50.00 GiB in 702 seconds (72.9 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 22.70GB
INFO: prune older backups with retention: keep-last=4
INFO: removing backup 'proxmox_daily:backup/vzdump-qemu-101-2021_09_23-00_00_03.vma.lzo'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 101 (00:12:00)
INFO: Backup finished at 2021-09-27 00:12:02
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2021-09-27 00:12:02
INFO: status = running
INFO: VM Name: UServer23
INFO: include disk 'virtio0' 'local:103/vm-103-disk-1.qcow2' 50000M
INFO: include disk 'virtio1' 'local:103/vm-103-disk-0.qcow2' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox_daily/dump/vzdump-qemu-103-2021_09_27-00_12_02.vma.lzo'
INFO: started backup task '8546ff18-feea-4e0e-ad7b-976da2fa0c5f'
INFO: resuming VM again
ERROR: VM 103 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 103 failed - VM 103 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-09-27 00:12:07
INFO: Starting Backup of VM 105 (qemu)
INFO: Backup started at 2021-09-27 00:12:07
INFO: status = running
INFO: VM Name: UServer11
INFO: include disk 'virtio0' 'local:105/vm-105-disk-1.qcow2' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox_daily/dump/vzdump-qemu-105-2021_09_27-00_12_07.vma.lzo'
INFO: started backup task 'e52240b0-63b6-481f-a76f-169b4b0b898e'
INFO: resuming VM again
INFO:   0% (322.6 MiB of 100.0 GiB) in 3s, read: 107.5 MiB/s, write: 96.7 MiB/s
INFO:   1% (1.1 GiB of 100.0 GiB) in 13s, read: 81.1 MiB/s, write: 77.8 MiB/s
INFO:   2% (2.0 GiB of 100.0 GiB) in 21s, read: 118.5 MiB/s, write: 116.7 MiB/s
INFO:   3% (3.0 GiB of 100.0 GiB) in 46s, read: 41.2 MiB/s, write: 41.0 MiB/s
INFO:   4% (4.0 GiB of 100.0 GiB) in 1m, read: 73.5 MiB/s, write: 70.7 MiB/s
INFO:   5% (5.1 GiB of 100.0 GiB) in 1m 16s, read: 67.5 MiB/s, write: 67.4 MiB/s
...
INFO:  93% (93.0 GiB of 100.0 GiB) in 27m 37s, read: 53.6 MiB/s, write: 53.3 MiB/s
INFO:  94% (94.1 GiB of 100.0 GiB) in 27m 53s, read: 66.4 MiB/s, write: 63.1 MiB/s
INFO:  95% (95.0 GiB of 100.0 GiB) in 28m 10s, read: 57.1 MiB/s, write: 56.7 MiB/s
INFO:  96% (96.0 GiB of 100.0 GiB) in 28m 27s, read: 60.5 MiB/s, write: 57.6 MiB/s
INFO:  97% (97.0 GiB of 100.0 GiB) in 28m 46s, read: 53.3 MiB/s, write: 53.1 MiB/s
INFO:  98% (98.1 GiB of 100.0 GiB) in 29m 3s, read: 61.5 MiB/s, write: 59.6 MiB/s
INFO:  99% (99.0 GiB of 100.0 GiB) in 29m 28s, read: 39.9 MiB/s, write: 39.6 MiB/s
INFO: 100% (100.0 GiB of 100.0 GiB) in 29m 49s, read: 47.6 MiB/s, write: 44.3 MiB/s
INFO: backup is sparse: 4.48 GiB (4%) total zero data
INFO: transferred 100.00 GiB in 1789 seconds (57.2 MiB/s)
INFO: archive file size: 79.85GB
INFO: prune older backups with retention: keep-last=4
INFO: removing backup 'proxmox_daily:backup/vzdump-qemu-105-2021_09_23-00_53_50.vma.lzo'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 105 (00:29:59)
INFO: Backup finished at 2021-09-27 00:42:06
INFO: Starting Backup of VM 106 (qemu)
INFO: Backup started at 2021-09-27 00:42:06
INFO: status = running
INFO: VM Name: UServer22
INFO: include disk 'virtio0' 'local:106/vm-106-disk-1.qcow2' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox_daily/dump/vzdump-qemu-106-2021_09_27-00_42_06.vma.lzo'
INFO: started backup task 'e6eadee3-b67e-4314-bf33-abb9bc8459d5'
INFO: resuming VM again
INFO:   0% (300.9 MiB of 100.0 GiB) in 3s, read: 100.3 MiB/s, write: 84.4 MiB/s
INFO:   1% (1.0 GiB of 100.0 GiB) in 12s, read: 85.1 MiB/s, write: 81.6 MiB/s
INFO:   2% (2.0 GiB of 100.0 GiB) in 23s, read: 90.5 MiB/s, write: 90.0 MiB/s
INFO:   3% (3.0 GiB of 100.0 GiB) in 57s, read: 30.1 MiB/s, write: 28.2 MiB/s
INFO:   4% (4.0 GiB of 100.0 GiB) in 1m 18s, read: 50.2 MiB/s, write: 49.4 MiB/s
...
INFO:  93% (93.1 GiB of 100.0 GiB) in 28m 2s, read: 78.5 MiB/s, write: 76.0 MiB/s
INFO:  94% (94.0 GiB of 100.0 GiB) in 28m 15s, read: 70.3 MiB/s, write: 70.3 MiB/s
INFO:  95% (95.1 GiB of 100.0 GiB) in 28m 35s, read: 56.0 MiB/s, write: 54.4 MiB/s
INFO:  96% (96.1 GiB of 100.0 GiB) in 28m 51s, read: 63.2 MiB/s, write: 63.2 MiB/s
INFO:  97% (97.0 GiB of 100.0 GiB) in 29m 6s, read: 63.6 MiB/s, write: 58.2 MiB/s
INFO: 100% (100.0 GiB of 100.0 GiB) in 29m 16s, read: 304.6 MiB/s, write: 59.5 MiB/s
INFO: backup is sparse: 4.90 GiB (4%) total zero data
INFO: transferred 100.00 GiB in 1756 seconds (58.3 MiB/s)
INFO: archive file size: 69.69GB
INFO: prune older backups with retention: keep-last=4
INFO: removing backup 'proxmox_daily:backup/vzdump-qemu-106-2021_09_22-00_43_25.vma.lzo'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 106 (00:29:23)
INFO: Backup finished at 2021-09-27 01:11:29
INFO: Backup job finished with errors

TASK ERROR: job errors


Code: 
Sep 27 00:12:02 hades vzdump[2067050]: INFO: Finished Backup of VM 101 (00:12:00)
Sep 27 00:12:02 hades vzdump[2067050]: INFO: Starting Backup of VM 103 (qemu)
Sep 27 00:12:03 hades systemd[1]: 101.scope: Succeeded.
Sep 27 00:12:03 hades systemd[1]: 101.scope: Consumed 2min 1.980s CPU time.
Sep 27 00:12:07 hades vzdump[2067050]: VM 103 qmp command failed - VM 103 qmp command 'cont' failed - got timeout
Sep 27 00:12:07 hades vzdump[2067050]: ERROR: Backup of VM 103 failed - VM 103 qmp command 'cont' failed - got timeout
Sep 27 00:12:07 hades vzdump[2067050]: INFO: Starting Backup of VM 105 (qemu)
 
same Problem here.
PBS is installed as a Proxmox VM.
Used Storage is NFS mpunted like this.
Code: 
192.168.x.x:/mnt/Daten/pbs /mnt/t70a nfs rw,noac,actimeo=0 0 0

Some machines randomly have Problems with backup
Code: 
2021-09-29 00:11:38 INFO: Starting Backup of VM 102 (qemu)
2021-09-29 00:11:38 INFO: status = running
2021-09-29 00:11:38 INFO: VM Name: xxx
2021-09-29 00:11:38 INFO: include disk 'scsi0' 'sandata2:vm-102-disk-0' 32G
2021-09-29 00:11:38 INFO: backup mode: snapshot
2021-09-29 00:11:38 INFO: ionice priority: 7
2021-09-29 00:11:38 INFO: creating Proxmox Backup Server archive 'vm/102/2021-09-28T22:11:38Z'
2021-09-29 00:11:38 INFO: issuing guest-agent 'fs-freeze' command
2021-09-29 00:13:39 INFO: issuing guest-agent 'fs-thaw' command
2021-09-29 00:13:39 ERROR: VM 102 qmp command 'backup' failed - backup connect failed: command error: http request timed out
2021-09-29 00:13:39 INFO: aborting backup job
2021-09-29 00:13:39 INFO: resuming VM again
2021-09-29 00:13:39 ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: http request timed out
Code: 
2021-09-29 00:19:45 INFO: Starting Backup of VM 108 (qemu)
2021-09-29 00:19:45 INFO: status = running
2021-09-29 00:19:45 INFO: VM Name: xxx
2021-09-29 00:19:45 INFO: include disk 'virtio0' 'sandata2:vm-108-disk-0' 2G
2021-09-29 00:19:45 INFO: include disk 'virtio1' 'sandata2:vm-108-disk-1' 200G
2021-09-29 00:19:46 INFO: backup mode: snapshot
2021-09-29 00:19:46 INFO: ionice priority: 7
2021-09-29 00:19:46 INFO: creating Proxmox Backup Server archive 'vm/108/2021-09-28T22:19:45Z'
2021-09-29 00:21:46 ERROR: VM 108 qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed out
2021-09-29 00:21:46 INFO: aborting backup job
2021-09-29 00:21:46 INFO: resuming VM again
2021-09-29 00:21:46 ERROR: Backup of VM 108 failed - VM 108 qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed out

proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve) pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e) pve-kernel-5.11: 7.0-7 pve-kernel-helper: 7.0-7 pve-kernel-5.11.22-4-pve: 5.11.22-9 pve-kernel-5.11.22-3-pve: 5.11.22-7 pve-kernel-5.11.22-1-pve: 5.11.22-2 ceph-fuse: 15.2.13-pve1 corosync: 3.1.5-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.22-pve1 libproxmox-acme-perl: 1.3.0 libproxmox-backup-qemu0: 1.2.0-1 libpve-access-control: 7.0-4 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-9 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-2 libpve-storage-perl: 7.0-11 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-4 lxcfs: 4.0.8-pve2 novnc-pve: 1.2.0-3 proxmox-backup-client: 2.0.10-1 proxmox-backup-file-restore: 2.0.10-1 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.3-6 pve-cluster: 7.0-3 pve-container: 4.0-9 pve-docs: 7.0-5 pve-edk2-firmware: 3.20200531-1 pve-firewall: 4.2-3 pve-firmware: 3.3-1 pve-ha-manager: 3.3-1 pve-i18n: 2.5-1 pve-qemu-kvm: 6.0.0-4 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-14 smartmontools: 7.2-1 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.0.5-pve1
 
Last edited:
Two days ago I updated to the latest version, rebooted the whole host and after the first backup, one VM failed again. Unfortunately, the log still doesn't tell me something interesting...

While searching the needle in the haystack, I just changed the NFS version from 3 to 4. Let's see whether that changes something. I'll let you know.

Code: 
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-4-pve: 4.13.13-35
pve-kernel-4.4.98-3-pve: 4.4.98-103
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-9
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-11
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.10-1
proxmox-backup-file-restore: 2.0.10-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-3
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-14
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

Code: 
INFO: VM Name: UServer22
INFO: include disk 'virtio0' 'local:106/vm-106-disk-1.qcow2' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox_daily/dump/vzdump-qemu-106-2021_09_29-01_23_17.vma.lzo'
INFO: started backup task 'bb59fe3f-447e-41cb-b962-127dc96347cb'
INFO: resuming VM again
ERROR: VM 106 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 106 failed - VM 106 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-09-29 01:23:29

Code: 
Sep 29 01:23:17 hades vzdump[277454]: INFO: Starting Backup of VM 106 (qemu)
Sep 29 01:23:21 hades vzdump[277454]: VM 106 qmp command failed - VM 106 qmp command 'cont' failed - got timeout
Sep 29 01:23:29 hades vzdump[277454]: ERROR: Backup of VM 106 failed - VM 106 qmp command 'cont' failed - got timeout
Sep 29 01:23:29 hades vzdump[277454]: INFO: Backup job finished with errors
Sep 29 01:23:29 hades vzdump[277454]: job errors
Sep 29 01:23:29 hades vzdump[277450]: <root@pam> end task UPID:hades:00043BCE:009E4230:61539064:vzdump::root@pam: job errors
 
Incidentally, I'm having the same issue, but only on one of my Proxmox clusters.

In my home lab, where the problem exists, I'm running 2 PVE-7 hosts and one host with PBS-1.1 (haven't upgraded yet). The PVE host VM and container storage connect via LVM over iSCSI on 10Gbe. I have about 9 QEMU VMs running on this and a few LXC containers. During backups, I get the occasional "qmp command 'cont' failed - got timeout." Sometimes this results in the guest remounting its filesystem read-only, but not every time.

In my production system, I have 9 PVE-7 hosts using CEPH-RBD storage for VMs and LXC. I have zero issues with the "qmp command 'cont' failed - got timeout" there. Not even one.

Possible storage issue?
 
Just for the protocol, the backup on the latest version failed as well.

Code: 
proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-4-pve: 4.13.13-35
pve-kernel-4.4.98-3-pve: 4.4.98-103
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-10
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-3
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-16
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
 
Hi!
I am also seeing this issue,
Currently every automatic backup fails on a single vm (800gb),
sometimes manual backup also fails, but on-retry it works (the vm and the proxmox node is mostly idle),
while on a smaller vm (100gb) it works every time.
The bigger vm has a storage volume from another pool that is on rotating media, while the majority of the data is on nvme storage.

The error from a manual backup run failing:
Code: 
INFO: starting new backup job: vzdump 103 --node proxmox --mode snapshot --storage pbs --remove 0
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2021-10-18 09:14:34
INFO: status = running
INFO: VM Name: linux-server
INFO: include disk 'sata0' 'local-zfs:vm-103-disk-0' 200G
INFO: include disk 'sata1' 'local-zfs:vm-103-disk-1' 100G
INFO: include disk 'sata2' 'backup-disks:103/vm-103-disk-0.qcow2' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/103/2021-10-18T07:14:34Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '30018634-f18c-4139-87b1-c7fc0423d81b'
INFO: resuming VM again
ERROR: VM 103 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 103 failed - VM 103 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-10-18 09:14:48
INFO: Backup job finished with errors
TASK ERROR: job errors

the journal log of a recent failed automatic backup with debug packages installed:
Code: 
Oct 19 06:30:40 proxmox vzdump[1323934]: INFO: Starting Backup of VM 103 (qemu)
Oct 19 06:30:41 proxmox proxmox-backup-api[3975]: successful auth for user 'root@pam'
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: starting new backup on datastore 'inc-backup': "vm/103/2021-10-19T04:30:40Z"
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: download 'index.json.blob' from previous backup.
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: register chunks in 'drive-sata0.img.fidx' from previous backup.
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: download 'drive-sata0.img.fidx' from previous backup.
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: created new fixed index 1 ("vm/103/2021-10-19T04:30:40Z/drive-sata0.img.fidx")
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: register chunks in 'drive-sata1.img.fidx' from previous backup.
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: download 'drive-sata1.img.fidx' from previous backup.
Oct 19 06:30:41 proxmox proxmox-backup-proxy[3994]: created new fixed index 2 ("vm/103/2021-10-19T04:30:40Z/drive-sata1.img.fidx")
Oct 19 06:30:42 proxmox proxmox-backup-proxy[3994]: register chunks in 'drive-sata2.img.fidx' from previous backup.
Oct 19 06:30:42 proxmox proxmox-backup-proxy[3994]: download 'drive-sata2.img.fidx' from previous backup.
Oct 19 06:30:42 proxmox proxmox-backup-proxy[3994]: created new fixed index 3 ("vm/103/2021-10-19T04:30:40Z/drive-sata2.img.fidx")
Oct 19 06:30:42 proxmox proxmox-backup-proxy[3994]: add blob "/bkup/incremental/vm/103/2021-10-19T04:30:40Z/qemu-server.conf.blob" (569 bytes, comp: 569)
Oct 19 06:30:42 proxmox kernel:  zd32: p1 p2 p3
Oct 19 06:30:49 proxmox vzdump[1323934]: VM 103 qmp command failed - VM 103 qmp command 'cont' failed - got timeout
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: backup failed: connection error: Transport endpoint is not connected (os error 107)
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: removing failed backup
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: removing backup snapshot "/bkup/incremental/vm/103/2021-10-19T04:30:40Z"
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: TASK ERROR: connection error: Transport endpoint is not connected (os error 107)
Oct 19 06:30:49 proxmox kernel:  zd32: p1 p2 p3
Oct 19 06:30:49 proxmox vzdump[1323934]: ERROR: Backup of VM 103 failed - VM 103 qmp command 'cont' failed - got timeout
Oct 19 06:30:49 proxmox vzdump[1323934]: INFO: Backup job finished with errors
Oct 19 06:30:49 proxmox vzdump[1323934]: job errors
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:49 proxmox vzdump[1323933]: <root@pam> end task UPID:proxmox:0014339E:01F2DFD5:616E49C9:vzdump::root@pam: job errors
Oct 19 06:30:49 proxmox CRON[1323928]: pam_unix(cron:session): session closed for user root
Oct 19 06:30:49 proxmox proxmox-backup-api[3975]: successful auth for user 'root@pam'
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:49 proxmox proxmox-backup-api[3975]: successful auth for user 'root@pam'
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:49 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:50 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:50 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:50 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:50 proxmox proxmox-backup-proxy[3994]: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
Oct 19 06:30:58 proxmox proxmox-backup-api[3975]: successful auth for user 'root@pam'

the pveversion -r output is as follows:
Code: 
# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-5
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-16
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
 
Last edited:
Same issue here: qmp command 'cont' failed - got timeout

Oddly, it happens with only 1 specific VM. All the other ones backup just fine (to PBS). The only difference is that this particular VM is the "busiest" one, handling more network connections and having more filesystem writes.

I'm not sure if this is related as well:
  • without qemu guest agent, I simply get this backup error and the VM continues operation as normal.
  • with qemu guest agent the VM goes into some kind of semi-hang where some processes are stuck, others continu to operate normally.
INFO: starting new backup job: vzdump 125 --node pve-ams --mode snapshot --storage backup --remove 0
INFO: Starting Backup of VM 125 (qemu)
INFO: Backup started at 2021-10-19 09:25:08
INFO: status = running
INFO: VM Name: mail
INFO: include disk 'scsi0' 'local-btrfs:125/vm-125-disk-0.raw' 150G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/125/2021-10-19T07:25:08Z'
INFO: started backup task 'cda5ac80-8228-4f37-9246-b8653414a736'
INFO: resuming VM again
ERROR: VM 125 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 125 failed - VM 125 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-10-19 09:25:23
INFO: Backup job finished with errors
TASK ERROR: job errors
Click to expand...

proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-16
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
Click to expand...

I fired up a snapshot of this VM on another host and it can do the backup just fine. Of course that one lacks actual activity.

Also curious is that making snapshots works fine, but making backups fails. Both very consistently so. (On rare occasion the backup will succeed.)

Snapshot log:
saving VM state and RAM using storage 'local-btrfs'
34.48 MiB in 1s
214.48 MiB in 2s
449.18 MiB in 3s
699.93 MiB in 4s
904.30 MiB in 7s
915.16 MiB in 10s
1.05 GiB in 11s
1.05 GiB in 12s
1.17 GiB in 13s
1.34 GiB in 14s
1.48 GiB in 15s
completed saving the VM state in 16s, saved 1.54 GiB
snapshotting 'drive-scsi0' (local-btrfs:125/vm-125-disk-0.raw)
TASK OK
Click to expand...
 
Hello, same issue here.

INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --node PROXMOX2 --storage pbs
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2021-10-20 09:51:16
INFO: status = running
INFO: VM Name: UbuWEBSERVER
INFO: include disk 'scsi1' 'ssd120gb:101/vm-101-disk-0.raw' 60G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2021-10-20T07:51:16Z'
INFO: started backup task '9db4e3fe-d879-4498-994e-2a19b1b9ab74'
INFO: resuming VM again
ERROR: VM 101 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - VM 101 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-10-20 09:51:25
INFO: Backup job finished with errors
TASK ERROR: job errors
Click to expand...

I also had a bit of issues trying to shutdown this VM:
TASK ERROR: VM quit/powerdown failed - got timeout
Click to expand...

pveversion:
root@PROXMOX2:~# pveversion --verbose
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.11.22-4-pve: 5.11.22-8
ceph-fuse: 15.2.14-pve1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
Click to expand...

I have another VM on the same host and another host with others 2 VMs.
I'm able to backup to pbs every VM except this one.
The error started last night, until yesterday I backed up this VM without any issue (and I didn't update any packages or pve version).
 
Last edited:
I've noticed that if I don't backup to PBS, but backup to the local file-system, the problem does not occur. Of course local backups are not so useful..

I wonder how PBS can cause this. And why for this specific VM and not others. (If it's a network issue I expect all VM's to have the same problem.)

The PBS log does not show anything special:

Oct 20 21:47:14 backup proxmox-backup-proxy[826]: starting new backup on datastore 'pve.ams': "vm/125/2021-10-20T19:47:14Z"
Oct 20 21:47:14 backup proxmox-backup-proxy[826]: download 'index.json.blob' from previous backup.
Oct 20 21:47:14 backup proxmox-backup-proxy[826]: register chunks in 'drive-scsi0.img.fidx' from previous backup.
Oct 20 21:47:14 backup proxmox-backup-proxy[826]: download 'drive-scsi0.img.fidx' from previous backup.
Oct 20 21:47:15 backup proxmox-backup-proxy[826]: created new fixed index 1 ("vm/125/2021-10-20T19:47:14Z/drive-scsi0.img.fidx")
Oct 20 21:47:15 backup proxmox-backup-proxy[826]: add blob "/sdb/pve.ams/vm/125/2021-10-20T19:47:14Z/qemu-server.conf.blob" (351 bytes, comp: 351)
Oct 20 21:47:24 backup proxmox-backup-proxy[826]: backup ended and finish failed: backup ended but finished flag is not set.
Oct 20 21:47:24 backup proxmox-backup-proxy[826]: removing unfinished backup
Oct 20 21:47:24 backup proxmox-backup-proxy[826]: TASK ERROR: backup ended but finished flag is not set.
Oct 20 21:47:24 backup proxmox-backup-proxy[826]: removing backup snapshot "/sdb/pve.ams/vm/125/2021-10-20T19:47:14Z"
Click to expand...
 
Same issue here, freshly installed backup server . all backups are working but i cannot backup one vm.
Debugged for hours without any success
Code: 
INFO: started backup task 'f076c6d3-2cb2-4bca-bc7d-b87a765c3db6'
INFO: resuming VM again
ERROR: VM 103 qmp command 'cont' failed - got timeout
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back