Error when finishing task of backup

D

dmembibre

Member
Sep 23, 2020
16
3
8
38
Hi,

Since last night the most task of backups finish with the next error:

ERROR: VM 801045 qmp command 'query-backup' failed - unable to connect to VM 801045 qmp socket - timeout after 5974 retries
INFO: aborting backup job

and the vm freezes, but if i check the same task in pbs the result is:

2020-09-23T02:12:29+02:00: successfully closed fixed index 2
2020-09-23T02:12:30+02:00: add blob "/rpool/p4q/bck-vms-p4q/vm/801045/2020-09-23T00:00:02Z/index.json.blob" (379 bytes, comp: 379)
2020-09-23T02:28:40+02:00: successfully finished backup
2020-09-23T02:28:40+02:00: backup finished successfully
2020-09-23T02:28:40+02:00: TASK OK

My PBS version is 0.8-19 and 6.2-11 for pve-manager.

can someone help me?

Thanks.
 
Hi,

Yesterday, before upgrading to 0.8-17 the task of backup was completed without errors and without freeze the vm

INFO: 97% (5.6 GiB of 5.8 GiB) in 3m 45s, read: 77.3 MiB/s, write: 77.3 MiB/s
INFO: 100% (5.8 GiB of 5.8 GiB) in 3m 47s, read: 60.0 MiB/s, write: 60.0 MiB/s
INFO: backup was done incrementally, reused 3.44 TiB (99%)
INFO: transferred 5.75 GiB in 227 seconds (25.9 MiB/s)
INFO: Finished Backup of VM 801045 (00:03:50)
INFO: Backup finished at 2020-09-22 02:03:52
INFO: Backup job finished successfully
TASK OK

Thanks.
 
Hello guys, first of ALL, thank you for this great tool Proxmox VE and Backup!

I have same problem here, after update yesterday:

Code: 
100: 2020-09-23 02:12:16 ERROR: VM 100 qmp command 'query-backup' failed - got timeout
100: 2020-09-23 02:12:16 INFO: aborting backup job
100: 2020-09-23 02:22:16 ERROR: VM 100 qmp command 'backup-cancel' failed - unable to connect to VM 100 qmp socket - timeout after 5985 retries
100: 2020-09-23 02:22:16 ERROR: Backup of VM 100 failed - VM 100 qmp command 'query-backup' failed - got timeout

But if we look at PBS, backup is finished OK:

Code: 
()
2020-09-22T23:00:27-03:00: starting new backup on datastore 'backup': "vm/100/2020-09-23T02:00:02Z"
2020-09-22T23:00:27-03:00: download 'index.json.blob' from previous backup.
2020-09-22T23:00:27-03:00: register chunks in 'drive-scsi0.img.fidx' from previous backup.
2020-09-22T23:00:27-03:00: download 'drive-scsi0.img.fidx' from previous backup.
2020-09-22T23:00:27-03:00: created new fixed index 1 ("vm/100/2020-09-23T02:00:02Z/drive-scsi0.img.fidx")
2020-09-22T23:00:27-03:00: add blob "/backup/vm/100/2020-09-23T02:00:02Z/qemu-server.conf.blob" (438 bytes, comp: 438)
2020-09-23T02:02:15-03:00: Upload statistics for 'drive-scsi0.img.fidx'
2020-09-23T02:02:15-03:00: UUID: e4aaa2e097344ae1ad89baa5fef48fce
2020-09-23T02:02:15-03:00: Checksum: 07af4a922ea8e38b860160f01024bb908251d5e126c41f900a199ceebd851f75
2020-09-23T02:02:15-03:00: Size: 858993459200
2020-09-23T02:02:15-03:00: Chunk count: 204800
2020-09-23T02:02:15-03:00: Upload size: 50746884096 (5%)
2020-09-23T02:02:15-03:00: Duplicates: 192701+46 (94%)
2020-09-23T02:02:15-03:00: Compression: 31%
2020-09-23T02:02:15-03:00: successfully closed fixed index 1
2020-09-23T02:02:15-03:00: add blob "/backup/vm/100/2020-09-23T02:00:02Z/index.json.blob" (322 bytes, comp: 322)
2020-09-23T02:38:36-03:00: successfully finished backup
2020-09-23T02:38:36-03:00: backup had errors but finished: connection error: Transport endpoint is not connected (os error 107)
2020-09-23T02:38:36-03:00: TASK OK
 
Hi,

As you can see on pbs log, during the backup it was a connectivity^network error(connection error ....). And because of that also the client after some retry decide to cancel the backup task. In the end because of this cancelation, pbs have consider that the task is finished (succes with errors).

So I guess your problem is related with your network (you could investigate this possibility )

Good luck/ Bafta !
 
More information about the issue:
PVE: proxmox-ve: 6.2-1 (running kernel: 5.4.60-1-pve) or proxmox-ve: 6.2-2 (running kernel: 5.4.60-1-pve)
PBS: problem on higher version than 0.8.16-1. PBS is VM with storage on RBD, free 900GB of the 4TB. PBS is remotely (other data center).
Backup job at end fails or freezes VM for 1-6 minutes on PVE. The same job is successful on PBS.
 
Can confirm the problem VM FREEZING while in backup.

Worst case.

@stoiko-ivanov and Proxmox-Team, can we get a patch on this ASAP?
 
Sralityhe said:
Can confirm the problem VM FREEZING while in backup.

Worst case.

@stoiko-ivanov and Proxmox-Team, can we get a patch on this ASAP?
Click to expand...

This is a beta, VMs freezing should not really happen and is not ideal, but some fallout during a beta can be expected.

For now, follow the workaround which seems to help the people affected here, while we are investigating this.
 
  • Like
Reactions: IT ProCare
Same problem here. Certain VMs freeze after backup was finished. It's always the same VMs, but not always all of them.

Will do a downgrade for now and watch this thread.

I filed a bug on bugzilla - see https://bugzilla.proxmox.com/show_bug.cgi?id=3043 and added some screenshots:
- output of proxmox ve backup report of vm 211
- output of proxmox backup server report of vm 211
- output on console of vm 211
 

Attachments

  • pbs1.png
    pbs1.png
    17 KB · Views: 25
  • pbs2.png
    pbs2.png
    49.5 KB · Views: 25
  • pbs3.png
    pbs3.png
    103.3 KB · Views: 24
Last edited:
A new pve-qemu-kvm package with version 5.1.0-3 just got available on the pvetest repository.

It includes a fix for the hanging VMs and QMP timeouts at the end of backup jobs.

If you can, please update and either do a fresh start of the VM (reboot over API works too) or live migrate it to an already updated node, to make it use the new KVM/QEMU executable.
 
  • Like
Reactions: Sralityhe
See: https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_test_repo for details on the pvetest repository.

If you want to only upgrade the pve-qemu package you can do something like:
Bash: 
# add pvetest repo
echo 'deb http://download.proxmox.com/debian/pve buster pvetest' > /etc/apt/sources.list.d/pvetest.list
apt update
# updgrade only QEMU related packages
apt install pve-qemu-kvm qemu-server
# remove pvetest again to avoid other packages updating
rm /etc/apt/sources.list.d/pvetest.list
apt update
 
  • Like
Reactions: RobFantini
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back