ERROR: VM 103 qmp command 'query-backup' failed - got timeout ?

F

fpausp

Renowned Member
Aug 31, 2010
633
42
93
Austria near Vienna
Hi, I run the latest version ov Proxmox VE and Proxmox Backup Server...

When I try to backup my VMs I always get an error on just one machine (vm103):


Code: 
INFO:  75% (1.1 TiB of 1.5 TiB) in 34m 43s, read: 550.7 MiB/s, write: 6.3 MiB/s
INFO:  76% (1.1 TiB of 1.5 TiB) in 37m 23s, read: 95.5 MiB/s, write: 921.6 KiB/s
INFO:  77% (1.1 TiB of 1.5 TiB) in 39m 23s, read: 127.5 MiB/s, write: 136.5 KiB/s
INFO:  78% (1.1 TiB of 1.5 TiB) in 41m 33s, read: 118.7 MiB/s, write: 94.5 KiB/s
INFO:  79% (1.2 TiB of 1.5 TiB) in 43m 23s, read: 138.8 MiB/s, write: 335.1 KiB/s
INFO:  80% (1.2 TiB of 1.5 TiB) in 44m 23s, read: 266.5 MiB/s, write: 68.3 KiB/s
INFO:  81% (1.2 TiB of 1.5 TiB) in 44m 47s, read: 634.8 MiB/s, write: 0 B/s
INFO:  82% (1.2 TiB of 1.5 TiB) in 44m 52s, read: 3.6 GiB/s, write: 0 B/s
INFO:  83% (1.2 TiB of 1.5 TiB) in 45m  3s, read: 1.5 GiB/s, write: 0 B/s
INFO:  84% (1.2 TiB of 1.5 TiB) in 45m  6s, read: 4.8 GiB/s, write: 0 B/s
INFO:  85% (1.2 TiB of 1.5 TiB) in 45m  9s, read: 4.9 GiB/s, write: 0 B/s
INFO:  86% (1.3 TiB of 1.5 TiB) in 45m 12s, read: 5.0 GiB/s, write: 0 B/s
INFO:  87% (1.3 TiB of 1.5 TiB) in 45m 15s, read: 5.0 GiB/s, write: 0 B/s
INFO:  88% (1.3 TiB of 1.5 TiB) in 45m 18s, read: 4.8 GiB/s, write: 0 B/s
INFO:  89% (1.3 TiB of 1.5 TiB) in 45m 21s, read: 4.3 GiB/s, write: 0 B/s
INFO:  90% (1.3 TiB of 1.5 TiB) in 45m 24s, read: 4.9 GiB/s, write: 0 B/s
INFO:  91% (1.3 TiB of 1.5 TiB) in 45m 27s, read: 4.9 GiB/s, write: 0 B/s
INFO:  92% (1.3 TiB of 1.5 TiB) in 45m 30s, read: 5.0 GiB/s, write: 0 B/s
INFO:  93% (1.4 TiB of 1.5 TiB) in 45m 37s, read: 2.1 GiB/s, write: 0 B/s
INFO:  94% (1.4 TiB of 1.5 TiB) in 45m 40s, read: 5.0 GiB/s, write: 0 B/s
INFO:  95% (1.4 TiB of 1.5 TiB) in 45m 43s, read: 4.8 GiB/s, write: 0 B/s
INFO:  96% (1.4 TiB of 1.5 TiB) in 45m 46s, read: 5.1 GiB/s, write: 0 B/s
INFO:  97% (1.4 TiB of 1.5 TiB) in 45m 49s, read: 4.9 GiB/s, write: 0 B/s
INFO:  98% (1.4 TiB of 1.5 TiB) in 45m 52s, read: 5.0 GiB/s, write: 0 B/s
INFO:  99% (1.5 TiB of 1.5 TiB) in 45m 56s, read: 4.6 GiB/s, write: 0 B/s
ERROR: VM 103 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job

Do you know the error and how can I solve it?
 
Hi, Guys. I just want to confirm the issue still exists. All of our VM-s are Debain (ranging from 7-10) with qemu-agent (and only a couple without). Issue appears to be random and bricks VM (only reboot helps). After the reboot it works OK for example 7 days and the issue starts to reappear. Also on the forum I found that updating the qemu server from the test repo possibly solves the issue (we have done the update but the issue still appears). When i do the manual backup to PBS it works without a problem.

Code: 
INFO: Starting Backup of VM 163 (qemu)
INFO: Backup started at 2020-11-05 03:33:39
INFO: status = running
INFO: VM Name: xxx
INFO: include disk 'scsi0' 'shared_lvm:vm-163-disk-0' 10G
INFO: include disk 'scsi1' 'shared_lvm:vm-163-disk-0' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/163/2020-11-05T02:33:39Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 163 qmp command 'guest-fsfreeze-thaw' failed - got timeout
ERROR: VM 163 qmp command 'backup' failed - got timeout
ERROR: Backup of VM 163 failed - VM 163 qmp command 'backup' failed - got timeout
INFO: Failed at 2020-11-05 03:34:

We have the newest stable version of proxmox-VE (community subscription, cluster of 5 nodes with 10G interface-s, seperate for cluster, VM data and FC for shared storage) and proxmox-backup.

Code: 
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-3-pve: 5.3.13-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2




Backup Server 0.9-6

We've been using PBS for a while.

Thanks.

BR.

Michael

P.S.

probably also related to https://forum.proxmox.com/threads/qmp-command-backup-failed-got-timeout.77749/
 
Last edited:
fpausp said:
Hi, I run the latest version ov Proxmox VE and Proxmox Backup Server...

When I try to backup my VMs I always get an error on just one machine (vm103):


Code: 
INFO:  75% (1.1 TiB of 1.5 TiB) in 34m 43s, read: 550.7 MiB/s, write: 6.3 MiB/s
INFO:  76% (1.1 TiB of 1.5 TiB) in 37m 23s, read: 95.5 MiB/s, write: 921.6 KiB/s
INFO:  77% (1.1 TiB of 1.5 TiB) in 39m 23s, read: 127.5 MiB/s, write: 136.5 KiB/s
INFO:  78% (1.1 TiB of 1.5 TiB) in 41m 33s, read: 118.7 MiB/s, write: 94.5 KiB/s
INFO:  79% (1.2 TiB of 1.5 TiB) in 43m 23s, read: 138.8 MiB/s, write: 335.1 KiB/s
INFO:  80% (1.2 TiB of 1.5 TiB) in 44m 23s, read: 266.5 MiB/s, write: 68.3 KiB/s
INFO:  81% (1.2 TiB of 1.5 TiB) in 44m 47s, read: 634.8 MiB/s, write: 0 B/s
INFO:  82% (1.2 TiB of 1.5 TiB) in 44m 52s, read: 3.6 GiB/s, write: 0 B/s
INFO:  83% (1.2 TiB of 1.5 TiB) in 45m  3s, read: 1.5 GiB/s, write: 0 B/s
INFO:  84% (1.2 TiB of 1.5 TiB) in 45m  6s, read: 4.8 GiB/s, write: 0 B/s
INFO:  85% (1.2 TiB of 1.5 TiB) in 45m  9s, read: 4.9 GiB/s, write: 0 B/s
INFO:  86% (1.3 TiB of 1.5 TiB) in 45m 12s, read: 5.0 GiB/s, write: 0 B/s
INFO:  87% (1.3 TiB of 1.5 TiB) in 45m 15s, read: 5.0 GiB/s, write: 0 B/s
INFO:  88% (1.3 TiB of 1.5 TiB) in 45m 18s, read: 4.8 GiB/s, write: 0 B/s
INFO:  89% (1.3 TiB of 1.5 TiB) in 45m 21s, read: 4.3 GiB/s, write: 0 B/s
INFO:  90% (1.3 TiB of 1.5 TiB) in 45m 24s, read: 4.9 GiB/s, write: 0 B/s
INFO:  91% (1.3 TiB of 1.5 TiB) in 45m 27s, read: 4.9 GiB/s, write: 0 B/s
INFO:  92% (1.3 TiB of 1.5 TiB) in 45m 30s, read: 5.0 GiB/s, write: 0 B/s
INFO:  93% (1.4 TiB of 1.5 TiB) in 45m 37s, read: 2.1 GiB/s, write: 0 B/s
INFO:  94% (1.4 TiB of 1.5 TiB) in 45m 40s, read: 5.0 GiB/s, write: 0 B/s
INFO:  95% (1.4 TiB of 1.5 TiB) in 45m 43s, read: 4.8 GiB/s, write: 0 B/s
INFO:  96% (1.4 TiB of 1.5 TiB) in 45m 46s, read: 5.1 GiB/s, write: 0 B/s
INFO:  97% (1.4 TiB of 1.5 TiB) in 45m 49s, read: 4.9 GiB/s, write: 0 B/s
INFO:  98% (1.4 TiB of 1.5 TiB) in 45m 52s, read: 5.0 GiB/s, write: 0 B/s
INFO:  99% (1.5 TiB of 1.5 TiB) in 45m 56s, read: 4.6 GiB/s, write: 0 B/s
ERROR: VM 103 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job

Do you know the error and how can I solve it?
Click to expand...
This topic is old and I'll share what I found as a solution.

First I entered my VM that was not running the backup and updated the repositories with (#apt update).
After updating the repositories I executed the command to install the Qemu Guest Agent again ( #apt install qemu-guest-agent).

And then the backup worked normally.
 
I have a very similar problem when backing up a Windows server. However, the difference is that the backup is not even started.
I will try reinstalling the qemu guest agent and give you feedback.

Code: 
INFO: Starting Backup of VM 888 (qemu)
INFO: Backup started at 2023-05-25 xx:xx:xx
INFO: status = running
INFO: VM Name: customer-server01
INFO: include disk 'scsi0' 'customer-storage01:vm-888-disk-1' 100G
INFO: include disk 'scsi1' 'customer-storage01:vm-888-disk-3' 800G
INFO: include disk 'efidisk0' 'customer-storage01:vm-888-disk-0' 528K
INFO: include disk 'tpmstate0' 'customer-storage01:vm-888-disk-4' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: skip unused drive 'customer-storage01:vm-888-disk-2' (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/888/2023-05-25Txx:xx:xxZ'
INFO: attaching TPM drive to QEMU for backup
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 888 qmp command 'backup' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 888 failed - VM 888 qmp command 'backup' failed - got timeout
INFO: Failed at 2023-05-25 xx:xx:xx
 
After some time without the problem, the errors occurred again. The backups of several machines ended with a timeout. In addition, I could not access a list of the backups on the Proxmox backup server in the VM's backup overview; there was also a timeout here.

When I looked at the backup server, I could see a large number of running tasks and a high IO load on the hard disks.

The tasks were verifications of large machines and garbage collection. After a reboot, the listing is fast again and also the backup is no problem.
 
Sebi-S said:
After some time without the problem, the errors occurred again. The backups of several machines ended with a timeout. In addition, I could not access a list of the backups on the Proxmox backup server in the VM's backup overview; there was also a timeout here.

When I looked at the backup server, I could see a large number of running tasks and a high IO load on the hard disks.

The tasks were verifications of large machines and garbage collection. After a reboot, the listing is fast again and also the backup is no problem.
Click to expand...

I join to the "club", last proxmox pve (8.0.4), qemu-agent installed, windows 2019 server, backup started at midnight and stopped at 64%.

INFO: 63% (473.5 GiB of 750.0 GiB) in 2h 23m, read: 25.7 MiB/s, write: 22.4 MiB/s
INFO: 64% (480.1 GiB of 750.0 GiB) in 2h 26m 20s, read: 33.7 MiB/s, write: 21.7 MiB/s
ERROR: VM 100 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: VM 100 qmp command 'backup-cancel' failed - unable to connect to VM 100 qmp socket - timeout after 5988 retries
INFO: resuming VM again
ERROR: Backup of VM 100 failed - VM 100 qmp command 'cont' failed - unable to connect to VM 100 qmp socket - timeout after 450 retries
INFO: Failed at 2023-11-20 02:51:15
INFO: Backup job finished with errors
TASK ERROR: job errors

After that no VNC connection, vncproxy timeout, unable to reach the VM, no shutdown, only a forced stop

It was a local backup

Honestly speaking I'm having hard times with backup in the last time..
 
If I'm not mistaken, there was a bug when backing up VMs with two or more disks. Check for updates and take a look at the changelog of pve-manager.
 
Hi,
marziniconsulting said:
I join to the "club", last proxmox pve (8.0.4), qemu-agent installed, windows 2019 server, backup started at midnight and stopped at 64%.

INFO: 63% (473.5 GiB of 750.0 GiB) in 2h 23m, read: 25.7 MiB/s, write: 22.4 MiB/s
INFO: 64% (480.1 GiB of 750.0 GiB) in 2h 26m 20s, read: 33.7 MiB/s, write: 21.7 MiB/s
ERROR: VM 100 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: VM 100 qmp command 'backup-cancel' failed - unable to connect to VM 100 qmp socket - timeout after 5988 retries
INFO: resuming VM again
ERROR: Backup of VM 100 failed - VM 100 qmp command 'cont' failed - unable to connect to VM 100 qmp socket - timeout after 450 retries
INFO: Failed at 2023-11-20 02:51:15
INFO: Backup job finished with errors
TASK ERROR: job errors

After that no VNC connection, vncproxy timeout, unable to reach the VM, no shutdown, only a forced stop

It was a local backup

Honestly speaking I'm having hard times with backup in the last time..
Click to expand...
in your case, it fails in the middle of the backup and telling from the error messages and similar issues in the past, it looks like QEMU process got completely stuck. Please post the output of pveversion -v and qm config 100.

Should the issue happen again, you can use apt install pve-qemu-kvm-dbgsym gdb to install the relevant debug symbols and debugger. And then obtain backtraces with:
Code: 
gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/100.pid)
assuming that 100 is the ID of the VM.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back