qmp command Backup failed got timeout

W

wire2hire

Member
Oct 20, 2020
6
1
8
40
Hello,

when i want make a backup from my vm it gives me a error:

NFO: starting new backup job: vzdump 100 --mode snapshot --remove 0 --node pve01 --storage PBS
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-10-20 15:39:43
INFO: status = running
INFO: VM Name: SERVER100
INFO: include disk 'scsi1' 'STORAGE:vm-100-disk-0' 480G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2020-10-20T13:39:43Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 100 qmp command 'backup' failed - got timeout
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - got timeout
INFO: Failed at 2020-10-20 15:40:49
INFO: Backup job finished with errors
TASK ERROR: job errors

pve 6.2.-12
pbs: 0.9-1 ( on the same HOST)

Tried it :

1. VM runs
2. VM off
3. agent enable and disable
4. Downgrade pbs to 0.8.16-1

Have anyone the same failure?
 
  • Like
Reactions: eTIC
wire2hire said:
Hello,

when i want make a backup from my vm it gives me a error:

NFO: starting new backup job: vzdump 100 --mode snapshot --remove 0 --node pve01 --storage PBS
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-10-20 15:39:43
INFO: status = running
INFO: VM Name: SERVER100
INFO: include disk 'scsi1' 'STORAGE:vm-100-disk-0' 480G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2020-10-20T13:39:43Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 100 qmp command 'backup' failed - got timeout
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - got timeout
INFO: Failed at 2020-10-20 15:40:49
INFO: Backup job finished with errors
TASK ERROR: job errors

pve 6.2.-12
pbs: 0.9-1 ( on the same HOST)

Tried it :

1. VM runs
2. VM off
3. agent enable and disable
4. Downgrade pbs to 0.8.16-1

Have anyone the same failure?
Click to expand...
I had the same error a few days ago. A VM with a Windows Server showed a similar problem. In my case the backup managed to finish, but it went from 45m before the error to 8 hours, with transfer rates for read/write of 10MB/s instead the 180MB/s as in previous days.

My Proxmox version: 6.2-4

Backup Resume:

....
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/NFS002/dump/vzdump-qemu-201-2020_10_20-05_00_02.vma.lzo'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 201 qmp command 'guest-fsfreeze-thaw' failed - got timeout
INFO: started backup task 'bb655090-0774-4d8c-8731-e133fc18d67d'
INFO: resuming VM again
...

at finish

...
INFO: transferred 322122 MB in 30683 seconds (10 MB/s)
INFO: archive file size: 180.68GB
INFO: delete old backup '/mnt/pve/NFS002/dump/vzdump-qemu-201-2020_10_13-05_00_01.vma.lzo'
INFO: Finished Backup of VM 201 (08:31:37)
INFO: Backup finished at 2020-10-20 13:31:39
INFO: Backup job finished successfully
TASK OK
...
 
Last edited:
here a syslog without agent.

Oct 21 09:50:16 pve01 pvedaemon[27971]: INFO: Starting Backup of VM 100 (qemu)
Oct 21 09:50:19 pve01 pvedaemon[27971]: VM 100 qmp command failed - VM 100 qmp command 'guest-ping' failed - got timeout
Oct 21 09:50:29 pve01 proxmox-backup-api[1313]: successful auth for user 'root@pam'
Oct 21 09:50:39 pve01 proxmox-backup-api[1313]: successful auth for user 'root@pam'
Oct 21 09:50:49 pve01 proxmox-backup-api[1313]: successful auth for user 'root@pam'
Oct 21 09:50:59 pve01 proxmox-backup-api[1313]: successful auth for user 'root@pam'
Oct 21 09:51:00 pve01 systemd[1]: Starting Proxmox VE replication runner...
Oct 21 09:51:00 pve01 systemd[1]: pvesr.service: Succeeded.
Oct 21 09:51:00 pve01 systemd[1]: Started Proxmox VE replication runner.
Oct 21 09:51:09 pve01 proxmox-backup-api[1313]: successful auth for user 'root@pam'
Oct 21 09:51:19 pve01 proxmox-backup-api[1313]: successful auth for user 'root@pam'
Oct 21 09:51:19 pve01 pvedaemon[27971]: VM 100 qmp command failed - VM 100 qmp command 'backup' failed - got timeout
Oct 21 09:51:19 pve01 pvedaemon[27971]: ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - got timeout
Oct 21 09:51:19 pve01 pvedaemon[27971]: INFO: Backup job finished with errors
Oct 21 09:51:19 pve01 pvedaemon[27971]: job errors
Oct 21 09:51:19 pve01 pvedaemon[1740]: <root@pam> end task UPID:pve01:00006D43:0000F5BB:5F8FE838:vzdump:100:root@pam: job errors
 
and syslog with pbs downgrade to 0.8-16

Oct 21 09:54:33 pve01 pvedaemon[1739]: <root@pam> starting task UPID:pve01:00007ACC:00015A05:5F8FE939:vzdump:100:root@pam:
Oct 21 09:54:33 pve01 pvedaemon[31436]: INFO: starting new backup job: vzdump 100 --storage PBS --node pve01 --remove 0 --mode snapshot
Oct 21 09:54:33 pve01 pvedaemon[31436]: INFO: Starting Backup of VM 100 (qemu)
Oct 21 09:54:36 pve01 proxmox-backup-api[31114]: successful auth for user 'root@pam'
Oct 21 09:54:36 pve01 pvedaemon[31436]: VM 100 qmp command failed - VM 100 qmp command 'guest-ping' failed - got timeout
Oct 21 09:54:49 pve01 proxmox-backup-api[31114]: successful auth for user 'root@pam'
Oct 21 09:54:53 pve01 systemd[1]: Starting Cleanup of Temporary Directories...
Oct 21 09:54:53 pve01 systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Oct 21 09:54:53 pve01 systemd[1]: Started Cleanup of Temporary Directories.
Oct 21 09:54:59 pve01 proxmox-backup-api[31114]: successful auth for user 'root@pam'
Oct 21 09:55:00 pve01 systemd[1]: Starting Proxmox VE replication runner...
Oct 21 09:55:00 pve01 systemd[1]: pvesr.service: Succeeded.
Oct 21 09:55:00 pve01 systemd[1]: Started Proxmox VE replication runner.
Oct 21 09:55:09 pve01 proxmox-backup-api[31114]: successful auth for user 'root@pam'
Oct 21 09:55:12 pve01 pvedaemon[1738]: <root@pam> successful auth for user 'root@pam'
Oct 21 09:55:19 pve01 proxmox-backup-api[31114]: successful auth for user 'root@pam'
Oct 21 09:55:29 pve01 proxmox-backup-api[31114]: successful auth for user 'root@pam'
Oct 21 09:55:36 pve01 pvedaemon[31436]: VM 100 qmp command failed - VM 100 qmp command 'backup' failed - got timeout
Oct 21 09:55:36 pve01 pvedaemon[31436]: ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - got timeout
Oct 21 09:55:36 pve01 pvedaemon[31436]: INFO: Backup job finished with errors
Oct 21 09:55:36 pve01 pvedaemon[31436]: job errors
Oct 21 09:55:36 pve01 pvedaemon[1739]: <root@pam> end task UPID:pve01:00007ACC:00015A05:5F8FE939:vzdump:100:root@pam: job errors
 
When i start a Backup to non PBS storage, its just run.

I I have noticed, that no task show up in PBS! Or should it show up, when backup ist started?


On one another host with the same Versions Backup runs smooth.

Have anyone a idee? The Vm ist a Windows Server.
 
Last edited:
I have seen it give a Bug on Bugzilla and the State is fix, but i can alwys not Backup to pbs. VZDUMP on local Proxmox is function.

what more can I do to narrow down the problem?
 
Hello,

same problem here. PBS 1.0-1 and PVE 6.2-15.

When the qmp command times out, the corresponding VMs are unresponsive for multiple minutes. Sometimes they don't even come alive again and I need to stop the VM the hard way and start it again. Reboot or something like that isn't possible then anymore as even then the qmp command for that times out. Looks like the qmp socket dies completely.

Turned off now backups via PBS. Hope that there is a solution soon.
 
Last edited:
Hi. In my case above, I have found a temporary solution that has worked well for me.

As simple as restoring one of those 8-hour backups to a new VM. From that moment on, the backup time has been reduced to 35 minutes.

The reason why it has been solved, would be related to some problem in the structure or parameterization of the original VM. But I don't have it clear either, it is only a hypothesis. ;)
 
Last edited:
We are encountering this problem on current pve 6.3 and pbs 1.0.5 enterprise repo.

The VM was left in a "frozen" state, only responding to hard "stop". (no ping, no console)

Ceph Octopus Storage, 10Gbe, nvme only, nothing obvious found in the host syslog

the previous backups runs where ok

PBS task Log

Code: 
2020-12-19T09:00:04+01:00: starting new backup on datastore 'data': "vm/103/2020-12-19T08:00:02Z"
2020-12-19T09:00:04+01:00: download 'index.json.blob' from previous backup.
2020-12-19T09:00:04+01:00: register chunks in 'drive-scsi0.img.fidx' from previous backup.
2020-12-19T09:00:05+01:00: download 'drive-scsi0.img.fidx' from previous backup.
2020-12-19T09:00:05+01:00: created new fixed index 1 ("vm/103/2020-12-19T08:00:02Z/drive-scsi0.img.fidx")
2020-12-19T09:00:06+01:00: register chunks in 'drive-scsi1.img.fidx' from previous backup.
2020-12-19T09:00:07+01:00: download 'drive-scsi1.img.fidx' from previous backup.
2020-12-19T09:00:14+01:00: created new fixed index 2 ("vm/103/2020-12-19T08:00:02Z/drive-scsi1.img.fidx")
2020-12-19T09:01:13+01:00: register chunks in 'drive-scsi2.img.fidx' from previous backup.
2020-12-19T09:01:14+01:00: download 'drive-scsi2.img.fidx' from previous backup.
2020-12-19T09:01:14+01:00: created new fixed index 3 ("vm/103/2020-12-19T08:00:02Z/drive-scsi2.img.fidx")
2020-12-19T09:01:44+01:00: register chunks in 'drive-scsi3.img.fidx' from previous backup.
2020-12-19T09:01:48+01:00: download 'drive-scsi3.img.fidx' from previous backup.
2020-12-19T09:01:48+01:00: created new fixed index 4 ("vm/103/2020-12-19T08:00:02Z/drive-scsi3.img.fidx")
2020-12-19T09:02:12+01:00: add blob "/mnt/datastore/data/vm/103/2020-12-19T08:00:02Z/qemu-server.conf.blob" (806 bytes, comp: 806)
2020-12-19T09:02:12+01:00: add blob "/mnt/datastore/data/vm/103/2020-12-19T08:00:02Z/fw.conf.blob" (209 bytes, comp: 209)
2020-12-19T13:12:38+01:00: backup ended and finish failed: backup ended but finished flag is not set.
2020-12-19T13:12:38+01:00: removing unfinished backup
2020-12-19T13:12:38+01:00: TASK ERROR: backup ended but finished flag is not set.

Backup Logs

Code: 
107: 2020-12-19 09:00:02 INFO: Starting Backup of VM 107 (qemu)
107: 2020-12-19 09:00:02 INFO: status = running
107: 2020-12-19 09:00:02 INFO: VM Name: xxxx.xxx.xxxx
107: 2020-12-19 09:00:02 INFO: include disk 'scsi0' 'FastSSDPool:vm-107-disk-1' 100G
107: 2020-12-19 09:00:02 INFO: backup mode: snapshot
107: 2020-12-19 09:00:02 INFO: ionice priority: 7
107: 2020-12-19 09:00:02 INFO: snapshots found (not included into backup)
107: 2020-12-19 09:00:02 INFO: creating Proxmox Backup Server archive 'vm/107/2020-12-19T08:00:02Z'
107: 2020-12-19 09:00:05 INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
107: 2020-12-19 09:01:05 ERROR: VM 107 qmp command 'backup' failed - got timeout
107: 2020-12-19 09:01:05 INFO: aborting backup job
107: 2020-12-19 09:01:10 ERROR: Backup of VM 107 failed - VM 107 qmp command 'backup' failed - got timeout

Code: 
103: 2020-12-19 09:00:02 INFO: Starting Backup of VM 103 (qemu)
103: 2020-12-19 09:00:02 INFO: status = running
103: 2020-12-19 09:00:02 INFO: VM Name: yyyyyyy.yyyyyyy.yyyyyyy
103: 2020-12-19 09:00:02 INFO: include disk 'scsi0' 'FastSSDPool:vm-103-disk-0' 514252M
103: 2020-12-19 09:00:02 INFO: include disk 'scsi1' 'FastSSDPool:vm-103-disk-1' 1000G
103: 2020-12-19 09:00:02 INFO: include disk 'scsi2' 'FastSSDPool:vm-103-disk-2' 300G
103: 2020-12-19 09:00:02 INFO: include disk 'scsi3' 'FastSSDPool:vm-103-disk-3' 500G
103: 2020-12-19 09:00:02 INFO: backup mode: snapshot
103: 2020-12-19 09:00:02 INFO: ionice priority: 7
103: 2020-12-19 09:00:02 INFO: snapshots found (not included into backup)
103: 2020-12-19 09:00:02 INFO: creating Proxmox Backup Server archive 'vm/103/2020-12-19T08:00:02Z'
103: 2020-12-19 09:00:02 INFO: issuing guest-agent 'fs-freeze' command
103: 2020-12-19 09:01:03 INFO: issuing guest-agent 'fs-thaw' command
103: 2020-12-19 09:01:13 ERROR: VM 103 qmp command 'guest-fsfreeze-thaw' failed - got timeout
103: 2020-12-19 09:01:13 ERROR: VM 103 qmp command 'backup' failed - got timeout
103: 2020-12-19 09:01:13 INFO: aborting backup job
103: 2020-12-19 09:12:12 ERROR: VM 103 qmp command 'backup-cancel' failed - got timeout
103: 2020-12-19 09:12:12 ERROR: Backup of VM 103 failed - VM 103 qmp command 'backup' failed - got timeout
 
Last edited:
Hello,

I'm encountering the exact same behaviour when the CPU usage on the Proxmox Backup Host is too high, due to for example the option "Verify on new backup". This makes the PBS verifying multiple backups in parallel and the CPU usage jumps between 80-100%. The next backups are failing then and some VMs are stuck for some time or don't even come alive. Then they need to be killed.

I've disabled any kind of backup verification during the creation of backups. Since then, I have no problems. During the night backups are made and during the day, the PBS is verifying the backups. :)
 
wire2hire said:
Hello,

when i want make a backup from my vm it gives me a error:

NFO: starting new backup job: vzdump 100 --mode snapshot --remove 0 --node pve01 --storage PBS
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-10-20 15:39:43
INFO: status = running
INFO: VM Name: SERVER100
INFO: include disk 'scsi1' 'STORAGE:vm-100-disk-0' 480G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2020-10-20T13:39:43Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 100 qmp command 'backup' failed - got timeout
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - got timeout
INFO: Failed at 2020-10-20 15:40:49
INFO: Backup job finished with errors
TASK ERROR: job errors

pve 6.2.-12
pbs: 0.9-1 ( on the same HOST)

Tried it :

1. VM runs
2. VM off
3. agent enable and disable
4. Downgrade pbs to 0.8.16-1

Have anyone the same failure?
Click to expand...
This topic is old and I'll share what I found as a solution.

First I entered my VM that was not running the backup and updated the repositories with (#apt update).
After updating the repositories I executed the command to install the Qemu Guest Agent again ( #apt install qemu-guest-agent).

And then the backup worked normally.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom