Backup to PBS failed - qmp command 'query-pbs-bitmap-info' failed - got timeout

I

ioanv

Well-Known Member
Dec 11, 2014
47
4
48
Hello all

I have several VMs (mostly CentOS and Ubuntu) distributed across a few proxmox nodes. I have configured backup tasks for all VMs on all nodes.
However, constantly, some of the VMs are failing to backup to PBS.

PBS version in 2.1, Proxmox version 7.1-8. (However I had the same issue on prox6 and on PBS 1.0).
Backup fails either scheduled on manually.

Interesting enough, some other VMs are doing the backup just fine.

Here are the logs for a failed manual backup:

On PBS:

Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: starting new backup on datastore 'servers': "vm/112/2021-12-13T12:24:03Z"
Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: GET /previous: 400 Bad Request: no valid previous backup
Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: created new fixed index 1 ("vm/112/2021-12-13T12:24:03Z/drive-ide0.img.fidx")
Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: add blob "/mnt/datastore/servers/vm/112/2021-12-13T12:24:03Z/qemu-server.conf.blob" (319 bytes, comp: 319)
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: backup ended and finish failed: backup ended but finished flag is not set.
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: removing unfinished backup
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: removing backup snapshot "/mnt/datastore/servers/vm/112/2021-12-13T12:24:03Z"
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: TASK ERROR: backup ended but finished flag is not set.

On Proxmox node:

INFO: starting new backup job: vzdump 112 --remove 0 --node prox4 --mode snapshot --storage pbserver
INFO: Starting Backup of VM 112 (qemu)
INFO: Backup started at 2021-12-13 14:24:03
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: mimir
INFO: include disk 'ide0' 'mimir_s3_nfs:112/vm-112-disk-1.raw' 86G
INFO: creating Proxmox Backup Server archive 'vm/112/2021-12-13T12:24:03Z'
INFO: starting kvm to execute backup task
INFO: started backup task '1c7e6de2-640c-46af-a64f-f0f3fad409cf'
ERROR: VM 112 qmp command 'query-pbs-bitmap-info' failed - got timeout
INFO: aborting backup job
INFO: stopping kvm after backup task
trying to acquire lock...
OK
ERROR: Backup of VM 112 failed - VM 112 qmp command 'query-pbs-bitmap-info' failed - got timeout
INFO: Failed at 2021-12-13 14:24:15
INFO: Backup job finished with errors
TASK ERROR: job errors

Also the VMs with problems are able to backup on local storage just fine.

INFO: starting new backup job: vzdump 112 --remove 0 --storage local --compress zstd --mode snapshot --node prox4
INFO: Starting Backup of VM 112 (qemu)
INFO: Backup started at 2021-12-13 14:52:44
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: mimir
INFO: include disk 'ide0' 'mimir_s3_nfs:112/vm-112-disk-1.raw' 86G
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-112-2021_12_13-14_52_44.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '11f89ad4-868d-431b-ad8e-a859fcf97ce5'
INFO: 0% (257.4 MiB of 86.0 GiB) in 3s, read: 85.8 MiB/s, write: 74.0 MiB/s
INFO: 1% (891.8 MiB of 86.0 GiB) in 11s, read: 79.3 MiB/s, write: 47.3 MiB/s
INFO: 2% (1.7 GiB of 86.0 GiB) in 23s, read: 74.0 MiB/s, write: 63.5 MiB/s
INFO: 3% (2.6 GiB of 86.0 GiB) in 37s, read: 64.7 MiB/s, write: 64.4 MiB/s

Any ideas?

Thank you
Vlad
 
Hello,

Did you notice high I/O in your Proxmox VE server during the backup time? - if so, could you please try to set the Bandwidth limit by going to Datacenter->Options->Bandwidth Limits
 
Moayad said:
Hello,

Did you notice high I/O in your Proxmox VE server during the backup time? - if so, could you please try to set the Bandwidth limit by going to Datacenter->Options->Bandwidth Limits
Click to expand...
Hi

Thanks for your response. I didn't noticed high traffic and I do not think this is the reason because some of the VMs do the backup corectly while others fail but it is always the same VMs that fail.
They also fail when I run the backup manually when there is virtually no activity on the VMs.
Even some of the failing VMs are shutdown.

Kind regards
Vlad
 
Hi have exactly the same problem all servers backup correctly but actually one server fail to backup :



PVE 7.1

Code: 
proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-5
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

VM Config :

Code: 
agent: 1
bootdisk: ide0
cores: 2
ide0: netapp_sata:132/vm-132-disk-0.qcow2,size=50G
ide2: none,media=cdrom
memory: 4096
name: xxxxx
net0: e1000=xxxxx,bridge=vmbr0,tag=172
numa: 0
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=4235a5a2-9e4f-457c-a4b1-d2f61d59fcfd
sockets: 2

VM OS : Windows Server

Code: 
132: 2022-01-27 02:18:11 INFO: Starting Backup of VM 132 (qemu)
132: 2022-01-27 02:18:11 INFO: status = running
132: 2022-01-27 02:18:11 INFO: VM Name: xxxxx
132: 2022-01-27 02:18:11 INFO: include disk 'ide0' 'netapp_sata:132/vm-132-disk-0.qcow2' 50G
132: 2022-01-27 02:18:11 INFO: backup mode: snapshot
132: 2022-01-27 02:18:11 INFO: ionice priority: 0
132: 2022-01-27 02:18:11 INFO: creating Proxmox Backup Server archive 'vm/132/2022-01-27T01:18:11Z'
132: 2022-01-27 02:18:11 INFO: issuing guest-agent 'fs-freeze' command
132: 2022-01-27 02:18:20 INFO: issuing guest-agent 'fs-thaw' command
132: 2022-01-27 02:18:30 ERROR: VM 132 qmp command 'guest-fsfreeze-thaw' failed - got timeout
132: 2022-01-27 02:18:30 INFO: started backup task 'fc5ae6e6-6a18-496b-bb91-985ba732a43f'
132: 2022-01-27 02:18:30 INFO: resuming VM again
132: 2022-01-27 02:18:35 ERROR: VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout
132: 2022-01-27 02:18:35 INFO: aborting backup job
132: 2022-01-27 02:18:41 INFO: resuming VM again
132: 2022-01-27 02:18:41 ERROR: Backup of VM 132 failed - VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout

Best Regards
 
Last edited:
Dear All,

I have the same issues, can you help?

Code: 
On PBS:

2022-02-22T14:11:24+07:00: starting new backup on datastore 'PBS-xxx-xxx': "vm/xxx/2022-02-22T07:11:24Z"
2022-02-22T14:11:24+07:00: download 'index.json.blob' from previous backup.
2022-02-22T14:11:24+07:00: backup ended and finish failed: backup ended but finished flag is not set.
2022-02-22T14:11:24+07:00: removing unfinished backup
2022-02-22T14:11:24+07:00: TASK ERROR: backup ended but finished flag is not set.

On ProxmoxVE:

()
INFO: starting new backup job: vzdump xxx --node ProxmoxXXX --storage PBS-xxx-xxx --mode snapshot --all 0 --mailto xxx@xxx.xxx.yyy --mailnotification always
INFO: Starting Backup of VM xxx (qemu)
INFO: Backup started at 2022-02-22 14:11:24
INFO: status = running
INFO: VM Name: XXX.xxx.xxx.yyy
INFO: include disk 'sata0' 'local-lvm:vm-xxx-disk-0' 80G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/xxx/2022-02-22T07:11:24Z'
ERROR: VM xxx qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM xxx failed - VM xxx qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2022-02-22 14:11:24
INFO: Backup job finished with errors
TASK ERROR: job errors

Thanks,
Tuan Ngo
 
Last edited:
liberodark said:
Hi have exactly the same problem all servers backup correctly but actually one server fail to backup :



PVE 7.1

Code: 
proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-5
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

VM Config :

Code: 
agent: 1
bootdisk: ide0
cores: 2
ide0: netapp_sata:132/vm-132-disk-0.qcow2,size=50G
ide2: none,media=cdrom
memory: 4096
name: xxxxx
net0: e1000=xxxxx,bridge=vmbr0,tag=172
numa: 0
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=4235a5a2-9e4f-457c-a4b1-d2f61d59fcfd
sockets: 2

VM OS : Windows Server

Code: 
132: 2022-01-27 02:18:11 INFO: Starting Backup of VM 132 (qemu)
132: 2022-01-27 02:18:11 INFO: status = running
132: 2022-01-27 02:18:11 INFO: VM Name: xxxxx
132: 2022-01-27 02:18:11 INFO: include disk 'ide0' 'netapp_sata:132/vm-132-disk-0.qcow2' 50G
132: 2022-01-27 02:18:11 INFO: backup mode: snapshot
132: 2022-01-27 02:18:11 INFO: ionice priority: 0
132: 2022-01-27 02:18:11 INFO: creating Proxmox Backup Server archive 'vm/132/2022-01-27T01:18:11Z'
132: 2022-01-27 02:18:11 INFO: issuing guest-agent 'fs-freeze' command
132: 2022-01-27 02:18:20 INFO: issuing guest-agent 'fs-thaw' command
132: 2022-01-27 02:18:30 ERROR: VM 132 qmp command 'guest-fsfreeze-thaw' failed - got timeout
132: 2022-01-27 02:18:30 INFO: started backup task 'fc5ae6e6-6a18-496b-bb91-985ba732a43f'
132: 2022-01-27 02:18:30 INFO: resuming VM again
132: 2022-01-27 02:18:35 ERROR: VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout
132: 2022-01-27 02:18:35 INFO: aborting backup job
132: 2022-01-27 02:18:41 INFO: resuming VM again
132: 2022-01-27 02:18:41 ERROR: Backup of VM 132 failed - VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout

Best Regards
Click to expand...
Hey,

i have the same issue.

Code: 
110: 2022-03-04 23:49:08 INFO: Starting Backup of VM 110 (qemu)
110: 2022-03-04 23:49:08 INFO: status = running
110: 2022-03-04 23:49:08 INFO: VM Name: xxxxxxxx
110: 2022-03-04 23:49:08 INFO: include disk 'scsi0' 'data:110/vm-110-disk-0.raw' 7856145K
110: 2022-03-04 23:49:08 INFO: backup mode: snapshot
110: 2022-03-04 23:49:08 INFO: ionice priority: 7
110: 2022-03-04 23:49:08 INFO: creating Proxmox Backup Server archive 'vm/110/2022-03-04T22:49:08Z'
110: 2022-03-04 23:49:08 INFO: started backup task '0a212af2-7cb9-4a73-a8e1-736911c18517'
110: 2022-03-04 23:49:08 INFO: resuming VM again
110: 2022-03-04 23:49:19 ERROR: VM 110 qmp command 'query-pbs-bitmap-info' failed - got timeout
110: 2022-03-04 23:49:19 INFO: aborting backup job
110: 2022-03-04 23:49:28 INFO: resuming VM again
110: 2022-03-04 23:49:28 ERROR: Backup of VM 110 failed - VM 110 qmp command 'query-pbs-bitmap-info' failed - got timeout

Code: 
Header
Proxmox
Virtual Environment 7.1-8
Search
Virtual Machine 110 (xxxxxx) on node 'pve2'
data
Filter VMID
Server View
Logs
()
agent: 0
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=6.1.0,ctime=1643113140
name: xxxxxxx
net0: e1000=76:D2:63:9F:5A:A9,bridge=vmbr40
numa: 0
onboot: 1
ostype: l26
scsi0: data:110/vm-110-disk-0.raw,size=7856145K
scsihw: virtio-scsi-pci
smbios1: uuid=0ddb4b72-e89c-4750-99e9-4eb97bd0d574
sockets: 1
startup: up=250
vmgenid: 80241eb8-fb95-4080-bb7b-ebca99df6167
#qmdump#map:scsi0:drive-scsi0:data:raw:


On an other Datastore it works. Only on the PBS Datastore i got the issue.

Anyone ideas?
 
I have two Datastores on my PBS:

Datastore 1 = directly mounted to a NFS Share
Datastore 2 = mounted to an USB Drive

I'm able to backup all containers and VMs to the USB Drive. I'm not able to backup 1 VM to Datastore 1.

Can somebody help?

Rafael
 
Hi,

Currently we are all waiting for a fix from Proxmox for PBS.
I don't know if we should play our support licenses on this topic or not.
But if it still drags I will make a support request with the PBS license.

Best Regards
 
It seems the 7.1-12 resolves the issue. I was able to complete full backups using PBS and local-btrfs without errors!
 
Well, not really, at least for me.

On a 7.1-12 PVE cluster, 3 nodes, full flash ceph storage.
3 VMs. Windows 2019 VMs. Exactely the same, excepted saize of data disk.

2 of them have never had a problem during backup.
Third one has same issue (backup mode = snapshot), nealy 2 days on 3...

Same qemu-guest agent, same VirtIO drivers (215).

A "backup now" never triggers the problem.

A backup job will fail 2 times on 3...

Any idea?

Thanks,

Christophe.
 
I'm still seeing this error. This crashes the VM and sometimes even corrupts the VM FS. Do we have any information from the Proxmox Team regarding this bug?


To complete this message, the error seems to always happen on the same few VMs. Its either a Windows VM, a debian 9.5 or a older custom linux based box. The Debian VM is 2TB which might be an additional cause?
 
Last edited:
Hi, we have same issue with one VM, since the Proxmox Update (7.2)

127: 2022-05-15 16:49:52 ERROR: VM 127 qmp command 'query-pbs-bitmap-info' failed - got timeout
127: 2022-05-15 16:49:52 INFO: aborting backup job
127: 2022-05-15 16:49:55 INFO: resuming VM again
127: 2022-05-15 16:49:55 ERROR: Backup of VM 127 failed - VM 127 qmp command 'query-pbs-bitmap-info' failed - got timeout
 
Hi,
for those of you using a Ceph storage without krbd, please see this thread for more information and workarounds.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom