Backup to PBS failed - qmp command 'query-pbs-bitmap-info' failed - got timeout

ioanv · Dec 13, 2021

Hello all

I have several VMs (mostly CentOS and Ubuntu) distributed across a few proxmox nodes. I have configured backup tasks for all VMs on all nodes.
However, constantly, some of the VMs are failing to backup to PBS.

PBS version in 2.1, Proxmox version 7.1-8. (However I had the same issue on prox6 and on PBS 1.0).
Backup fails either scheduled on manually.

Interesting enough, some other VMs are doing the backup just fine.

Here are the logs for a failed manual backup:

On PBS:

Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: starting new backup on datastore 'servers': "vm/112/2021-12-13T12:24:03Z"
Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: GET /previous: 400 Bad Request: no valid previous backup
Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: created new fixed index 1 ("vm/112/2021-12-13T12:24:03Z/drive-ide0.img.fidx")
Dec 13 14:24:04 alta proxmox-backup-proxy[3619639]: add blob "/mnt/datastore/servers/vm/112/2021-12-13T12:24:03Z/qemu-server.conf.blob" (319 bytes, comp: 319)
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: backup ended and finish failed: backup ended but finished flag is not set.
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: removing unfinished backup
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: removing backup snapshot "/mnt/datastore/servers/vm/112/2021-12-13T12:24:03Z"
Dec 13 14:24:14 alta proxmox-backup-proxy[3619639]: TASK ERROR: backup ended but finished flag is not set.

On Proxmox node:

INFO: starting new backup job: vzdump 112 --remove 0 --node prox4 --mode snapshot --storage pbserver
INFO: Starting Backup of VM 112 (qemu)
INFO: Backup started at 2021-12-13 14:24:03
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: mimir
INFO: include disk 'ide0' 'mimir_s3_nfs:112/vm-112-disk-1.raw' 86G
INFO: creating Proxmox Backup Server archive 'vm/112/2021-12-13T12:24:03Z'
INFO: starting kvm to execute backup task
INFO: started backup task '1c7e6de2-640c-46af-a64f-f0f3fad409cf'
ERROR: VM 112 qmp command 'query-pbs-bitmap-info' failed - got timeout
INFO: aborting backup job
INFO: stopping kvm after backup task
trying to acquire lock...
OK
ERROR: Backup of VM 112 failed - VM 112 qmp command 'query-pbs-bitmap-info' failed - got timeout
INFO: Failed at 2021-12-13 14:24:15
INFO: Backup job finished with errors
TASK ERROR: job errors

Also the VMs with problems are able to backup on local storage just fine.

INFO: starting new backup job: vzdump 112 --remove 0 --storage local --compress zstd --mode snapshot --node prox4
INFO: Starting Backup of VM 112 (qemu)
INFO: Backup started at 2021-12-13 14:52:44
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: mimir
INFO: include disk 'ide0' 'mimir_s3_nfs:112/vm-112-disk-1.raw' 86G
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-112-2021_12_13-14_52_44.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '11f89ad4-868d-431b-ad8e-a859fcf97ce5'
INFO: 0% (257.4 MiB of 86.0 GiB) in 3s, read: 85.8 MiB/s, write: 74.0 MiB/s
INFO: 1% (891.8 MiB of 86.0 GiB) in 11s, read: 79.3 MiB/s, write: 47.3 MiB/s
INFO: 2% (1.7 GiB of 86.0 GiB) in 23s, read: 74.0 MiB/s, write: 63.5 MiB/s
INFO: 3% (2.6 GiB of 86.0 GiB) in 37s, read: 64.7 MiB/s, write: 64.4 MiB/s

Any ideas?

Thank you
Vlad

Moayad · Dec 14, 2021

Hello,

Did you notice high I/O in your Proxmox VE server during the backup time? - if so, could you please try to set the Bandwidth limit by going to Datacenter->Options->Bandwidth Limits

ioanv · Dec 14, 2021

Moayad said:
Hello,

Did you notice high I/O in your Proxmox VE server during the backup time? - if so, could you please try to set the Bandwidth limit by going to Datacenter->Options->Bandwidth Limits

Hi

Thanks for your response. I didn't noticed high traffic and I do not think this is the reason because some of the VMs do the backup corectly while others fail but it is always the same VMs that fail.
They also fail when I run the backup manually when there is virtually no activity on the VMs.
Even some of the failing VMs are shutdown.

Kind regards
Vlad

ioanv · Jan 2, 2022

Any ideas?

liberodark · Jan 27, 2022

Hi have exactly the same problem all servers backup correctly but actually one server fail to backup :

PVE 7.1

Code:

proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-5
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

VM Config :

Code:

agent: 1
bootdisk: ide0
cores: 2
ide0: netapp_sata:132/vm-132-disk-0.qcow2,size=50G
ide2: none,media=cdrom
memory: 4096
name: xxxxx
net0: e1000=xxxxx,bridge=vmbr0,tag=172
numa: 0
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=4235a5a2-9e4f-457c-a4b1-d2f61d59fcfd
sockets: 2

VM OS : Windows Server

Code:

132: 2022-01-27 02:18:11 INFO: Starting Backup of VM 132 (qemu)
132: 2022-01-27 02:18:11 INFO: status = running
132: 2022-01-27 02:18:11 INFO: VM Name: xxxxx
132: 2022-01-27 02:18:11 INFO: include disk 'ide0' 'netapp_sata:132/vm-132-disk-0.qcow2' 50G
132: 2022-01-27 02:18:11 INFO: backup mode: snapshot
132: 2022-01-27 02:18:11 INFO: ionice priority: 0
132: 2022-01-27 02:18:11 INFO: creating Proxmox Backup Server archive 'vm/132/2022-01-27T01:18:11Z'
132: 2022-01-27 02:18:11 INFO: issuing guest-agent 'fs-freeze' command
132: 2022-01-27 02:18:20 INFO: issuing guest-agent 'fs-thaw' command
132: 2022-01-27 02:18:30 ERROR: VM 132 qmp command 'guest-fsfreeze-thaw' failed - got timeout
132: 2022-01-27 02:18:30 INFO: started backup task 'fc5ae6e6-6a18-496b-bb91-985ba732a43f'
132: 2022-01-27 02:18:30 INFO: resuming VM again
132: 2022-01-27 02:18:35 ERROR: VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout
132: 2022-01-27 02:18:35 INFO: aborting backup job
132: 2022-01-27 02:18:41 INFO: resuming VM again
132: 2022-01-27 02:18:41 ERROR: Backup of VM 132 failed - VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout

Best Regards

leen15 · Feb 3, 2022

Same problem here. Only on one of four VMs.

djsami · Feb 6, 2022

leen15 said:
Same problem here. Only on one of four VMs.

You have 1200 virtual servers, 14 of them, I don't understand why?

vnngoanhtuan · Feb 22, 2022

Dear All,

I have the same issues, can you help?

Code:

On PBS:

2022-02-22T14:11:24+07:00: starting new backup on datastore 'PBS-xxx-xxx': "vm/xxx/2022-02-22T07:11:24Z"
2022-02-22T14:11:24+07:00: download 'index.json.blob' from previous backup.
2022-02-22T14:11:24+07:00: backup ended and finish failed: backup ended but finished flag is not set.
2022-02-22T14:11:24+07:00: removing unfinished backup
2022-02-22T14:11:24+07:00: TASK ERROR: backup ended but finished flag is not set.

On ProxmoxVE:

()
INFO: starting new backup job: vzdump xxx --node ProxmoxXXX --storage PBS-xxx-xxx --mode snapshot --all 0 --mailto xxx@xxx.xxx.yyy --mailnotification always
INFO: Starting Backup of VM xxx (qemu)
INFO: Backup started at 2022-02-22 14:11:24
INFO: status = running
INFO: VM Name: XXX.xxx.xxx.yyy
INFO: include disk 'sata0' 'local-lvm:vm-xxx-disk-0' 80G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/xxx/2022-02-22T07:11:24Z'
ERROR: VM xxx qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM xxx failed - VM xxx qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: Failed at 2022-02-22 14:11:24
INFO: Backup job finished with errors
TASK ERROR: job errors

Thanks,
Tuan Ngo

florianwilde · Mar 4, 2022

liberodark said:

Hi have exactly the same problem all servers backup correctly but actually one server fail to backup :

PVE 7.1

Code:

proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-5
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

VM Config :

Code:

agent: 1
bootdisk: ide0
cores: 2
ide0: netapp_sata:132/vm-132-disk-0.qcow2,size=50G
ide2: none,media=cdrom
memory: 4096
name: xxxxx
net0: e1000=xxxxx,bridge=vmbr0,tag=172
numa: 0
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=4235a5a2-9e4f-457c-a4b1-d2f61d59fcfd
sockets: 2

VM OS : Windows Server

Code:

132: 2022-01-27 02:18:11 INFO: Starting Backup of VM 132 (qemu)
132: 2022-01-27 02:18:11 INFO: status = running
132: 2022-01-27 02:18:11 INFO: VM Name: xxxxx
132: 2022-01-27 02:18:11 INFO: include disk 'ide0' 'netapp_sata:132/vm-132-disk-0.qcow2' 50G
132: 2022-01-27 02:18:11 INFO: backup mode: snapshot
132: 2022-01-27 02:18:11 INFO: ionice priority: 0
132: 2022-01-27 02:18:11 INFO: creating Proxmox Backup Server archive 'vm/132/2022-01-27T01:18:11Z'
132: 2022-01-27 02:18:11 INFO: issuing guest-agent 'fs-freeze' command
132: 2022-01-27 02:18:20 INFO: issuing guest-agent 'fs-thaw' command
132: 2022-01-27 02:18:30 ERROR: VM 132 qmp command 'guest-fsfreeze-thaw' failed - got timeout
132: 2022-01-27 02:18:30 INFO: started backup task 'fc5ae6e6-6a18-496b-bb91-985ba732a43f'
132: 2022-01-27 02:18:30 INFO: resuming VM again
132: 2022-01-27 02:18:35 ERROR: VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout
132: 2022-01-27 02:18:35 INFO: aborting backup job
132: 2022-01-27 02:18:41 INFO: resuming VM again
132: 2022-01-27 02:18:41 ERROR: Backup of VM 132 failed - VM 132 qmp command 'query-pbs-bitmap-info' failed - got timeout

Best Regards

Hey,

i have the same issue.

Code:

110: 2022-03-04 23:49:08 INFO: Starting Backup of VM 110 (qemu)
110: 2022-03-04 23:49:08 INFO: status = running
110: 2022-03-04 23:49:08 INFO: VM Name: xxxxxxxx
110: 2022-03-04 23:49:08 INFO: include disk 'scsi0' 'data:110/vm-110-disk-0.raw' 7856145K
110: 2022-03-04 23:49:08 INFO: backup mode: snapshot
110: 2022-03-04 23:49:08 INFO: ionice priority: 7
110: 2022-03-04 23:49:08 INFO: creating Proxmox Backup Server archive 'vm/110/2022-03-04T22:49:08Z'
110: 2022-03-04 23:49:08 INFO: started backup task '0a212af2-7cb9-4a73-a8e1-736911c18517'
110: 2022-03-04 23:49:08 INFO: resuming VM again
110: 2022-03-04 23:49:19 ERROR: VM 110 qmp command 'query-pbs-bitmap-info' failed - got timeout
110: 2022-03-04 23:49:19 INFO: aborting backup job
110: 2022-03-04 23:49:28 INFO: resuming VM again
110: 2022-03-04 23:49:28 ERROR: Backup of VM 110 failed - VM 110 qmp command 'query-pbs-bitmap-info' failed - got timeout

Code:

Header
Proxmox
Virtual Environment 7.1-8
Search
Virtual Machine 110 (xxxxxx) on node 'pve2'
data
Filter VMID
Server View
Logs
()
agent: 0
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=6.1.0,ctime=1643113140
name: xxxxxxx
net0: e1000=76:D2:63:9F:5A:A9,bridge=vmbr40
numa: 0
onboot: 1
ostype: l26
scsi0: data:110/vm-110-disk-0.raw,size=7856145K
scsihw: virtio-scsi-pci
smbios1: uuid=0ddb4b72-e89c-4750-99e9-4eb97bd0d574
sockets: 1
startup: up=250
vmgenid: 80241eb8-fb95-4080-bb7b-ebca99df6167
#qmdump#map:scsi0:drive-scsi0:data:raw:

On an other Datastore it works. Only on the PBS Datastore i got the issue.

Anyone ideas?

Loni78 · Apr 11, 2022

I have two Datastores on my PBS:

Datastore 1 = directly mounted to a NFS Share
Datastore 2 = mounted to an USB Drive

I'm able to backup all containers and VMs to the USB Drive. I'm not able to backup 1 VM to Datastore 1.

Can somebody help?

Rafael

liberodark · Apr 11, 2022

Hi,

Currently we are all waiting for a fix from Proxmox for PBS.
I don't know if we should play our support licenses on this topic or not.
But if it still drags I will make a support request with the PBS license.

Best Regards

sarsenal · Apr 20, 2022

Any update on this patch? We have the same issue.

Cheers

sarsenal · Apr 20, 2022

It seems the 7.1-12 resolves the issue. I was able to complete full backups using PBS and local-btrfs without errors!

christophe · May 4, 2022

Well, not really, at least for me.

On a 7.1-12 PVE cluster, 3 nodes, full flash ceph storage.
3 VMs. Windows 2019 VMs. Exactely the same, excepted saize of data disk.

2 of them have never had a problem during backup.
Third one has same issue (backup mode = snapshot), nealy 2 days on 3...

Same qemu-guest agent, same VirtIO drivers (215).

A "backup now" never triggers the problem.

A backup job will fail 2 times on 3...

Any idea?

Thanks,

Christophe.

mxscbv · May 10, 2022

The same problem on 7.2-3.

Taledo · May 16, 2022

I'm still seeing this error. This crashes the VM and sometimes even corrupts the VM FS. Do we have any information from the Proxmox Team regarding this bug?

To complete this message, the error seems to always happen on the same few VMs. Its either a Windows VM, a debian 9.5 or a older custom linux based box. The Debian VM is 2TB which might be an additional cause?

nmmn · May 16, 2022

Hi, we have same issue with one VM, since the Proxmox Update (7.2)

127: 2022-05-15 16:49:52 ERROR: VM 127 qmp command 'query-pbs-bitmap-info' failed - got timeout
127: 2022-05-15 16:49:52 INFO: aborting backup job
127: 2022-05-15 16:49:55 INFO: resuming VM again
127: 2022-05-15 16:49:55 ERROR: Backup of VM 127 failed - VM 127 qmp command 'query-pbs-bitmap-info' failed - got timeout

fiona · May 17, 2022

Hi,
for those of you using a Ceph storage without krbd, please see this thread for more information and workarounds.

Taledo · May 18, 2022

Hello Fabian,

Sadly I'm not using CEPH storage.

fireon · Jun 6, 2022

Same here, but only with one VM.

Backup to PBS failed - qmp command 'query-pbs-bitmap-info' failed - got timeout

Well-Known Member

Proxmox Staff Member

Well-Known Member

Well-Known Member

Member

Member

Renowned Member

New Member

New Member

Active Member

Member

Renowned Member

Renowned Member

Renowned Member

Member

Member

Renowned Member

Proxmox Staff Member

Member

Distinguished Member