One VM won't backup to PBS (debian 11)

[Erk]

I have a Proxmox 7.0-10 server that has 3 VMs running on it, 2 are FreeBSD the other is Debian 11 running mostly email.

All 3 VMs were recently moved across from a 6.4.9 server to the new 7.0 server whilst the were shutdown, using ftp for the HD raw image and the .conf files, all 3 are running fine. I have done 3 backups of the Debian 11 mail server (VM 201) to a PBS 1.1-12 server this week which went fine.

Then I installed proxmox-backup-client on the Debian 11 VM and have been using it to back up /home and /var/mail to the pbs1 (PBS 1.1-12) server which is working fine.
Today I decided I wanted to back up the full VM to the same PBS server and I am getting the following error.

INFO: starting new backup job: vzdump 201 --node prox2 --remove 0 --mode snapshot --storage pbs1
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2021-07-23 09:44:14
INFO: status = running
INFO: VM Name: mail
INFO: include disk 'ide0' 'local-btrfs:201/vm-201-disk-0.raw' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/201/2021-07-22T23:44:14Z'
INFO: started backup task 'f498f72c-264f-4bc4-8972-ddd74eff28ec'
INFO: resuming VM again
ERROR: VM 201 qmp command 'cont' failed - got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 201 failed - VM 201 qmp command 'cont' failed - got timeout
INFO: Failed at 2021-07-23 09:44:23
INFO: Backup job finished with errors
TASK ERROR: job errors

I tried doing a "Backup Now" to a PBS 2.0-4 server and get a similar error.

The pbs2 (PBS 2.0-4) server had never received a backup from VM 201 (Debian 11 VM), or from the proxmox-backup-client so it would be a full new backup if it didn't give an error.

The other 2 VMs on the Proxmox 7 server back up fine to either of the PBS servers with no errors, it's just this one VM that refuses to back up. I have restarted it, tried qm unlock just in case. I don't understand what the 'cont' command is, I presume it's something to do with taking the snapshot.

One thing of note, VM 201 doesn't seem to stop when I log into the console and do a 'shutdown -P now' it seems to kill all the processes but the icon in the Proxmox GUI remains black. After a few minutes, I have to do a Stop from the GUI.

 

[Erk]

Update, ran out of ideas, so I rebooted the PVE 7.0 server and the VM is able to do backups to both PBS servers again.
I has rebooted the VM several times just in case something apt updated needed it.
So in the end it was the PVE 7.0 server reboot that cleared the problem.
The previous successful backup was at 2021-07-21T15:00:01Z and I checked the dpkg.log on the PVE 7.0 server and there had been no updates since that backup to explain what might have changed.

 

[Erk]

FreeBSD 11.4-RELESE VMs don't seem to be honoring the powerdown signals, they stop all the processes and site there. I wonder if that's why they wont backup to PBS? They will vzdump ok. In fact they won't even backup to PBS even in stop mode, the error message from:

qmp command 'cont' failed - got timeout
to
qmp command 'query-pbs-bitmap-info' failed - got timeout

 

[rynomite]

I have the same exact problem. I have a pfSense running FreeBSD 11.3 which cannot be backed up on PBS v2 but it can locally on the host PVE v7, and another running FreeBSD 12.2 pfSense 2.5.2.

The offending pfSense 2.4.5 was stuck thinking it was 'up to date' but actually it wasn't. Because the local package manager didn't update (probably some corruption being the source of the issue), I had to backup the config, install fresh pfSense (over top of the same VM disk), and import the config back in.

 

[rynomite]

I have the same exact problem. I have a pfSense running FreeBSD 11.3 which cannot be backed up on PBS v2 but it can locally on the host PVE v7, and another running FreeBSD 12.2 pfSense 2.5.2.

The offending pfSense 2.4.5 was stuck thinking it was 'up to date' but actually it wasn't. Because the local package manager didn't update (probably some corruption being the source of the issue), I had to backup the config, install fresh pfSense (over top of the same VM disk), and import the config back in.

Update to my post: Overwriting the same disk would not backup. I actually needed to create a new disk on the same VM, perform a fresh export of pfSense config (from the upgraded 2.5.2), install of pfSense using a fresh ISO of new 2.5.2 media, and then import the new backup file.

An interesting note the old disk was RAW format whereas the new one created as QCOW2. I probably imported that old disk (which was QCOW2) but Proxmox considered it RAW, so I wonder if that was a factor in the PBS initial backup commands which triggered the error.