ERROR: Backup of VM 202 failed - unable to find VM '202'

[proxwolfe]

So I have been seeing errors in the Tasks pane of my PVE gui recently.

Combing through the log I found the following line in there:

ERROR: Backup of VM 202 failed - unable to find VM '202'

The thing is: While it is correct that there is no VM 202 (and also no CT 202), no backup scheduled for such VM (or CT) in the first place.

So where does this error message come from?

Thanks!

 

[proxwolfe]

Just to clarify:

There probably was a VM 202 at some point that I removed.

So for some reason, PVE still thinks that it should back it up although it does not list a backup job for it that I could remove.

 

[fiona]

Hi,
do you get anything when you grep 202 /etc/pve/vzdump.cron /etc/pve/jobs.cfg? Otherwise, please post the contents of these two files and your pveversion -v.

 

[proxwolfe]

Hi,

no, there is no output.

Here are the two files:

GNU nano 5.4 /etc/pve/vzdump.cron # cluster wide vzdump cron schedule # Automatically generated file - do not edit PATH="/usr/sbin:/usr/bin:/sbin:/bin"

GNU nano 5.4 /etc/pve/jobs.cfg vzdump: 9d958d4757f3a0151b17ac2e3b40db550d0e9815:1 schedule 01:00 enabled 1 mailnotification always mode snapshot quiet 1 storage tx13301.pbs.store2 vmid 206,200,201,203,204,205,106,101,102,103,104,300,107,109,301,105,400,401,402,403,207

For completeness sake here the error message in context:

INFO: Finished Backup of VM 201 (00:07:36) INFO: Backup finished at 2022-03-05 11:11:38 ERROR: Backup of VM 202 failed - unable to find VM '202' INFO: Failed at 2022-03-05 11:11:38 INFO: Starting Backup of VM 203 (lxc)

 

[fiona]

Is the node part of a cluster? When was the ID 202 removed from the list? Does running systemctl reload-or-restart pvescheduler.service help?

 

[proxwolfe]

No cluster.

Not sure when it was removed, but we're talking about several weeks minimum.

Will try to reload the pvescheduler and report back in a couple of days. But I am not overly optimistic, as I had to restart the server a few days ago (which would have caused the scheduler to reload as well) and the issue persists.

Thanks!

 

[fiona]

Sounds like a very strange bug then. Could you also share the beginning of the backup log (part before backup of the first VM starts) and post the output of pveversion -v? You might also want to check the installed files with

debsums -s pve-manager qemu-server pve-container libpve-guest-common-perl pve-cluster

just to be sure. debsums can be installed with the package of the same name and no output means everything is as expected.

 

[jrucker]

I ran into this problem as well.
I removed a few VMs over the weekend, and another one yesterday. The backup job runs nightly, and has errored on each one of the VMs I removed.

I'm not using a cluster.
I also did a reboot of the entire system yesterday, with no change.

I originally was using the "Selected VMs" option, and not "All VMs" in the backup config GUI page.
Before I posted here this morning, I noticed that, and changed it to "ALL VMs" as the VMs I wasn't backing up were deleted over the weekend.
/etc/pve/jobs.cfg does not list the VM IDs like @proxwolfe's does in the file he pasted, I'm assuming because I changed to the "ALL VMs" config.
I'll report back tomorrow to let you know if changing to "ALL VMs" has stopped the error

Here's the output from my pveversion -v :

proxmox-ve: 7.1-1 (running kernel: 5.13.19-5-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-8
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-7
pve-kernel-5.13.19-5-pve: 5.13.19-13
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

 

[fiona]

I ran into this problem as well.
I removed a few VMs over the weekend, and another one yesterday. The backup job runs nightly, and has errored on each one of the VMs I removed.

I'm not using a cluster.
I also did a reboot of the entire system yesterday, with no change.

I originally was using the "Selected VMs" option, and not "All VMs" in the backup config GUI page.
Before I posted here this morning, I noticed that, and changed it to "ALL VMs" as the VMs I wasn't backing up were deleted over the weekend.
/etc/pve/jobs.cfg does not list the VM IDs like @proxwolfe's does in the file he pasted, I'm assuming because I changed to the "ALL VMs" config.

Currently there is a bug (we are working on it) that VM IDs are not removed from the backup job config even if purge is used. So if you didn't manually remove the IDs from the backup job, the error is expected in a sense. But in @proxwolfe 's case the ID isn't in the job config anymore which is very strange.

I'll report back tomorrow to let you know if changing to "ALL VMs" has stopped the error

Thanks.

 

[jrucker]

Thanks.

Backup ran last night without erroring.
Switching to "all" could be a decent temporary workaround, but obviously isn't ideal.

 

[jrucker]

Ran into another related issue. Creating a new VM with the same ID as a previously deleted VM fails. See screenshot

 

[jrucker]

Ran into another related issue. Creating a new VM with the same ID as a previously deleted VM fails. See screenshot

Rebooting the physical machine caused this error to go away.