Backup failed on stopped VM that shares PCI device

[BenediktS]

We have two VMs that share a PCI device. VM 192 and 194.
These two VMs are never online at the same time. When we need a special programm we stop VM 194 and start 192.
So this VM can use the PCI device. When we are finished, we stop the VM 192 and start the VM 194 again.

When we try to backup the VMs we get the error.

ERROR: Backup of VM 192 failed - PCI device '0000:01:00.0' already in use by VMID '194'

That's true..... But the VM 192 does not run. So why do you care if the PCI device is in use? You are not making a backup of the PCi device.
You should b eable to make a backup of the stopped VM 192 ,even if the PCI device is in use with another VM.

INFO: Starting Backup of VM 192 (qemu)
INFO: Backup started at 2023-12-22 04:08:52
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: vsx-Vidya
INFO: include disk 'scsi0' 'VM_Festplatten_NVME:vm-192-disk-1' 1G
INFO: include disk 'scsi1' 'VM_Festplatten_NVME:vm-192-disk-3' 244199M
INFO: include disk 'scsi2' 'VM_Festplatten_NVME:vm-192-disk-4' 976763M
INFO: include disk 'efidisk0' 'VM_Festplatten_NVME:vm-192-disk-0' 528K
INFO: include disk 'tpmstate0' 'VM_Festplatten_NVME:vm-192-disk-2' 4M
INFO: snapshots found (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/192/2023-12-22T03:08:52Z'
INFO: starting kvm to execute backup task
ERROR: Backup of VM 192 failed - PCI device '0000:01:00.0' already in use by VMID '194'
INFO: Failed at 2023-12-22 04:08:52

 

[Dunuin]

So why do you care if the PCI device is in use?

Because a VM has to be running in order to back up its disks. And you can't start a VM that requires ressources that aren't available.

You should b eable to make a backup of the stopped VM 192 ,even if the PCI device is in use with another VM.

Technical limitation. Backups make use of QEMU features (snapshots are for example done on the QEMU and not on the ZFS/LVM-Thin/Qcow2 layer, dirty-bitmapping, ...) so QEMU needs to be running to be able to access the block devices and for that that the KVM process has to be running.

There are some work arounds usimg vzdump hook scripts. You could write a script that shutsdown both VM when the backups job starts and later start that VM, that got shutdown, after the backup job has finished. This is how I'm doing it.
Another option would be to write a script that will remove the pci passthroughed device from the not running VM when the backup job starts and adds it again once the job finiahed,

 

[BenediktS]

But is it really booting a stopped VM? Because i think that could also have unwanted side effects.
(And that would also be relevant for us to know if it is really starting the VM on backup)

Or is it just using the QEMU in some way where it is not booting but only (don't know how to name it) "enabling" a VM.
Because then it could use a config with only the disks enabled and not all the other stuff.

 

[Dunuin]

But is it really booting a stopped VM?

It won't boot the guestOS but still needs to run the VM.

Because then it could use a config with only the disks enabled and not all the other stuff.

You could open a feature request ticket for that at https://bugzilla.proxmox.com but until then you could do that yourself using hook scripts.

 

[N0AGI]

I realize this is a dated thread. But, wanted to follow up and ask whether there was any solution here since I have the same scenario.

 

[Dunuin]

I wrote a hook script as a workaround that will shutdown all VMs sharing a VM, then do the backup job and later start that VM again that got shutdown. That way the backup job won't fail and all VMs will be backed up. But you of cause got downtime as the running VM still needs to be shut down so the PCI device isn't in use.

 

[leesteken]

There is some work being done for templates with PCI(e) passthrough (and ignore the passthrough during backup): https://lists.proxmox.com/pipermail/pve-devel/2024-June/064238.html Maybe it could be extended to a user-settable option for other VMs as well (but you can't start the VM until the backup finishes)?

EDIT: Looks like they are considering it: https://bugzilla.proxmox.com/show_bug.cgi?id=3352