yes, the way backup for stopped vms works is that it has to start a vm process in 'stop' mode (nothing is actually executed) and the stopped again
when the mdev is already cleaned up.. maybe it's just an oversight on their part?i have error Failed to destroy vGPU,
https://www.youtube.com/watch?v=_36yNWw_07g&t=6s
I think the issue is as previously mentioned, it depends on driver version. But I am out of my depth here.
Did you roll back the driver in the VM to a 510 branch driver or the equivalent guest driver?
there is a way you can acquire a license for dev and "testing" of course. dm and I will provide some info/instructions
I noticed the same on my end. Rebooting the VM does indeed clear the error. I got puzzled for a bit when cloning VMs trying to find why they would all BSOD till I figured that stopping/restarting them solved the issue.
echo 1 > /sys/bus/mdev/devices/$UUID/remove. We haven't viewed more stack-trace errors in dmesg logs./usr/share/perl5/PVE/QemuServer.pmsub cleanup_pci_devices {
my ($vmid, $conf) = @_;
sleep(15); # Added by me, credits to xenon96
foreach my $key (keys %$conf) {
next if $key !~ m/^hostpci(\d+)$/;
my $hostpciindex = $1;
my $uuid = PVE::SysFSTools::generate_mdev_uuid($vmid, $hostpciindex);
my $d = parse_hostpci($conf->{$key});
if ($d->{mdev}) {
# sleep(5); # Modification by Xenon96
# NOTE: avoid PVE::SysFSTools::pci_cleanup_mdev_device as it requires PCI ID and we
# don't want to break ABI just for this two liner
my $dev_sysfs_dir = "/sys/bus/mdev/devices/$uuid";
PVE::SysFSTools::file_write("$dev_sysfs_dir/remove", "1") if -e $dev_sysfs_dir;
}
}
PVE::QemuServer::PCI::remove_pci_reservation($vmid);
}Is a typo in me comment, for caution I put sleep(15) in the code.
nvidia-smi on that node, getting the PID of the vGPU process, and killing it manually./usr/share/perl5/PVE/QemuServer.pm, shutting down a VM from the PVE GUI properly terminates the vGPU process. I will update if I encounter any issues. Thanks to @xenon96 and @aslopez_irontec for the temporary fix!sub cleanup_pci_devices {
my ($vmid, $conf) = @_;
foreach my $key (keys %$conf) {
next if $key !~ m/^hostpci(\d+)$/;
my $hostpciindex = $1;
my $uuid = PVE::SysFSTools::generate_mdev_uuid($vmid, $hostpciindex);
my $d = parse_hostpci($conf->{$key});
if ($d->{mdev}) {
# NOTE: avoid PVE::SysFSTools::pci_cleanup_mdev_device as it requires PCI ID and we
# don't want to break ABI just for this two liner
my $dev_sysfs_dir = "/sys/bus/mdev/devices/$uuid";
# some nvidia vgpu driver versions want to clean the mdevs up themselves, and error
# out when we do it first. so wait for 10 seconds and then try it
my $pciid = $d->{pciid}->[0]->{id};
my $info = PVE::SysFSTools::pci_device_info("$pciid");
if ($info->{vendor} eq '10de') {
sleep 10;
}
PVE::SysFSTools::file_write("$dev_sysfs_dir/remove", "1") if -e $dev_sysfs_dir;
}
}
PVE::QemuServer::PCI::remove_pci_reservation($vmid);
}Hi.
).