Nvidia PCIE Passthrough on Ubuntu VM returning "Unable to determine the device handle for GPU 0000:01:00.0: Unknown Error"

[Maxi1134]

Hi guys!I hope you are all doing great today.

I am in the process of trying to pipe a 1070TI to my Plex VM in order to do transcoding.

I initially followed this guide https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough
And managed to add the PCI device in the proxmox GUI https://i.imgur.com/I3lBx2L.png

Installed the latest NVIDIA-drivers that are compatible with (https://github.com/keylase/nvidia-patch .

But when running "nvidia-smi" I get the following message:

"Unable to determine the device handle for GPU 0000:01:00.0: Unknown Error"


I did spend about 5 hours trying to google-fu my way out of it, but alas I am unable to find any solution for Ubuntu. The closest I find is some thread for an Error 43 on Windows 10.

I guess that my question really is:
How "TF" can I fix this?


Thanks for reading!

\----------------------------------------------------------------------------------------------------------------------------------------------

Config files:Grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff"
GRUB_CMDLINE_LINUX=""

VMID.conf:

args: -machine type=q35,kernel_irqchip=on
bootdisk: scsi0
cores: 2
cpu: host
hostpci0: 03:00,pcie=1,rombar=0
ide0: PlexDataOne:vm-106-disk-0,backup=0,size=7452G
machine: q35
memory: 32000
name: Plex
net0: virtio=96:5A:B7:1D:A79,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
sata0: PlexDataTwo:vm-106-disk-0,backup=0,size=7452G
scsi0: local-lvm:vm-106-disk-0,size=400G
scsihw: virtio-scsi-pci
smbios1: uuid=ac3e61b8-ba37-4bb0-b03a-b1e1867d8335
sockets: 4
vmgenid: 9502be1c-8e43-41d6-9875-9d90bba5d67f

/etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1b82,10de:10f0 disable_vga=1



Command results:

lspci -v: https://hastebin.com/epakopuwop.sql

 

[r.jochum]

You could try my Error 43 fix maybe the linux nvidia driver is now the same shitty thing as the Windows driver (Disallowing Passthrough).

Make sure to use the correct ROM for you 1070Ti

 

[Maxi1134]

You could try my Error 43 fix maybe the linux nvidia driver is now the same shitty thing as the Windows driver (Disallowing Passthrough).

Make sure to use the correct ROM for you 1070Ti

Unfortunately, I will be returning the GPU if I don't manage to achieve this. I cannot afford to brick it by touching the Bios of the card.

 

[r.jochum]

Hmm you boot bios?

Try to change that:

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,kvm=off'
bios: ovmf

The Important parts for Windows are: "args -cpu 'host,+hv_vendor_id=proxmox,kvm=off'" I think mine would work with just these.

But OS/Plesk needs to know howto boot UEFI

 

[Maxi1134]

Hmm you boot bios?

Try to change that:

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,kvm=off'
bios: ovmf

The Important parts for Windows are: "args -cpu 'host,+hv_vendor_id=proxmox,kvm=off'" I think mine would work with just these.

But OS/Plesk needs to know howto boot UEFI

I read that KVM=OFF really destroys the performances, is that any true?

As for the second part. I am running a Linux VM not an ubuntu one.
I am trying to use the GPU cores to transcode in plex

 

[r.jochum]

I know about Linux I just assume that both have the same bug (intodruced by nvidia) atm.

Seems you'r right with "kvm=off" is bad, for me it works good enough.

This guy also had to do kvm=off.

 

[Maxi1134]

I know about Linux I just assume that both have the same bug (intodruced by nvidia) atm.

Seems you'r right with "kvm=off" is bad, for me it works good enough.

This guy also had to do kvm=off.

I've added


args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,kvm=off'


And it now works. Nvidia-smi returns the driver and everything. I'll see how the performances are.

Hopefully someone has a way to get full performances while passing through an Nvidia GPU

 

[r.jochum]

I could try to benchmark my Windows VM with/without kvm=off

 

[btyeung]

Confirmed it worked for me as well. Nvidia GTX 1650 Ti, adding "args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,kvm=off'" as above.

Thanks a ton!!!

 

[Joris L.]

kvm=off should really be a last resort
just in case, you can only use one GPU with one VM at a time, somehow i had the impression you were trying to do 2 VM 1 GPU

considering you appear to use an amd chipset (x3?0 x4?0 x5?0 ?)

you could try, consider to put iommu first instead of amd_iommu

iommu=pt amd_iommu=forced_isolation nobar noats video=vsafb:off video=efifb:off pci=noats,pcie_bus_perf,pcie_scan_all,big_root_window pcie_ports=native

for /etc/modules consider

amd_iommu_v2

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

msr

The bios for your card may help, you can get one here, worked for me (Radeon VII)

https://www.techpowerup.com/vgabios...X+1070+Ti&interface=&memType=&memSize=&since=

 

[joulester]

I still have this problem today and can´t get i to work without kvm=off.
Did someone find a solution?

 

[Joris L.]

I still have this problem today and can´t get i to work without kvm=off.
Did someone find a solution?

Having had a quick look at https://www.kernel.org/doc/html/v5.10/admin-guide/kernel-parameters.html i did not find kvm=off should have impact on kvm, it is most likely an nvidia specific boot parameter. If you do not experience performance issues and passthrough works, stick to it.

 

[nitrag]

Confirmed it worked for me as well. Nvidia GTX 1650 Ti, adding "args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,kvm=off'" as above.

Thanks a ton!!!

Weird, I hadn't been running passthrough in 12+ months and I tried to again and was having issues until I ran this command. I guess the VM's kernel got updated. Thanks. Nvidia 2070S