[SOLVED] Nvidia L40S GPU PCI Pass-through BAR0 Error

A

amtiskaw

Member
Mar 11, 2021
13
1
23
58
Howdy
I have a Nvidia L40s that I'm passing through to an Ubuntu 24.04 VM
Proxmox 9.0.10 running on 2 PowerEdge R760s, with a qdevice.

At boot, the VM console loops this error:

Code: 
NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
NVRM: BAR0 is 0M @ 0x0 (PCI:0000:01:00.0)

I don't know why it's referring to PCI:0000:01:00.0 as that's a NIC. The GPU is 0000:b5:00.0

The conf for the VM confirms it's using the GPU

Code: 
hostpci0: 0000:b5:00,pcie=1

pvesh get /nodes/hn-pve-1/hardware/pci --pci-class-blacklist "" confirms I have the PCI IDs correct

Code: 
 0x020000 │ 0x165f │ 0000:01:00.0 │         41 │ 0x14e4 │ NetXtreme BCM5720 Gigabit Ethernet PCIe
 0x030200 │ 0x26b9 │ 0000:b5:00.0 │         25 │ 0x10de │ AD102GL [L40S]

What am I doing wrong here?
 
Last edited:
The error messages suggest that the guest driver can’t access the GPU’s memory space and is falling back to 0x0.

Please check and try... make sure your VM uses machine: q35 with pcie=1, and that “Above 4G Decoding” and “Resizable BAR” are enabled in BIOS.

Confirm vfio-pci is bound to 0000:b5:00.0 and not to the Broadcom NIC at 0000:01:00.0.

If you’re using Dell R760, disable SR-IOV temporarily to avoid ACS remapping issues.

Reboot host and VM.

Fingers crossed.
 
The fix turned out to be using seaBIOS and no Q35.

J

Thread 'Cannot Passthrough H100 PCIE GPU to VMs'

I'm new to the Proxmox Forum,but i already use PVE for more than three years.I'm sure that I've read the basic docs like
  1. https://pve.proxmox.com/wiki/PCI_Passthrough#Enable_the_IOMMU
  2. https://pve.proxmox.com/wiki/PCI_Passthrough#Required_Modules
  3. https://pve.proxmox.com/wiki/PCI_Passthrough#IOMMU_Interrupt_Remapping
  4. https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_Isolation
  5. I'm working on a beast of a server with:
    • 144 x Intel(R) Xeon(R) Platinum 8452Y (2 Sockets)
    • 256G ram
    • H100 PCIe GPU card
My goal is passthrough the...
 
  • Like
Reactions: readyspace
You must log in or register to reply here.
Top Bottom