MSIX PBA outside of specified BAR | Adding GPU PCI lane (no vGPU) to machine

[robseb]

Hi Everybody,

we have a Dell PowerEdge R7525 with 2x AMD EPYC 74F3 24-Core Processor.
Inside this machine are two Nvidia A16 Grapic Cards.
On this PVE we want to run 8 virtual machines. Every machine should have one pci lane assigned (this config we had with vmware).

Every GPU has 4 PCI addresses:




What things did we check?:

- IOMMU activated

- In /etc/modules we added following lines:
- vfio
- vfio_iommu_type1
- vfio_virqfd
- vfio_pci


- In etc/modprobe.d/pve-blacklist.conf we added following lines:
- nouveau
- nvidia
- nvidiafb



The machine is configured like that:



Now lets get to the problem:

Whenever we add a pci lane of the graphics card, the following error occurs:

Task viewer: VM 133 - Start


kvm: -device vfio-pci,host=0000:29:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0: vfio 0000:29:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR
TASK ERROR: start failed: QEMU exited with code 1

Does anyone have any ideas or suggestions as to what could be causing this?
Many thanks in advance!

 

[markus1234]

Grub configurations looks now like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

After this, we can start the vm with the PCI(GPU) device:

echo 1 > /sys/bus/pci/devices/0000\:29\:00.0/remove
 
echo 1 > /sys/bus/pci/rescan

 

[dcsapak]

hi,
maybe there is also a bios setting for the bar size? e.g. somebody here solved it with such a setting:

V

Thread 'PCIe passtrough problems - MSIX PBA outside of specified BAR'

Jan 28, 2024

Hi, so I have spend whole day trying to find a solution.

Running Proxmox Virtual Environment 7.3-3

The setup is a Supermicro AOC-S2308L-L8i (9207-8I) RAID that shows up as LSI MegaRAID SAS 2108 - attached over a Thunderbolt 3 dock JHL7540.

There is a error when starting a VM, where I have passed though the RAID.

kvm: -device vfio-pci,host=0000:07:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:07:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR

Generally, it seems there is a problem with setting BAR memory size. There is also some overlap...

• vanarebane
• Replies: 2
• Forum: Proxmox VE: Installation and configuration

 

[markus1234]

hi,
maybe there is also a bios setting for the bar size? e.g. somebody here solved it with such a setting:

V

Thread 'PCIe passtrough problems - MSIX PBA outside of specified BAR'

Jan 28, 2024

Hi, so I have spend whole day trying to find a solution.

Running Proxmox Virtual Environment 7.3-3

The setup is a Supermicro AOC-S2308L-L8i (9207-8I) RAID that shows up as LSI MegaRAID SAS 2108 - attached over a Thunderbolt 3 dock JHL7540.

There is a error when starting a VM, where I have passed though the RAID.

kvm: -device vfio-pci,host=0000:07:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:07:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR

Generally, it seems there is a problem with setting BAR memory size. There is also some overlap...

• vanarebane
• Replies: 2
• Forum: Proxmox VE: Installation and configuration

Thank you.
Unfortunately we can not change the bus size.
The core problem is that the Nvidia GPU is in use from the system.

So the solution or workaround (thx Michi) :
Add following settings:

/etc/modules

vfio
vfio_iommu_type1
vfio_virqfd
vfio_pci

/etc/modprobe.d/pve-blacklist.conf

blacklist nouveau
blacklist nvidia
blacklist nvidiafb

/etc/default/grub

#for AMD
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pci=realloc=off"

#for Intel
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pci=realloc=off"

update-grub

Create a systemd service:
/etc/systemd/system/nvidia-rescan.service (this fires BEFORE Proxmox is starting VMs)

[Unit]
Description=Remove Nvidia GPU from Devices and Rescan
After=multi-user.target
Before=pve-guests.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo 1 > /sys/bus/pci/devices/0000:29:00.0/remove && \
                        echo 1 > /sys/bus/pci/devices/0000:2a:00.0/remove && \
                        echo 1 > /sys/bus/pci/devices/0000:2b:00.0/remove && \
                        echo 1 > /sys/bus/pci/devices/0000:2c:00.0/remove && \
                        echo 1 > /sys/bus/pci/devices/0000:85:00.0/remove && \
                        echo 1 > /sys/bus/pci/devices/0000:86:00.0/remove && \
                        echo 1 > /sys/bus/pci/devices/0000:87:00.0/remove && \
                        echo 1 > /sys/bus/pci/devices/0000:88:00.0/remove && \
                        echo 1 > /sys/bus/pci/rescan'
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

Enable Service

systemctl enable nvidia-rescan.service

#systemctl start nvidia-rescan.service
#systemctl status nvidia-rescan.service