Proxmox 8.0 vs 7.4: can no longer pass through PCIE

G

Grunchy

New Member
Apr 17, 2023
7
2
3
Ok I have a couple of these HP Proliant servers (DL380P Gen.8) both updated with latest system ROM "P70 05/24/2019"

They each have 2x E5-2697 V2, 256 GB DDR3, and a Tesla P4 apiece (trying to do vGPU).

Proxmox 7.4, IOMMU works great!
# dmesg | grep -e DMAR -e IOMMU [ 0.007887] ACPI: DMAR 0x00000000BDDAD200 0004E0 (v01 HP ProLiant 00000001 \xd2? 0000162E) [ 0.007945] ACPI: Reserving DMAR table memory at [mem 0xbddad200-0xbddad6df] [ 0.665910] DMAR: IOMMU enabled <---------------- observe ! [ 1.549539] DMAR: Host address width 46 [ 1.549540] DMAR: DRHD base: 0x000000fbefe000 flags: 0x0 [ 1.549547] DMAR: dmar0: reg_base_addr fbefe000 ver 1:0 cap d2078c106f0466 ecap f020de [ 1.549550] DMAR: DRHD base: 0x000000dbffe000 flags: 0x1 [ 1.549554] DMAR: dmar1: reg_base_addr dbffe000 ver 1:0 cap d2078c106f0466 ecap f020de [ 1.549556] DMAR: RMRR base: 0x000000bdffd000 end: 0x000000bdffffff [ 1.549558] DMAR: RMRR base: 0x000000bdff6000 end: 0x000000bdffcfff [ 1.549559] DMAR: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff [ 1.549561] DMAR: RMRR base: 0x000000bdf7f000 end: 0x000000bdf82fff [ 1.549561] DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff [ 1.549563] DMAR: RMRR base: 0x000000bdf6e000 end: 0x000000bdf6efff [ 1.549564] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff [ 1.549565] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff [ 1.549566] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes [ 1.549633] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000000e8000-0x00000000000e8fff] [ 1.549636] DMAR: RMRR base: 0x000000bddde000 end: 0x000000bdddefff [ 1.549637] DMAR: ATSR flags: 0x0 [ 1.549640] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbefe000 IOMMU 0 [ 1.549642] DMAR-IR: IOAPIC id 8 under DRHD base 0xdbffe000 IOMMU 1 [ 1.549643] DMAR-IR: IOAPIC id 0 under DRHD base 0xdbffe000 IOMMU 1 [ 1.549645] DMAR-IR: HPET id 0 under DRHD base 0xdbffe000 [ 1.549646] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit. [ 1.549648] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting. [ 1.550485] DMAR-IR: Enabled IRQ remapping in xapic mode [ 2.086339] DMAR: No SATC found [ 2.086345] DMAR: dmar0: Using Queued invalidation [ 2.086354] DMAR: dmar1: Using Queued invalidation [ 2.203446] DMAR: Intel(R) Virtualization Technology for Directed I/O

Proxmox 8.0, IOMMU not happening.
# dmesg |grep -e DMAR -e IOMMU [ 0.006598] ACPI: DMAR 0x00000000BDDAD200 0004E0 (v01 HP ProLiant 00000001 \xd2? 0000162E) [ 0.006655] ACPI: Reserving DMAR table memory at [mem 0xbddad200-0xbddad6df] [ 2.080484] DMAR: Host address width 46 [ 2.080486] DMAR: DRHD base: 0x000000fbefe000 flags: 0x0 [ 2.080493] DMAR: dmar0: reg_base_addr fbefe000 ver 1:0 cap d2078c106f0466 ecap f020de [ 2.080496] DMAR: DRHD base: 0x000000dbffe000 flags: 0x1 [ 2.080506] DMAR: dmar1: reg_base_addr dbffe000 ver 1:0 cap d2078c106f0466 ecap f020de [ 2.080508] DMAR: RMRR base: 0x000000bdffd000 end: 0x000000bdffffff [ 2.080511] DMAR: RMRR base: 0x000000bdff6000 end: 0x000000bdffcfff [ 2.080512] DMAR: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff [ 2.080513] DMAR: RMRR base: 0x000000bdf7f000 end: 0x000000bdf82fff [ 2.080514] DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff [ 2.080515] DMAR: RMRR base: 0x000000bdf6e000 end: 0x000000bdf6efff [ 2.080516] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff [ 2.080517] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff [ 2.080519] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes [ 2.080588] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000000e8000-0x00000000000e8fff] [ 2.080590] DMAR: RMRR base: 0x000000bddde000 end: 0x000000bdddefff [ 2.080591] DMAR: ATSR flags: 0x0 [ 2.080596] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbefe000 IOMMU 0 [ 2.080598] DMAR-IR: IOAPIC id 8 under DRHD base 0xdbffe000 IOMMU 1 [ 2.080599] DMAR-IR: IOAPIC id 0 under DRHD base 0xdbffe000 IOMMU 1 [ 2.080600] DMAR-IR: HPET id 0 under DRHD base 0xdbffe000 [ 2.080602] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit. [ 2.080603] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting. [ 2.081405] DMAR-IR: Enabled IRQ remapping in xapic mode

On Proxmox 7.4 here is /etc/default/grub
GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` #GRUB_CMDLINE_LINUX_DEFAULT="quiet" GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on" GRUB_CMDLINE_LINUX=""

On Proxmox 8.0 here is /etc/default/grub
GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init intremap=no_x2apic_optout" GRUB_CMDLINE_LINUX=""

Obviously, /etc/modules
vfio vfio_iommu_type1 vfio_pci vfio_virqfd
 
Last edited:
  • Like
Reactions: Maximiliano and Max Carrara
I tried a few more commands to see what happens:

# lsmod | grep vfio vfio_pci 16384 0 vfio_pci_core 94208 1 vfio_pci irqbypass 16384 2 vfio_pci_core,kvm vfio_iommu_type1 49152 0 vfio 57344 3 vfio_pci_core,vfio_iommu_type1,vfio_pci iommufd 73728 1 vfio # modprobe vfio # modprobe vfio_iommu_type1 # modprobe vfio_pci # modprobe vfio_virqfd modprobe: FATAL: Module vfio_virqfd not found in directory /lib/modules/6.2.16-3-pve # dmesg | grep -i -e vfio [ 8.653198] VFIO - User Level meta-driver version: 0.3 # cat /lib/modules/$(uname -r)/modules.builtin | grep vfio (nothing)
 
Last edited:
I kinda gave up on 8.0, I've reverted back to PVE 7.4.

Another symptom I had trying out PVE 8.0 was when I went to install the nVidia GRID driver v16.0 (these commands worked perfectly on PVE 7.4 and GRID driver v15.3).
# chmod +x NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run # ./NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run --dkms ERROR: Unable to find the kernel source tree for the currently running kernel. Please make sure you have installed the kernel source files for your kernel and that they are properly configured; on Red Hat Linux systems, for example, be sure you have the 'kernel-source' or 'kernel-devel' RPM installed. If you know the correct kernel source files are installed, you may specify the kernel source path with the '--kernel-source-path' command line option.

I tried this:
# apt install pve-headers-6.2 Reading package lists... Done Building dependency tree... Done Reading state information... Done pve-headers-6.2 is already the newest version (8.0.3). pve-headers-6.2 set to manually installed. 0 upgraded, 0 newly installed, 0 to remove and 29 not upgraded.

Eventually, this was the only way to install the nVidia driver:
# ./NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run --dkms --kernel-source-path /usr/src/linux-headers-6.2.16-4-pve/
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back