[SOLVED] Unable to boot after trying to setup PCIe passthrough - PVE 8.0-2

[UltraHKR]

NOTE: Quick fix use package "driverctl", not the same as early binding but pretty close. Thanks @leesteken

Hello everyone

I have been beefing up my storage, so the configuration works properly on PVE 7.x, but it doesn't work on PVE 8.0-2 (I'm using proxmox-ve_8.0-2.iso)

My original HW config (lastest BIOS, FW, IPMI, etc):

1. Supermicro X8DTH-iF (no UEFI)
2. 2x Intel X5650
3. 192GB RAM
4. 2x Intel 82576 Gigabit NIC Onboard
5. 1TB SATA Boot riv on Intel AHCI controller
6. 1st Dell H310 (IT Mode Flashed using Fohdeesha guide) Unused
7. 2nd Dell H310 (IT Mode pass through to WinVM)
8. LSI 9206-16e (IT Mode Passthrough to TN Scale)

My HW config should on my signature, but I will post it here (lastest BIOS, FW, IPMI, etc):

1. Supermicro X8DTH-iF (no UEFI)
2. 192GB RAM
3. 2x Intel 82576 Gigabit NIC Onboard
4. 1st Dell H310 (IT Mode Flashed using Fohdeesha guide) Boot device
5. PVE Boot disks: 2x300GB SAS in ZFS RAID1
6. PVE VM Store: 4x 1TB SAS ZFS RAID0
7. 2nd Dell H310 (IT Mode pass through to WinVM)
8. LSI 9206-16e (IT Mode Passthrough to TN Scale)

I'm stumped i'm trying to do PCIe passthrough, I followed this guide:
PCI(e) Passthrough - Proxmox VE

The steps I followed:

• Changed PVE repositories to: “no-subscription”
• Added repositories to Debian: “non-free non-free-firmware”
• Updated all packages
• Installed openvswitch-switch-dpdk
• Install intel-microcode
• Reboot
• Setup OVS Bond + Bridge + 8256x HangUp Fix
• Modified default GRUB adding: “intel_iommu=on iommu=pt pcie_acs_override=downstream”
• Modified “/etc/modules”:

• vfio
  vfio_iommu_type1
  vfio_pci
  vfio_virqfd
  mpt2sas
  mpt3sas

• Ran "update-initramfs -u -k all" and "proxmox-boot-tool refresh"
• Reboot

Up to here it works fine, the machine comes back properly.

• Created “/etc/modprobe.d/vfio.conf”:
  options vfio_iommu_type1 allow_unsafe_interrupts=1
• Modified default GRUB adding: “ rd.driver.pre=vfio-pci"
• Ran "update-initramfs -u -k all" and "proxmox-boot-tool refresh"
• Reboot

Up to here it works fine, the machine comes back properly.

• Setup earlier VFIO according to (Post #4): PCI Passthrough Selection with Identical Devices | Proxmox Support Forum
• Created “/etc/initramfs-tools/scripts/init-top/vfio.sh”: (VFIO ONLY! on the 9206-16e)

• #!/bin/sh -e
  echo "vfio-pci" > /sys/devices/pci0000:80/0000:80:09.0/0000:86:00.0/0000:87:01.0/0000:88:00.0/driver_override
  echo "vfio-pci" > /sys/devices/pci0000:80/0000:80:09.0/0000:86:00.0/0000:87:09.0/0000:8a:00.0/driver_override
  modprobe -i vfio-pci

• Ran "update-initramfs -u -k all" and "proxmox-boot-tool refresh"
• Reboot

The machine boots, I get to the GRUB bootloader, and bam!

This is like my third reinstall, i have slowly trying to dissect where it goes wrong.
I have booted into the PVE install disk and the rpool loads fine, scrubs fine, etc...

Somewhere, somehow the grub / initramfs / boot config gets badly setup...

Can somebody help me out!?

Update: I'm doing something wrong tried on PVE 7.x (lastest) and I get to the same point...

Update #2: after removing every trace of VFIO, unloading zfs, mpt3sas and VFIO modules. Reloading mpt3sas & zfs at least the pool is imported.

Update #3: Booting from the old PVE 7.x (which was working), it boots to the same error, if I boot from the H310 SAS controller #1.


My OG reddit post for reference

 

[leesteken]

It sounds like you are preventing the host from accessing a drive controller that has the drives that you installed Proxmox on. I'm sorry but I have no experience with passthrough of one of multiple identical devices. Are you sure you are using the right one (and not both)? Would it not be much easier to use the LSI for Proxmox, and passthrough both H310 to different VMs?

 

[UltraHKR]

It sounds like you are preventing the host from accessing a drive controller that has the drives that you installed Proxmox on. I'm sorry but I have no experience with passthrough of one of multiple identical devices. Are you sure you are using the right one (and not both)? Would it not be much easier to use the LSI for Proxmox, and passthrough both H310 to different VMs?

My life would be so easy... If it was that easy

Those H310 are in fact LSI 9211-8i HBA's
That LSI 9206-16e is another LSI HBA, but connected to 2x JBOD (24x HDD's each)
NOTE: This one is 2x SAS controllers behind a PLX PCIe bridge, so in fact I have 4x LSI SAS HBA's

They all use mpt3sas kernel driver...

 

[leesteken]

My life would be so easy... If it was that easy

Those H310 are in fact LSI 9211-8i HBA's
That LSI 9206-16e is another LSI HBA, but connected to 2x JBOD (24x HDD's each)
NOTE: This one is 2x SAS controllers behind a PLX PCIe bridge, so in fact I have 4x LSI SAS HBA's

They all use mpt3sas kernel driver...

Right and it shows that I have no experience with this. It's clear you cannot blacklist the driver. Do you need to early bind them to vfio-pci (which appear to cause the boot issues)?

• Setup earlier VFIO according to (Post #4): PCI Passthrough Selection with Identical Devices | Proxmox Support Forum
• Created “/etc/initramfs-tools/scripts/init-top/vfio.sh”: (VFIO ONLY! on the 9206-16e)
• #!/bin/sh -e echo "vfio-pci" > /sys/devices/pci0000:80/0000:80:09.0/0000:86:00.0/0000:87:01.0/0000:88:00.0/driver_override echo "vfio-pci" > /sys/devices/pci0000:80/0000:80:09.0/0000:86:00.0/0000:87:09.0/0000:8a:00.0/driver_override modprobe -i vfio-pci
• Ran "update-initramfs -u -k all" and "proxmox-boot-tool refresh"
• Reboot

The machine boots, I get to the GRUB bootloader, and bam!

If the devices reset (FLR) properly then this should not be necessary and you can choose which ones to passthrough solely on the unique PCI-ID (instead of vendor:device-ID). Probably you already tried this. Maybe some of those PCI-IDs used for driver_override are incorrect and you are early binding the H310 that you need for Proxmox.

• Modified default GRUB adding: “intel_iommu=on iommu=pt pcie_acs_override=downstream”
• Modified “/etc/modules”:
• vfio vfio_iommu_type1 vfio_pci vfio_virqfd mpt2sas mpt3sas
• Ran "update-initramfs -u -k all" and "proxmox-boot-tool refresh"
• Reboot

You load vfio-pci via /etc/modules before running your script, so I'm not sure your script has any effect since modprobe detects that vfio-pci is already loaded.
Note that vfio_virqfd is no longer a separate module and no longer exists since PVE 8.

 

[UltraHKR]

Right and it shows that I have no experience with this. It's clear you cannot blacklist the driver. Do you need to early bind them to vfio-pci (which appear to cause the boot issues)?

I need early VFIO binding because (at least with one of the H310), mpt3sas loads and will not unlink from it...
Can't do pass-through without VFIO..........

If the devices reset (FLR) properly then this should not be necessary and you can choose which ones to passthrough solely on the unique PCI-ID (instead of vendor:device-ID). Probably you already tried this. Maybe some of those PCI-IDs used for driver_override are incorrect and you are early binding the H310 that you need for Proxmox.

I didn't use PCI-ID's trying to avoid the "chicken and egg" problem of VFIO vs mpt3sas loading order...
This problem is exacerbated because there are 2x devices (H310 same PCI-ID, different Bus ID) but one is the boot device.

You load vfio-pci via /etc/modules before running your script, so I'm not sure your script has any effect since modprobe detects that vfio-pci is already loaded.

I dont know enough about how Linux bootstraps itself to know which driver / script loads in what order...
Do you have something to get a better grasp on it (Linux boot process)?

Note that vfio_virqfd is no longer a separate module and no longer exists since PVE 8.

I didn't know about this, thanks

 

[leesteken]

I need early VFIO binding because (at least with one of the H310), mpt3sas loads and will not unlink from it...

RIght.

I didn't use PCI-ID's trying to avoid the "chicken and egg" problem of VFIO vs mpt3sas loading order...
This problem is exacerbated because there are 2x devices (H310 same PCI-ID, different Bus ID) but one is the boot device.

What you call Bus ID is what I meant with PCI-ID. Indeed, early binding with identical devices is a headache.

I dont know enough about how Linux bootstraps itself to know which driver / script loads in what order...
Do you have something to get a better grasp on it (Linux boot process)?

All modules mentions in /etc/modules are loaded at boot. Later modprobe's don't do anything unless you find rmmod the same module (in my experience). You also put mpt3sas in /etc/modules, so it will be loaded before your script also. I think that approach will not work.
Have you tried installing driverctl and configuring driver overrides for specific devices based on Bus ID (or what I call PCI-ID)?

 

[UltraHKR]

All modules mentions in /etc/modules are loaded at boot. Later modprobe's don't do anything unless you find rmmod the same module (in my experience). You also put mpt3sas in /etc/modules, so it will be loaded before your script also. I think that approach will not work.
Have you tried installing driverctl and configuring driver overrides for specific devices based on Bus ID (or what I call PCI-ID)?

@leesteken

Sorry for answering late, I'm extremely thankful for the "driverctl" package TIL.
(I owe you a beer, if you're ever in Quito, Ecuador)

It solve 70% of my problems... I did a quick test and it should work...

Now if I could fix VFIO early binding would be awesome, but I will leave that for another day...

Why? Proxmox sees the 44x HDD's that I use for TrueNAS Scale, and likes to use some of the unassigned space for (I think it uses LVM?) for swap...

Which makes TN Scale throw some very scary warnings...

I usually turn off my 2x JBOD's when doing heavy maintenance, but sometimes I forget and fixing it while easy its a little awful, my server takes 10min to restart.

So doing something as simple as boot Proxmox, (oh, snap! that...) rebooting into LiveCD, and booting into Proxmox again becomes a 45+ minutes ordeal... (Add the VM time start after that...)