for anyone having this issue run this and reboot:
proxmox-boot-tool kernal pin 6.8.12-15-pve
proxmox-boot-tool kernal pin 6.8.12-15-pve
Proxmox VE 9 PCIe Passthrough TrueNAS VM
- Thread starter Bribbo38
- Start date
I too ran into this issue passing through an LSI 9305-16i HBA to an OpenMediaVault VM after upgrading from Proxmox 8 to 9, where PCI passthrough worked great in v8, but failed in v9.
I updated the HBA firmware and bios (an endeavor), that didn't resolve the PCI passthrough failure.
I have the following kernel options set for `proxmox-boot-tool`: `intel_iommu=on pcie_acs_override=downstream,multifunction iommu=pt` and I have the following options set on the PCI Device passthrough options (a solution suggested above): `pcie=1,rombar=0`. However, these options didn't resolve the issue. As @FuzzyLobster and others mentioned above, the issue appears to be related to the kernel version. Kernel version 6.14.11-4-pve failed, but when I pinned to 6.8.12-15-pve, passthrough works again. Thanks all.
I updated the HBA firmware and bios (an endeavor), that didn't resolve the PCI passthrough failure.
I have the following kernel options set for `proxmox-boot-tool`: `intel_iommu=on pcie_acs_override=downstream,multifunction iommu=pt` and I have the following options set on the PCI Device passthrough options (a solution suggested above): `pcie=1,rombar=0`. However, these options didn't resolve the issue. As @FuzzyLobster and others mentioned above, the issue appears to be related to the kernel version. Kernel version 6.14.11-4-pve failed, but when I pinned to 6.8.12-15-pve, passthrough works again. Thanks all.
Last edited:
This is the exact issue I am having. Fresh install of proxmox 9, joined to existing cluster - the other node was upgraded.
I have decide to give it a go and install TrueNAS scale but wanted to ensure proxmox can run off of asmedia m.2 to 6 SATA mini-pcie card.
MW-N100-NAS motherboard has 6 sata slots but it does not seem they all run off of one controller:
class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsystem_devic
╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════════════════════════════╪══════╪════════════════
│ 0x010601 │ 0x54d3 │ 0000:00:17.0 │ 5 │ 0x8086 │ Alder Lake-N SATA AHCI Controller │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼────────────────
│ 0x010601 │ 0x1166 │ 0000:01:00.0 │ 13 │ 0x1b21 │ ASM1166 Serial ATA Controller │ │ 0x2116
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼────────────────
│ 0x010601 │ 0x0585 │ 0000:05:00.0 │ 17 │ 0x197b │ JMB58x AHCI SATA controller │ │ 0x0000
So I have decided to wing it and try to pass both over .
however, after multiple reboots after many tries, it will still not work.
0. BIOS has vt-d enabled
1. Grub has intel IOMMU enabled with passthrough option
2 vfio module is written with device ids for both
3 /etc/modules are saved and they load:
root@SM1:~# lsmod | grep vfio
vfio_pci 16384 0
vfio_pci_core 86016 1 vfio_pci
irqbypass 12288 2 vfio_pci_core,kvm
vfio_iommu_type1 49152 0
vfio 65536 3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd 110592 1 vfio
4 initramfs was updated and system rebooted (yet again)
so I guess the AHCI gets loaded first - before vfio has a chance to claim them (or other way round - vfio being assigned to them)
but because I boot from asmedia which needs ahci I cannot blacklist the driver globally.
any ideas ? am I on the right track ? and before you suggest - AI IS USELESS with this deep level of troubles.
is there a workaround ?
this is a mini-itx board with no pci-e slots. it only has two m.2 slots, 6 sata ports, embedded cpu and crazy networking (10Gbpe + 2 x 2.5 Gbe).
so SAS HBA is out of the question.
I did order nvme drive - will see if I can install pve on it when it arrives and have free reign over AHCI and blacklist if needed. but does nvme use AHCI as a driver ?
Or do I chalk it up to newest kernel known issue ?
I have checked ROM bar - intel does not have it, JMicron has but it is disabled.
I have decide to give it a go and install TrueNAS scale but wanted to ensure proxmox can run off of asmedia m.2 to 6 SATA mini-pcie card.
MW-N100-NAS motherboard has 6 sata slots but it does not seem they all run off of one controller:
class │ device │ id │ iommugroup │ vendor │ device_name │ mdev │ subsystem_devic
╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════════════════════════════╪══════╪════════════════
│ 0x010601 │ 0x54d3 │ 0000:00:17.0 │ 5 │ 0x8086 │ Alder Lake-N SATA AHCI Controller │ │ 0x7270
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼────────────────
│ 0x010601 │ 0x1166 │ 0000:01:00.0 │ 13 │ 0x1b21 │ ASM1166 Serial ATA Controller │ │ 0x2116
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼────────────────
│ 0x010601 │ 0x0585 │ 0000:05:00.0 │ 17 │ 0x197b │ JMB58x AHCI SATA controller │ │ 0x0000
So I have decided to wing it and try to pass both over .
however, after multiple reboots after many tries, it will still not work.
0. BIOS has vt-d enabled
1. Grub has intel IOMMU enabled with passthrough option
2 vfio module is written with device ids for both
3 /etc/modules are saved and they load:
root@SM1:~# lsmod | grep vfio
vfio_pci 16384 0
vfio_pci_core 86016 1 vfio_pci
irqbypass 12288 2 vfio_pci_core,kvm
vfio_iommu_type1 49152 0
vfio 65536 3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd 110592 1 vfio
4 initramfs was updated and system rebooted (yet again)
so I guess the AHCI gets loaded first - before vfio has a chance to claim them (or other way round - vfio being assigned to them)
but because I boot from asmedia which needs ahci I cannot blacklist the driver globally.
any ideas ? am I on the right track ? and before you suggest - AI IS USELESS with this deep level of troubles.
is there a workaround ?
this is a mini-itx board with no pci-e slots. it only has two m.2 slots, 6 sata ports, embedded cpu and crazy networking (10Gbpe + 2 x 2.5 Gbe).
so SAS HBA is out of the question.
I did order nvme drive - will see if I can install pve on it when it arrives and have free reign over AHCI and blacklist if needed. but does nvme use AHCI as a driver ?
Or do I chalk it up to newest kernel known issue ?
I have checked ROM bar - intel does not have it, JMicron has but it is disabled.
SOLUTION:
I have achieved this :
00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-N SATA AHCI Controller [8086:54d3]
DeviceName: Onboard - SATA
Subsystem: Intel Corporation Device [8086:7270]
Kernel driver in use: vfio-pci
Kernel modules: ahci
05:00.0 SATA controller [0106]: JMicron Technology Corp. JMB58x AHCI SATA controller [197b:0585]
Subsystem: JMicron Technology Corp. Device [197b:0000]
Kernel driver in use: vfio-pci
Kernel modules: ahci
but only after forcing vfio-pci to load before ahci driver loads.
This is achieved according to the longer, more complicated procedure of the two listed below:
github solution page
I am yet to test if I will be able to pass through now.
I have achieved this :
00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-N SATA AHCI Controller [8086:54d3]
DeviceName: Onboard - SATA
Subsystem: Intel Corporation Device [8086:7270]
Kernel driver in use: vfio-pci
Kernel modules: ahci
05:00.0 SATA controller [0106]: JMicron Technology Corp. JMB58x AHCI SATA controller [197b:0585]
Subsystem: JMicron Technology Corp. Device [197b:0000]
Kernel driver in use: vfio-pci
Kernel modules: ahci
but only after forcing vfio-pci to load before ahci driver loads.
This is achieved according to the longer, more complicated procedure of the two listed below:
github solution page
I am yet to test if I will be able to pass through now.
yeah, no dice.
PCI passthrough works (complains about rombar - I guess I will remove the option).
the vm won't boot from dvd (copied iso to vz templates iso folder) but it will pick up any EFI disks I have not yet wiped and boot from there.
I guess pci passthrough problem solved, another problem or m ore than one introduced.
it is 1am so bed time now , resume in the morning ...
After upgrading to proxmox 9 - I'm unable to passthrough my SATA controller (integrated Intel SATA controller from ASUS mainboard)
When i booted from kernel 6.8.12 - worked fine.
For everyone to know - just try to boot from older kernel. Adding `rombar=0` didn't work for me.
When i booted from kernel 6.8.12 - worked fine.
For everyone to know - just try to boot from older kernel. Adding `rombar=0` didn't work for me.
You might be onto something there!
Before upgrade from 8 -> 9, proxmox was booting from UEFI. And i know that for sure because in my BIOS, there is UEFI entry labeled: "proxmox".
After upgrading to 9, proxmox would not boot. Instead, i had to manually select nvme drive for legacy boot.
And I'm still having issues with my mainboard trying to switch-back automatically to UEFI after reboot, causing me pain in the a** for me having to hookup monitor and keyboard to change it.
So for some reason, proxmox broke UEFI boot, basically disabling Secure Boot.
I can confirm that this solved the issue with my ASM1166. The provided link is deprecated and refers to https://github.com/kiler129/early-vfio-pci-isolate which I used. Nevertheless I still had to set
Code:
pcie=1,rombar=0