This is my first experience with Proxmox. I am trying to use PCI passthrough on Proxmox for a PCI HBA card. My ultimate goal is a TrueNAS guest with a RAID-Z. The HBA card is in IT mode. I followed the passthrough guide on the wiki, and everything seemed to work fine until the very end. The results from the commands indicate PCI passthrough support. I can see serial numbers of invididual drives through Proxmox. Every indication is that the card is operating as a transparent JBOD. I can access the drives enough to see and alter partitions and run SMART tests through bash on the Proxmox host. I had no problem with configuring the passthrough. Until I tried to use the VM.
When I attach the mapped PCI HBA card, strange things happen. First, the VM goes through the BIOS loading step rather slowly. With it detached, I'd say it takes 2-3 seconds. When attached, it's closer to 20-30. I THINK the slowness is the PCI HBA card loading, but I don't really know how long this is supposed to take normally. I can see the PCI attached drives showing up in the boot sequence along with the card.
Now here's where it gets weird. With the mapped PCI attached, a virtual mounted ISO eventually boots (though again, BIOS is slow), and drives connected to PCI card are visible. However, booting from the virtual hard drive hangs indefinitely at "Booting from Hard Disk". It doesn't seem to matter what OS is installed or if there's no OS installed. It always hangs on "Booting from Hard Disk." I tried leaving it overnight and it never got unstuck.
The PCI card is NOT set as a boot device. First thing I checked. The virtual HD for the VM is on the same NVME as Proxmox itself, as intended.
Detaching the mapped PCI from the VM and booting that way immediately fixes the problem. VM BIOS no longer boots slowly, and doesn't hang on trying to boot from VHD.
/etc/modules has this.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
/etc/default/grub has this.
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream"
Anytime I make a change to the /etc/ files, I run:
update-grub
update-initramfs -u -k all
And also reboot Proxmox. Probably overkill but under the circumstances I want to be sure.
I tried several different setups here as found in various guides, including the official wiki, but haven't had any luck so far.
All hardward involved is new-ish (never used but sat on a shelf for several months before I had a chance to do the build).
Hardware
Motherboard: H13SAE-MF (AMD Ryzen CPU)
Drives: 4x WD Red 16TB
HBA card: Generic 9207-8i, preflashed for IT mode
For the card, Proxmox dashbaord shows:
Vendor: Broadcom/LSI
Device: SAS2308 PCI-Express Fusion-MPT SAS-2
dmesg shows:
[ 0.762198] mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05)
I could not find a more recent firmware version than that.
Things I have tried:
Tried a new fanout SATA cable. Tried the other fanout port on the card. Tried the other PCI-E on the board. No dice.
If I unplug the fanout cable from the HBA, the BIOS will still load slowly, but VM will eventually boot from VHD. I thought this might indicate a hard drive problem... so I tried plugging into each hard drive individually and testing like that. No VM boot. If it's a hard drive problem, it doesn't seem to be with any SINGLE drive.
I ran a long SMART test on all drives. No problems. Howeever, as mentioned, I think a DOA drive is unlikely at this point.
I looked into updating the hard drive firmware. According to WDs official support, they don't normally update drive firmware on Reds. There's some kind of proprietary Windows tool that can do it but I'd rather not screw with it unless there's no other choice.
Tried setting VM hardware in both BIOS and UEFI mode, and changing TrueNAS to match. As mentioned, it doesn't seem to actually matter what's installed on the virtual drive, including nothing. I also tried Debian instead of TrueNAS and got the same result. It doesn't seem to be a guest OS issue. In fact I don't think it's even getting that far. Booting from a VHD with no OS and the mapped PCI acttached hangs on "Booting from Hard Disk.", when I would expect it to say "No bootable device found."
I tried both i440fx and q35 machine types. Supposedly q35 is better for PCI passthrough? Also tried changing the advanced VIOMMU setting to "Intel - AMD compatible" as recommended based on some posts I read.
So far as I can tell, I don't see any sort of errors in any logs related to the VM. journalctl -b -e output is below.
At this point I suspect a problem with the HBA card, that it's either bad or somehow off-spec or something. But I wanted to verify that I've tried everything configuration-wise before I start spending even more money. If there's a problem with the config, I suspect it's either the GRUB settings or the VM settings.
When I attach the mapped PCI HBA card, strange things happen. First, the VM goes through the BIOS loading step rather slowly. With it detached, I'd say it takes 2-3 seconds. When attached, it's closer to 20-30. I THINK the slowness is the PCI HBA card loading, but I don't really know how long this is supposed to take normally. I can see the PCI attached drives showing up in the boot sequence along with the card.
Now here's where it gets weird. With the mapped PCI attached, a virtual mounted ISO eventually boots (though again, BIOS is slow), and drives connected to PCI card are visible. However, booting from the virtual hard drive hangs indefinitely at "Booting from Hard Disk". It doesn't seem to matter what OS is installed or if there's no OS installed. It always hangs on "Booting from Hard Disk." I tried leaving it overnight and it never got unstuck.
The PCI card is NOT set as a boot device. First thing I checked. The virtual HD for the VM is on the same NVME as Proxmox itself, as intended.
Detaching the mapped PCI from the VM and booting that way immediately fixes the problem. VM BIOS no longer boots slowly, and doesn't hang on trying to boot from VHD.
/etc/modules has this.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
/etc/default/grub has this.
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream"
Anytime I make a change to the /etc/ files, I run:
update-grub
update-initramfs -u -k all
And also reboot Proxmox. Probably overkill but under the circumstances I want to be sure.
I tried several different setups here as found in various guides, including the official wiki, but haven't had any luck so far.
All hardward involved is new-ish (never used but sat on a shelf for several months before I had a chance to do the build).
Hardware
Motherboard: H13SAE-MF (AMD Ryzen CPU)
Drives: 4x WD Red 16TB
HBA card: Generic 9207-8i, preflashed for IT mode
For the card, Proxmox dashbaord shows:
Vendor: Broadcom/LSI
Device: SAS2308 PCI-Express Fusion-MPT SAS-2
dmesg shows:
[ 0.762198] mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05)
I could not find a more recent firmware version than that.
Things I have tried:
Tried a new fanout SATA cable. Tried the other fanout port on the card. Tried the other PCI-E on the board. No dice.
If I unplug the fanout cable from the HBA, the BIOS will still load slowly, but VM will eventually boot from VHD. I thought this might indicate a hard drive problem... so I tried plugging into each hard drive individually and testing like that. No VM boot. If it's a hard drive problem, it doesn't seem to be with any SINGLE drive.
I ran a long SMART test on all drives. No problems. Howeever, as mentioned, I think a DOA drive is unlikely at this point.
I looked into updating the hard drive firmware. According to WDs official support, they don't normally update drive firmware on Reds. There's some kind of proprietary Windows tool that can do it but I'd rather not screw with it unless there's no other choice.
Tried setting VM hardware in both BIOS and UEFI mode, and changing TrueNAS to match. As mentioned, it doesn't seem to actually matter what's installed on the virtual drive, including nothing. I also tried Debian instead of TrueNAS and got the same result. It doesn't seem to be a guest OS issue. In fact I don't think it's even getting that far. Booting from a VHD with no OS and the mapped PCI acttached hangs on "Booting from Hard Disk.", when I would expect it to say "No bootable device found."
I tried both i440fx and q35 machine types. Supposedly q35 is better for PCI passthrough? Also tried changing the advanced VIOMMU setting to "Intel - AMD compatible" as recommended based on some posts I read.
So far as I can tell, I don't see any sort of errors in any logs related to the VM. journalctl -b -e output is below.
Code:
Jan 25 11:33:45 proxmox systemd[1]: Started 100.scope.
Jan 25 11:33:45 proxmox kernel: tap100i0: entered promiscuous mode
Jan 25 11:33:45 proxmox kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Jan 25 11:33:45 proxmox kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Jan 25 11:33:45 proxmox kernel: fwpr100p0: entered allmulticast mode
Jan 25 11:33:45 proxmox kernel: fwpr100p0: entered promiscuous mode
Jan 25 11:33:45 proxmox kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Jan 25 11:33:45 proxmox kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Jan 25 11:33:45 proxmox kernel: fwln100i0: entered allmulticast mode
Jan 25 11:33:45 proxmox kernel: fwln100i0: entered promiscuous mode
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Jan 25 11:33:45 proxmox kernel: tap100i0: entered allmulticast mode
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jan 25 11:33:45 proxmox kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Jan 25 11:33:46 proxmox kernel: vfio-pci 0000:03:00.0: resetting
Jan 25 11:33:46 proxmox kernel: vfio-pci 0000:03:00.0: reset done
Jan 25 11:33:46 proxmox kernel: vfio-pci 0000:03:00.0: resetting
Jan 25 11:33:46 proxmox kernel: vfio-pci 0000:03:00.0: reset done
Jan 25 11:33:46 proxmox pvedaemon[10089]: VM 100 started with PID 10101.
Jan 25 11:33:46 proxmox pvedaemon[9738]: <root@pam> end task UPID:proxmox:00002769:0004EB4A:697653F8:qmstart:100:root@pam: OK
Jan 25 11:33:46 proxmox pvedaemon[9738]: <root@pam> starting task UPID:proxmox:000027C4:0004EC07:697653FA:vncproxy:100:root@pam:
Jan 25 11:33:46 proxmox pvedaemon[10180]: starting vnc proxy UPID:proxmox:000027C4:0004EC07:697653FA:vncproxy:100:root@pam:
Jan 25 11:34:49 proxmox pvedaemon[1562]: worker exit
Jan 25 11:34:49 proxmox pvedaemon[1559]: worker 1562 finished
Jan 25 11:34:49 proxmox pvedaemon[1559]: starting 1 worker(s)
Jan 25 11:34:49 proxmox pvedaemon[1559]: worker 10339 started}At this point I suspect a problem with the HBA card, that it's either bad or somehow off-spec or something. But I wanted to verify that I've tried everything configuration-wise before I start spending even more money. If there's a problem with the config, I suspect it's either the GRUB settings or the VM settings.
