[SOLVED] Hailo 8 ai m.2 card crashes server when using passthrough.

[scyto]

Hi all, I have on obscure sceanrio and i have hit a dead end with my google-fu.

• I have a Hailo 8 M.2 AI co-processor.
• It is on an M.2 bifurcation card with 3 NVMes, the NVMEs work fine.
• The co-processor is in its own IOMMU gtoup.
• It works perfectly on the host (of course this is the last thing i tried, lol).

When i pass through the card to a VM and when the driver loads in the VM the whole physical server instantly resets. in the BMC i see a PCI SERR entry.
I dont seen any signs this is a graveful shutdown, it is a hard reset of the server.

  ID  |      TimeStamp      |    Sensor Name   |             Sensor Type            |                          Description                           
======|=====================|==================|====================================|================================================================
 736  | 05/19/2025 10:29:23 | BIOS             | critical_interrupt                 | PCIe SEL Log - Asserted
      |                     |                  |                                    | Data1: PCI SERR
      |                     |                  |                                    | Data2: PCI bus number for failed device: 0x00
      |                     |                  |                                    | Data3: PCI device number: 0x01 PCI function number: 0x01

things i have tried:

• echo null to the reset_methods on the device (this was to supress FLR errors)
• add a few vfio option to modprobe - namely
• passing the device as pci instead of pcie
• changing the PCIE speed in the BIOS of the x16 slot (this was at suggestion of asrock rack who make the mobo)
• addng viommu=intel to the machine type
• adding this to modprobe.d file options vfio-pci disable_vfio_pci_flr=1

Due to the sudden nature of the issue there is nothing that seems useful in jornactl on the host or on the guest below is the host.

May 19 10:27:53 pve-nas1 pvedaemon[2680]: start VM 101: UPID:pve-nas1:00000A78:00004CBB:682B6A19:qmstart:101:root@pam:
May 19 10:27:53 pve-nas1 pvedaemon[2008]: <root@pam> starting task UPID:pve-nas1:00000A78:00004CBB:682B6A19:qmstart:101:root@pam:
May 19 10:27:54 pve-nas1 chronyd[1807]: Selected source 73.65.80.137 (2.debian.pool.ntp.org)

thats the last line, it literally is a super hard reset

is there any magic someone has, or is this just one of those things where it doesn't work and i live with it?

 

[scyto]

I solved this, the device when it load the driver causes a hotplug event to occur

solution is to enable hotplug in the BIOS on the PCIE slot the card is connected to.

one small issue is that the device then disappears from the PCIE bus in the VM 2 seconds after the driver loads

 

[kobold81]

i have a cwwk n305 board (the purple nas board) and getting hailo to work was a nightmare.
Due to my setup, one nvme drive had a physical name on the pool instead of the by-id, so inserting the hailo8 led to zfs problems with that pool.

thanks to claude, I was given this solution:
I inserted hailo8 into the nvme slot next to the cpu. That is Root port 1 in bios and enabled "hotplug" for this port.
I added under Datacenter->Ressource mappings a ressource for the hailo devices and named this "hailo8"
Thenm, I added this as a new pcie device using the gui (click on advanced to toggle rombar off): "mapping=hailo8, rombar=0"

for testing, I ran on the proxox server "echo bus > /sys/bus/pci/devices/0000:01:00.0/reset_method" and then started the vm with fedora. Hailo8 appears (lspci -nn).
I added an udev rule so that this is run automatically:

cat > /etc/udev/rules.d/99-hailo-reset.rules << 'EOF'ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1e60", ATTR{device}=="0x2864", ATTR{reset_method}="bus"EOF