Proxmox Host unresponsive after starting VM with PCI Coral TPU passed through

[Charry2014]

This is driving me nuts - I hope someone can help. I am trying to pass-through a Coral Edge TPU M.2 through to a Debian VM to run Frigate there, and every time I start the guest the Proxmox host goes unresponsive for some 30 seconds with a communication failure (0) spin box, the Start operation returns Error: start failed: QEMU exited with code 1 and then the host restarts. The Proxmox host is v. 8.2.4, completely updated. I have been following instructions here and here. This even worked briefly, so the hardware is basically fine, but now this crashing/restarting behaviour is absolutely consistent.

In the Proxmox host the Coral TPU is detected and seems to be correctly registered - and seems to be on its own IOMMU group so passing this through to the VM should not cause the host issues.

lspci -vv
01:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU (prog-if ff)
    Subsystem: Global Unichip Corp. Coral Edge TPU
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 11
    IOMMU group: 8
    Region 0: Memory at d0100000 (64-bit, prefetchable) [disabled] [size=16K]
    Region 2: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=1M]
    Capabilities: [80] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25W
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap:    Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
            ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x1
            TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [d0] MSI-X: Enable- Count=128 Masked-
        Vector table: BAR=2 offset=00046800
        PBA: BAR=2 offset=00046068
    Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [f8] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
    Capabilities: [108 v1] Latency Tolerance Reporting
        Max snoop latency: 3145728ns
        Max no snoop latency: 3145728ns
    Capabilities: [110 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
               T_CommonMode=0us LTR1.2_Threshold=163840ns
        L1SubCtl2: T_PwrOn=10us
    Capabilities: [200 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Kernel driver in use: vfio-pci

The apex and gasket modules have been blacklisted in the host as described in the instructions, and I have tried so many things now that I start to lose track. The only debug I got from Proxmox is this timeout -

TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -name 'Copy-of-VM-frigate01,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=ef1fb56f-f64c-4856-9263-da039c2fcb8b' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/101.vnc,password=on' -cpu qemu64,+aes,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3 -m 16384 -object 'iothread,id=iothread-virtioscsi0' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' -device 'vmgenid,guid=77e70bbc-0daa-4612-9b56-69584cd45350' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=0000:01:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:bf9866bda414' -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101' -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' -drive 'file=/dev/pve/vm-101-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=BC:24:11:16:F8:3B,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256' -machine 'type=pc+pve0'' 
failed: got timeout

Can anyone give me a hint to debug this?

 

[justinclift]

Have you checked if there's any other device in the same vfio PCIe group being passed through at the same time?

I'm kind of thinking that some extra device(s) might be getting passed through to the VM. If it's device(s) needed for the system to function, that would potentially cause something like this.

An alternative option would be to drop back to kernel 6.5, as the 6.8 kernels caused all kinds of weird issues for people when they were released. Maybe this is one of the problems.

 

[Charry2014]

Thanks for the reply, unfortunately I checked this out as best I could figure out - the Coral is alone as 01:00 on IOMMU group 8 and only one device is using the vfio-pci driver which from the output above we can see is the Coral.

root@pmox02:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1f.2
/sys/kernel/iommu_groups/7/devices/0000:00:1f.0
/sys/kernel/iommu_groups/7/devices/0000:00:1f.6
/sys/kernel/iommu_groups/7/devices/0000:00:1f.4
/sys/kernel/iommu_groups/5/devices/0000:00:1d.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/1/devices/0000:00:00.0
/sys/kernel/iommu_groups/8/devices/0000:01:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:1e.0
/sys/kernel/iommu_groups/4/devices/0000:00:17.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.2
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/10/devices/dw-apb-uart.0
/sys/kernel/iommu_groups/0/devices/0000:00:02.0
/sys/kernel/iommu_groups/9/devices/idma64.0

root@pmox02:~# lspci -vv|grep vfio
    Kernel driver in use: vfio-pci

One thing that is an issue is I don't completely understand what I am doing - what drivers and installations must be installed on the Proxmox host, and which ones must be installed on the guest VM. Fluffing around blindly with something complex like this is risky, obviously.

As I said above - this used to work on the hardware and kernel I have so I am baffled what went wrong.

 

[justinclift]

Ahhh, just had an idea.

I wonder if the recent update of QEMU (which runs the VMs) from version 8.x to version 9.x might be causing the problem somehow?

As an experiment, it's probably worth trying out version 8.x QEMU and seeing if things magically start working again:

https://forum.proxmox.com/threads/q...no-subscription-as-of-now.149772/#post-678813

 

[Charry2014]

Thanks for the hint - I did not try this out as I completely reinstalled Proxmox from scratch and then, highly caffeinated, started following the instructions and looking in detail at all the output. It worked on Proxmox 8.2.2.

The bit that seemed to make the difference is here - using apt install pve-headers does not seem to do exactly the same thing as apt install pve-headers-$(uname -r). Having done this and then reinstalled gasket-dkms the /dev/apex_0 device appears and works just fine. It is possible that this just happened to be a coincidence, but that is how it appeared.

Some notes -

• You will need to apt install dkms as well as other packages in the instructions when starting with a clean Proxmox
• Add another source to apt, needed for pve-headers, as well as removing the Enterprise sources if you don't have a subsrciption.
  nano /etc/apt/sources.list
  Then add deb http://download.proxmox.com/debian/pve> bookworm pve-no-subscription
• I did not edit /etc/default/grub or /etc/modules but did rebuild the gasket-dkms module

Frigate now runs in an LXC container, with the Coral TPU passed through as pci:0 and everything is fine.

To set up the LXC container I used the basic instructions here, including the setup script for Docker, and fine tuning the container passthroughs here. Both the LXC container and the Docker container are privileged which may not be ideal but I will leave this as an exercise for the future.

Going back to the original problem - the most disturbing part about that was the complete collapse of the host for 30 seconds and total lack of debug information, so I am happy to have moved on.

 

[justinclift]

Awesome. Sounds like quality detective work, and those notes are likely going to be useful for future people (or yourself again later ) that need to get this working as well.