HBA controller busy/blocked despite only explicitly passed through to one VM

[cupcake]

Hello!

I'm completely new to Proxmox. I have 2 Windows server VMs running, 100 and 101. I have an HBA controller in IT-mode with one drive attached. I'm passing the controller through to 101.

If 101 is the only VM running, all is good, controller and drive is available inside of the VM. However, if 100 starts up before 101, 101 will throw an error when trying to start up. The controller has the addresses 0000:0a:00 and 0000:0c:00.


This is the error of 101 at start:

Spoiler

swtpm_setup: Not overwriting existing state file.
kvm: -device vfio-pci,host=0000:0a:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:0a:00.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy
stopping swtpm instance (pid 1415) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1

Edit: I've changed things around to start VM 101 before 100 and while 101 started up fine, now VM 100 is giving me an error on startup, about one of its pci devices:

Spoiler

swtpm_setup: Not overwriting existing state file.
kvm: -device vfio-pci,host=0000:07:04.0,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0: vfio 0000:07:04.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy
stopping swtpm instance (pid 1400) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1

I tried manually starting VM 100, and that worked, now both are running. I'm really confused now.



100 hardware PCI section:

Spoiler

101 hardware PCI section:

Spoiler

I'd appreciate any pointing into the right direction. Thanks!

 

[leesteken]

What is the (whole) output of cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done?

 

[cupcake]

@leesteken this is the output:

root@pve:~# cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
BOOT_IMAGE=/boot/vmlinuz-6.1.15-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on i915.enable_guc=3 i915.max_vfs=7
IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 10 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:7a40] (rev 11)
IOMMU group 11 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:7a38] (rev 11)
IOMMU group 12 00:1c.2 PCI bridge [0604]: Intel Corporation Device [8086:7a3a] (rev 11)
IOMMU group 13 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:7a3c] (rev 11)
IOMMU group 14 00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:7a30] (rev 11)
IOMMU group 15 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a04] (rev 11)
IOMMU group 15 00:1f.3 Audio device [0403]: Intel Corporation Device [8086:7a50] (rev 11)
IOMMU group 15 00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:7a23] (rev 11)
IOMMU group 15 00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:7a24] (rev 11)
IOMMU group 16 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU group 17 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU group 18 06:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa)
IOMMU group 18 07:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788]
IOMMU group 19 08:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:a704] (rev 01)
IOMMU group 20 09:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
IOMMU group 21 09:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
IOMMU group 22 09:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
IOMMU group 23 0a:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
IOMMU group 24 0c:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
IOMMU group 25 00:02.1 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 26 00:02.2 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 27 00:02.3 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 28 00:02.4 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 29 00:02.5 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 2 00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:a74d] (rev 01)
IOMMU group 30 00:02.6 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 31 00:02.7 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 04)
IOMMU group 3 00:0a.0 Signal processing controller [1180]: Intel Corporation Device [8086:a77d] (rev 01)
IOMMU group 4 00:0e.0 RAID bus controller [0104]: Intel Corporation Device [8086:a77f]
IOMMU group 5 00:14.0 USB controller [0c03]: Intel Corporation Device [8086:7a60] (rev 11)
IOMMU group 5 00:14.2 RAM memory [0500]: Intel Corporation Device [8086:7a27] (rev 11)
IOMMU group 6 00:15.0 Serial bus controller [0c80]: Intel Corporation Device [8086:7a4c] (rev 11)
IOMMU group 6 00:15.1 Serial bus controller [0c80]: Intel Corporation Device [8086:7a4d] (rev 11)
IOMMU group 6 00:15.2 Serial bus controller [0c80]: Intel Corporation Device [8086:7a4e] (rev 11)
IOMMU group 7 00:16.0 Communication controller [0780]: Intel Corporation Device [8086:7a68] (rev 11)
IOMMU group 8 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:7a62] (rev 11)
IOMMU group 9 00:1a.0 PCI bridge [0604]: Intel Corporation Device [8086:7a48] (rev 11)

 

[leesteken]

I don't see anything that would start to explain it. You can put in a startup delay that could put some time between starting one VM and the other. Maybe that will help as a work-around?
Please stop using optional kernel version 6.1 as it won't get updates or security fixes. Please switch to the latest optional kernel (6.2, currently).

 

[cupcake]

You can put in a startup delay that could put some time between starting one VM and the other. Maybe that will help as a work-around?

100 now starts 20 seconds after 101 and that worked

Please stop using optional kernel version 6.1 as it won't get updates or security fixes. Please switch to the latest optional kernel (6.2, currently).

Thanks, will do. Switched to 6.1 on my way to passthrough the iGPU which was huge pain for a newbie