PCI Passthrough: AMD-Vi: Completion-Wait loop timed out

[gerhardt]

Recently I installed a QLogic BCM57840 10 Gigabit Network Adapter on my AMD EPYC server and try to pass it through to a VM. Before this, a Intel I350 network card was successfully passed through to this VM. I am trying to replace the old Intel card with this QLogic card. When the system was booted, I saw some errors about the iommu of the QLogic card in the kernel log:

[    2.835852] pci 0000:40:08.3: Adding to iommu group 56
[    2.836682] AMD-Vi: Completion-Wait loop timed out
[    2.836682] fbcon: Taking over console
[    2.967620] pci 0000:41:00.0: Adding to iommu group 48
[    2.971614] AMD-Vi: Completion-Wait loop timed out
[    3.103429] AMD-Vi: Completion-Wait loop timed out
[    3.231083] pci 0000:41:00.1: Adding to iommu group 48
[    3.235080] AMD-Vi: Completion-Wait loop timed out
[    3.366750] AMD-Vi: Completion-Wait loop timed out
[    3.493801] pci 0000:41:00.2: Adding to iommu group 48
[    3.497799] AMD-Vi: Completion-Wait loop timed out
[    3.628921] AMD-Vi: Completion-Wait loop timed out
[    3.756157] pci 0000:41:00.3: Adding to iommu group 48
[    3.835843] pci 0000:42:00.0: Adding to iommu group 57
[    3.839841] AMD-Vi: Completion-Wait loop timed out
[    3.967375] pci 0000:42:00.2: Adding to iommu group 58
[    3.982936] AMD-Vi: Completion-Wait loop timed out
[    4.099057] pci 0000:43:00.0: Adding to iommu group 59
[    4.103053] AMD-Vi: Completion-Wait loop timed out
[    4.230686] pci 0000:43:00.1: Adding to iommu group 60
[    4.234680] AMD-Vi: Completion-Wait loop timed out
[    4.362423] pci 0000:43:00.2: Adding to iommu group 61
[    4.366420] AMD-Vi: Completion-Wait loop timed out
[    4.494127] pci 0000:43:00.3: Adding to iommu group 62
[    4.498124] AMD-Vi: Completion-Wait loop timed out
[    4.625404] pci 0000:44:00.0: Adding to iommu group 63
[    4.686385] AMD-Vi: Completion-Wait loop timed out
[    4.756726] pci 0000:45:00.0: Adding to iommu group 64
[    4.835821] pci 0000:c0:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.835822] pci 0000:c0:00.2: AMD-Vi: Extended features (0x58f77ef22294ade):
[    4.835823]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    4.835825] pci 0000:80:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.835826] pci 0000:80:00.2: AMD-Vi: Extended features (0x58f77ef22294ade):
[    4.835826]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    4.835828] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.835828] pci 0000:40:00.2: AMD-Vi: Extended features (0x58f77ef22294ade):
[    4.835829]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    4.835830] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    4.835831] pci 0000:00:00.2: AMD-Vi: Extended features (0x58f77ef22294ade):
[    4.835831]  PPR X2APIC NX GT IA GA PC GA_vAPIC
[    4.835832] AMD-Vi: Interrupt remapping enabled
[    4.835832] AMD-Vi: Virtual APIC enabled
[    4.835832] AMD-Vi: X2APIC enabled
[    4.836251] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.0 address=0x1fc85c8320]
[    4.836260] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.0 address=0x1fc85c8350]
[    4.836321] AMD-Vi: Lazy IO/TLB flushing enabled
[    4.837190] amd_uncore: AMD NB counters detected
[    4.837246] amd_uncore: AMD LLC counters detected
[    4.838895] LVT offset 0 assigned for vector 0x400

The QLogic card is pci 0000:41:00.0-0000:41:00.3. It kept showing me this error:

AMD-Vi: Completion-Wait loop timed out
iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.0 address=0x1fc85c8320]
iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.1 address=0x1fc85c83a0]
iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.2 address=0x1fc85c8420]
iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.3 address=0x1fc85c84e0]

Here's my VM config:

boot: cdn
bootdisk: scsi0
cores: 8
cpu: host
cpuunits: 1025
hostpci0: 41:00,pcie=1
machine: q35
memory: 2048
name: VM1
net0: virtio=9A:0F:0F:FA:EB:7D,bridge=vmbr1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,backup=0,cache=writeback,replicate=0,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=17b66d57-3daa-4dfb-8dc4-47f15c12d4bf
sockets: 1
startup: order=1
vmgenid: 6080b0fe-6f9a-4d30-806e-7b00a76bd31b

When I tried to turn on my VM, the same error began showing again:

[  300.895217] AMD-Vi: Completion-Wait loop timed out
[  301.027994] AMD-Vi: Completion-Wait loop timed out
[  301.160569] AMD-Vi: Completion-Wait loop timed out
[  301.293077] AMD-Vi: Completion-Wait loop timed out
[  301.557833] AMD-Vi: Completion-Wait loop timed out
[  301.690138] AMD-Vi: Completion-Wait loop timed out
[  301.762931] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.3 address=0x1fc85c8f70]
[  301.764406] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.1 address=0x1fc85c8f90]
[  301.764997] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.0 address=0x1fc85c8fa0]
[  301.895191] AMD-Vi: Completion-Wait loop timed out
[  303.764499] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.2 address=0x1fc85c90d0]
[  303.765000] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.1 address=0x1fc85c90e0]
[  303.894573] AMD-Vi: Completion-Wait loop timed out
[  304.180750] AMD-Vi: Completion-Wait loop timed out
[  304.313042] AMD-Vi: Completion-Wait loop timed out
[  304.445390] AMD-Vi: Completion-Wait loop timed out
[  304.762923] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.0 address=0x1fc85c9110]
[  304.763905] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.0 address=0x1fc85c9130]
[  304.894486] AMD-Vi: Completion-Wait loop timed out
[  305.026785] AMD-Vi: Completion-Wait loop timed out
[  305.162010] AMD-Vi: Completion-Wait loop timed out
[  307.375265] AMD-Vi: Completion-Wait loop timed out
[  307.511846] AMD-Vi: Completion-Wait loop timed out
[  307.649471] AMD-Vi: Completion-Wait loop timed out
[  307.762929] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.2 address=0x1fc85c9220]
[  307.763883] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.3 address=0x1fc85c9240]
[  307.948397] AMD-Vi: Completion-Wait loop timed out
[  308.083788] AMD-Vi: Completion-Wait loop timed out
[  320.202487] AMD-Vi: Completion-Wait loop timed out
[  320.762931] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.3 address=0x1fc85c9aa0]
[  320.763436] iommu ivhd2: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=41:00.2 address=0x1fc85c9ab0]

The console window of the VM was stuck in the BIOS and the whole system was almost hanging:



I also tried to do the same pass-through on VMWare ESXI and everything works fine. I am wondering why the pass-through failed on PVE.

 

[r.jochum]

Whats you'r Kernel cmdline?

 

[gerhardt]

Whats you'r Kernel cmdline?

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.44-2-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on

I also tried to add iommu=pt but still got the same error.

 

[r.jochum]

I googled your error and found this: https://unix.stackexchange.com/ques...a-failed-install-of-arch-linux-on-a-blank-ssd

You could try "acpi=off" and "iommu=soft" but as explained on the link its bad.