PCI pass-through: Unable to read/write to IOMMU perf counter

[VolkerB]

I am trying to set up a virtual multimedia server with the newest Proxmox vitualisation software. I try to pass-through my Digital Devices GmbH Cine V7 multimedia PCI card to a virtual Debian server. When I start the guest my host gets corrupted and I‘m losing contact to the machine. I follow the Instructions in the Proxmox Handbook about PCI pass-through and succeeded in binding the multimedia card to vfio. But I was not able to get rid of the „AMD-Vi: Unable to read/write to IOMMU perf counter“ error. I tried it by passing iommu=on iommu=pt to the kernel. When I used iommu=soft my system told me that it is not able to do so. I am using a ASUS PRIME A320M-K mainboard with an AMD Ryzen 3 3200G. In BIOS I set IOMMU to Enabled and SVM Mode to Enabled. Causes the board or the processor the error? Or did I miss another Bios setting? Thanks for help.

 

[leesteken]

Have you tried amd_iommu=on iommu=pt instead of iommu=on iommu=pt?
After that, can you show us your IOMMU groups with this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Which BIOS version are you using? Some BIOS versions for AMD Ryzen break PCI passthrough, and AMD-Vi: Unable to read/write to IOMMU perf counter indicates to me that not everything required is enabled or the BIOS is not optimal.

 

[VolkerB]

1. I used amd_iommu=on but forgot it in my post sorry.
2. Here are my IOMMU groups:
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 10 08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Picasso [1002:15d8] (rev c9)
IOMMU group 11 08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller [1002:15de]
IOMMU group 11 08:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
IOMMU group 11 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 [1022:15e0]
IOMMU group 11 08:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1 [1022:15e1]
IOMMU group 11 08:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
IOMMU group 12 09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 61)
IOMMU group 1 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] [1022:15d3]
IOMMU group 2 00:01.6 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0] [1022:15d3]
IOMMU group 3 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU group 4 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A [1022:15db]
IOMMU group 5 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B [1022:15dc]
IOMMU group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0 [1022:15e8]
IOMMU group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1 [1022:15e9]
IOMMU group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2 [1022:15ea]
IOMMU group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3 [1022:15eb]
IOMMU group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4 [1022:15ec]
IOMMU group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5 [1022:15ed]
IOMMU group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6 [1022:15ee]
IOMMU group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7 [1022:15ef]
IOMMU group 8 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bc] (rev 02)
IOMMU group 8 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b8] (rev 02)
IOMMU group 8 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b3] (rev 02)
IOMMU group 8 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 8 02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 8 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 8 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 8 03:00.0 Multimedia controller [0480]: Digital Devices GmbH Cine V7 [dd01:0006]
IOMMU group 8 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 8 06:00.0 RAID bus controller [0104]: Adaptec Series 6 - 6G SAS/PCIe 2 [9005:028b] (rev 01)
IOMMU group 9 07:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 [144d:a808]
3. My BIOS Version is:
Vendor: American Megatrends Inc.
Version: 5603
Release Date: 10/14/2020
Thanks again

 

[leesteken]

IOMMU group 8 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bc] (rev 02)
IOMMU group 8 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b8] (rev 02)
IOMMU group 8 03:00.0 Multimedia controller [0480]: Digital Devices GmbH Cine V7 [dd01:0006]
IOMMU group 8 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 8 06:00.0 RAID bus controller [0104]: Adaptec Series 6 - 6G SAS/PCIe 2 [9005:028b] (rev 01)

If you passthrough device 03:00.0, the Proxmox host will lose connection to everything else of IOMMU group 8 as well. This includes a RAID, a network, a SATA and a USB controller (don't worry about the PCI bridges). This is probably causing the host system problems. Can you put the PCI card into another slot and see if it gets in another IOMMU group?

 

[VolkerB]

I will try it out. What about the IOMMO perf counter message. is this counter necessary. Just for understanding.

 

[leesteken]

I looked at some AMD documentation about the A320 chipset. You'll need to put the card in the (only) x16 slot or get a riser/adapter from the M.2 slot to an extra x4 PCIe slot and put it in there. Alternatively, you can just split the IOMMU groups (and ignore that this might allow devices to read and write memory from the host and other VMs!) by adding pcie_acs_override=downstream to the kernel paramters (just after amd_iommu=on).

 

[leesteken]

I will try it out. What about the IOMMO perf counter message. is this counter necessary. Just for understanding.

Looks like people are working on a fix in the kernel. No idea if or when this will reach Proxmox.
EDIT: For someone it was just a case of enabling more in the BIOS.

 

[VolkerB]

1. I put the card in the x16 slot. When i looked for the IOMMU groups my multimedia card was alone in group 10. But my network card didn't work anymore.
2. So I am lucky with my hardware. Could I change something the CPU or the board. Are there any lists of hardware that will work fine in virtualization.

 

[leesteken]

1. I put the card in the x16 slot. When i looked for the IOMMU groups my multimedia card was alone in group 10. But my network card didn't work anymore.

This is probably because the name of your network device in Linux depends on its (internal) slot number, which can change when adding or removing PCI cards. This is easily fixed (if you can login to the console): do a dmesg | grep eth and look for the new name to which it is renamed and edit /etc/network/interfaces accordingly.

2. So I am lucky with my hardware. Could I change something the CPU or the board. Are there any lists of hardware that will work fine in virtualization.

If only, life would be so much easier... It all depends on combinations of hardware, BIOS versions and all kinds of software. For Ryzen CPUs at the moment the X570 is best for passthrough. PCI(e) passthrough is a bit hit or miss and hardware sellers don't seem to spend much effort to this relatively small group of people. Search the internet for success stories and try to keep up to date about hardware, the Linux kernel and VFIO...

 

[VolkerB]

I didn't know that the group of people using virtualization is that small. For me a computer has to help me solving problems. And as i found Proxmox I was astonished that I was able to get a virtual machine running and what was best I got access to the desktop of my virtual machine by just a mouse click. So I thought it would be that easy to put my real multimedia card in my virtual server and would have fun. This failed and so I decided to do something new and ask for help in a news group. This was a great experience for me. Perhaps I'll try the X570. So thanks a lot for your help.

 

[leesteken]

Did you not get it to work by putting it in the x16 slot and fixing the network interface? What problem did you encounter?

 

[VolkerB]

i just finished the Debian installation on the virtual machine. And I can see the multimedia card in the pci devices of the machine.
Everything is fine so far. I just have to test my raid controller. The lights on the card go on and off all the time and I can not remember if this is a normal behavior. the raid is not in use at the moment. It's a long time ago I used my computer with an open case. So putting the card in the x16 slot did it I think.