AMD-Vi: Unable to read/write to IOMMU perf counter

L

Lip90

Well-Known Member
Dec 12, 2018
31
0
46
34
Hi Guys,

helpless seeks help!

I want to hand over my NIC to a VM. Unfortunately that doesn't work. I think it's because of the IOMMU bug.

Mainboard: X570D4U
CPU: Ryzen Pro 4650G
Proxmox: 6.3 yesterday clean install

Bios:
SVM: Yes
Iommu: Auto / Enable

All settings have been tested, but I always get the same error

Code: 
root@pve2021:~# dmesg | grep -e DMAR -e IOMMU
[    0.758400] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    0.762671] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    5.600616] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

Code: 
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
GRUB_CMDLINE_LINUX=""

Code: 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
#i915.enable_gvt=1
 
Last edited:
iommu=pt
Add this and that message goes away.

That you can't passthrough your nic can have multiple issues, but most likely is that you didn't blacklisted your nic driver.

That you have shitty iommu groups, differs from mainboard to mainboard.

Or that you can't passthrough to a freebsd guest with q35 & pcie=1 as option.
 
Last edited:
Thanks for your replay. The message is still the same.
I chanced the Code und update-grup + reboot


Code: 
 If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

Code: 
root@pve2021:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.756398] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    0.759610] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.759611] pci 0000:00:00.2: AMD-Vi: Extended features (0x206d73ef22254ade):
[    0.759614] AMD-Vi: Interrupt remapping enabled
[    0.759614] AMD-Vi: Virtual APIC enabled
[    0.759614] AMD-Vi: X2APIC enabled
[    0.759766] AMD-Vi: Lazy IO/TLB flushing enabled
[    5.383051] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>


How can i Blacklist die I210 driver?
1615555244537.png

I think the goups look bad?
Code: 
root@pve2021:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/3/devices/0000:00:14.3
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.1
/sys/kernel/iommu_groups/1/devices/0000:2d:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/4/devices/0000:00:18.3
/sys/kernel/iommu_groups/4/devices/0000:00:18.1
/sys/kernel/iommu_groups/4/devices/0000:00:18.6
/sys/kernel/iommu_groups/4/devices/0000:00:18.4
/sys/kernel/iommu_groups/4/devices/0000:00:18.2
/sys/kernel/iommu_groups/4/devices/0000:00:18.0
/sys/kernel/iommu_groups/4/devices/0000:00:18.7
/sys/kernel/iommu_groups/4/devices/0000:00:18.5
/sys/kernel/iommu_groups/2/devices/0000:30:00.6
/sys/kernel/iommu_groups/2/devices/0000:00:08.0
/sys/kernel/iommu_groups/2/devices/0000:30:00.4
/sys/kernel/iommu_groups/2/devices/0000:31:00.0
/sys/kernel/iommu_groups/2/devices/0000:30:00.2
/sys/kernel/iommu_groups/2/devices/0000:30:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:08.1
/sys/kernel/iommu_groups/2/devices/0000:30:00.5
/sys/kernel/iommu_groups/2/devices/0000:31:00.1
/sys/kernel/iommu_groups/2/devices/0000:30:00.3
/sys/kernel/iommu_groups/2/devices/0000:30:00.1
/sys/kernel/iommu_groups/2/devices/0000:00:08.2
/sys/kernel/iommu_groups/0/devices/0000:20:00.0
/sys/kernel/iommu_groups/0/devices/0000:21:06.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.2
/sys/kernel/iommu_groups/0/devices/0000:26:00.0
/sys/kernel/iommu_groups/0/devices/0000:21:09.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/0/devices/0000:10:00.1
/sys/kernel/iommu_groups/0/devices/0000:2a:00.3
/sys/kernel/iommu_groups/0/devices/0000:2a:00.1
/sys/kernel/iommu_groups/0/devices/0000:21:05.0
/sys/kernel/iommu_groups/0/devices/0000:21:0a.0
/sys/kernel/iommu_groups/0/devices/0000:21:08.0
/sys/kernel/iommu_groups/0/devices/0000:2c:00.0
/sys/kernel/iommu_groups/0/devices/0000:29:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.1
/sys/kernel/iommu_groups/0/devices/0000:21:04.0
/sys/kernel/iommu_groups/0/devices/0000:2b:00.0
/sys/kernel/iommu_groups/0/devices/0000:10:00.0
/sys/kernel/iommu_groups/0/devices/0000:28:00.0
/sys/kernel/iommu_groups/0/devices/0000:2a:00.0
/sys/kernel/iommu_groups/0/devices/0000:27:00.0
 
Last edited:
The perf log error should not be a problem.

How you blacklist it?
Please try to use the search function, else you won't learn and ask questions for everything xD

Probably you can't even blacklist the whole driver, because you use the same nic for your host, just another port xD
But there is a solution for this either. Search for passthrough helper.

Cheers
 
What is the problem then? I can't even get through a USB hard drive. That wasn't a problem with my old XEON processor.

I also have a 350-t2 and an i211, i.e. I could pass one of the two cards through. If I do that, however, the whole system freezes.
 
Lip90 said:
What is the problem then? I can't even get through a USB hard drive. That wasn't a problem with my old XEON processor.

I also have a 350-t2 and an i211, i.e. I could pass one of the two cards through. If I do that, however, the whole system freezes.
Click to expand...
Yes, that makes absolutely sense, I don't exactly know what proxmox does in this case, but the igb driver module is loaded on i350 & i211 nics/ports... And you try to take the port/nic away from that loaded module, which either unloads the module = no connection at all anymore or it freezes/crashes the system.

However, that's absolutely normal.

You need first to unload the igb driver from the one port/nic that you want to passthrough and bind a "fake/dummy" driver (vfio) to it. Then you can passthrough the nic to the vm, without causing weird issues.

Did you searched at all?

Because if you had searched you had found already a solution, i have said even already above what you have to search for... But yes, i guess, googling is hard xD

Screenshot_20210312-192321.jpg

Cheers xD
 
I would be careful with such statements! I've been googling all day. Still no solution found. I also tried your miracle tool with a great result!

Code: 
root@pve2021:/usr/bin# cat /etc/systemd/system/vfio-helper.service
[Unit]
Description=Script to bind Devices on boot
Before=pve-guests.service

[Service]
Type=oneshot
EnvironmentFile=-/etc/vfio-helper.conf
ExecStart=/usr/bin/vfio-pci-bind.sh $BIND_DEVICES

[Install]
WantedBy=multi-user.target

Code: 
root@pve2021:/usr/bin# cat /etc/vfio-helper.conf
# Modify first and uncomment the line below!
# The line below is just an example, you can bind devices based on BUS or vendor:devid
# You can add as many devices you want, space separated.
# Below are 2 lines as example, if you want to bind one or multiple devices!
BIND_DEVICES="174c:55aa"
#BIND_DEVICES="0000:23:00.1 0000:06:00.0 10de:1b06 10de:10ef"

Look at this!
Code: 
root@pve2021:/usr/bin# vfio-pci-bind.sh
Error: Please provide Domain:Bus:Device.Function (dddd:bb:dd.f) and/or Version:Device (vvvv:dddd)
 
It looks like people are working on a fix in the kernel. No idea when this will reach Proxmox. For someone it was just a case of enabling more in the BIOS. In a similar post, it did not prevent successful passthrough.
It probably is the IOMMU grouping, which is weird because the X570 chipset should have the best support for passthrough because even the devices connected to the chipset are usually properly separated into nice groups. Explicit (un)binding is usually not necessary. This command gives a very readable overview of the groups: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done Can you show us?
Have you tried different BIOS versions (as the grouping is determined by the motherboard's UEFI BIOS)? Can you try the 1.00 version, before the support for newer CPUs, which have broken passthrough before until it was fixed again much later...
 
Code: 
root@pve2021:/usr/bin# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
IOMMU group 0 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 0 10:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 0 10:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 0 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57ad]
IOMMU group 0 21:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 0 21:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 0 21:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 0 21:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU group 0 21:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU group 0 21:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU group 0 26:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 0 27:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 0 28:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
IOMMU group 0 29:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
IOMMU group 0 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 0 2a:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 0 2a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 0 2b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 0 2c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 1 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 1 2d:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 [144d:a808]
IOMMU group 2 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 2 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 2 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 2 30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [1002:1636] (rev d9)
IOMMU group 2 30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
IOMMU group 2 30:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
IOMMU group 2 30:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 2 30:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 2 30:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor [1022:15e2] (rev 01)
IOMMU group 2 30:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
IOMMU group 2 31:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU group 2 31:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU group 3 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
IOMMU group 3 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 4 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 [1022:1448]
IOMMU group 4 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 [1022:1449]
IOMMU group 4 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 [1022:144a]
IOMMU group 4 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 [1022:144b]
IOMMU group 4 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 [1022:144c]
IOMMU group 4 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 [1022:144d]
IOMMU group 4 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 [1022:144e]
IOMMU group 4 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 [1022:144f]
 
The grouping is terrible. Can you try other BIOS versions? Currently, if you passthrough device 26:00.0 the host loses connection to all other devices in group 0: 3 network devices, 2 SATA and 2 USB controllers and the motherboard GPU. That is probably the reason that the system freezes, not the message about the IOMMU perf counter.
 
  • Like
Reactions: Ramalama
Yep, he has right, your groups are a bummer xD
Additional acs_override=downstream you could try that, if you can't or don't want or downgrade doesn't makes any effects.

Edit:
And only 4 immu groups aren't normal. Never seen so less. Normally there are 20-30 iommu groups.
Dunno if it's possible, but it strongly looks to me like they are wrong initialized or something. Maybe another kernel helps? I don't know, but you could try the pve-edge-kernel.
4 groups looks for me more like a software instead of hardware bug....
 
Last edited:
acs_override = downstream did not bring anything either. Bios update is already done. That doesn't help either. Do you think it makes sense to set up a new Proxmox? However, I only did that yesterday. Should I try ZFS instead of ex4? Do you think that would help?
 
He said downgrade to your first available bios, not upgrade xD

No, reinstalling wouldn't help you.
The only thing that is worth to try is the pve-edge kernel. If that doesn't work too, then you need to eat it and live with the virtio nic.
I210 -> vmbrXX -> virtionic(guest)
 
Lip90 said:
acs_override = downstream did not bring anything either.
Click to expand...
That should have split the IOMMU groups, and the motherboard should be capable? It always does this, unless a mistake has been made somewhere. Can you please tell us what you changed? Please note that you should add pcie_acs_override=downstream to the kernel parameters (people often call it ACS-override without PCIe). What does cat /proc/cmdline show?
Lip90 said:
Bios update is already done. That doesn't help either.
Click to expand...
Which version have you tried? Does it change anything? Did you reset CMOS? Please give more specific information.
Lip90 said:
Do you think it makes sense to set up a new Proxmox? However, I only did that yesterday. Should I try ZFS instead of ex4? Do you think that would help?
Click to expand...
No, all those things don't matter because it depends on the BIOS of your motherboard. Have you found other people using that same motherboard having success or problems on the internet? You could try contacting ASRock Rack and ask if there are Beta versions that have better IOMMU groups.
 
  • Like
Reactions: Ramalama
avw said:
That should have split the IOMMU groups, and the motherboard should be capable? It always does this, unless a mistake has been made somewhere. Can you please tell us what you changed? Please note that you should add pcie_acs_override=downstream to the kernel parameters (people often call it ACS-override without PCIe). What does cat /proc/cmdline show?

Which version have you tried? Does it change anything? Did you reset CMOS? Please give more specific information.

No, all those things don't matter because it depends on the BIOS of your motherboard. Have you found other people using that same motherboard having success or problems on the internet? You could try contacting ASRock Rack and ask if there are Beta versions that have better IOMMU groups.
Click to expand...

Lol, thanks for the hint, just seen asrock rack, then looked what he uses, and he uses the x570d4u....
I am using the x570d4i, that's the same board, just another form factor.

Iommu groups are perfect on both boards.
I have 27 or 37 groups, forgot the number, but can check later at home.

He has definitively setted up something wrong.
The only difference is his 4650g cpu vs my 5800x.
Or the cpu dictates the iommu groups???
Need to google xD

So back to the main issue, i bet there is something wrong in the bios, 4 groups isn't normal.
 
Last edited:
Hello everybody,

so i have installed pve-edge. did not help. Only the downgrade to the first BIOS version. However, the groupings still look bad, don't they?

Code: 
root@pve:~# dmesg |grep -e DMAR -e IOMMU -e AMD-Vi
[    0.731449] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    0.739534] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.739534] pci 0000:00:00.2: AMD-Vi: Extended features (0x206d73ef22254ade):
[    0.739537] AMD-Vi: Interrupt remapping enabled
[    0.739537] AMD-Vi: Virtual APIC enabled
[    0.739537] AMD-Vi: X2APIC enabled
[    0.739666] AMD-Vi: Lazy IO/TLB flushing enabled
[    5.164183] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
root@pve:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 10 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57ad]
IOMMU group 11 21:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 12 21:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 13 21:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 14 21:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU group 14 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 14 2a:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 14 2a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 15 21:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU group 15 2b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 16 21:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU group 16 2c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 17 26:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 18 27:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 19 28:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
IOMMU group 19 29:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
IOMMU group 20 2d:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 [144d:a808]
IOMMU group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 4 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 5 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 5 30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [1002:1636] (rev d9)
IOMMU group 5 30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
IOMMU group 5 30:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
IOMMU group 5 30:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 5 30:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 5 30:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor [1022:15e2] (rev 01)
IOMMU group 5 30:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
IOMMU group 5 31:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU group 5 31:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
IOMMU group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 [1022:1448]
IOMMU group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 [1022:1449]
IOMMU group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 [1022:144a]
IOMMU group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 [1022:144b]
IOMMU group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 [1022:144c]
IOMMU group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 [1022:144d]
IOMMU group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 [1022:144e]
IOMMU group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 [1022:144f]
IOMMU group 8 10:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 9 10:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
 
The IOMMU groups pretty good (don't worry about PCI bridges) except for group 5. I think this BIOS version is good enough to use. Most importantly: device 26:00.0 is now alone in a separate group!
EDIT: Is this BIOS version 1.00 (which I asked before)? What is pve-edge (because I have not heard it before)? Just to be sure: this is without ACS-override?
 
Last edited:
Hello,
I have a small Homeserver with the A520M-ITX/ac and Ryzen 4350G. I'm pretty new to proxmox and linux and had trouble to passthrough an Intel i340T4 but that was mostly because i used q35 machine with freebsd. Now its working fine although i get the same perf count error:
Code: 
dmesg |grep -e DMAR -e IOMMU -e AMD-Vi
[    0.650226] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter.
[    0.652469] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.652470] pci 0000:00:00.2: AMD-Vi: Extended features (0x206d73ef22254ade):
[    0.652474] AMD-Vi: Interrupt remapping enabled
[    0.652475] AMD-Vi: Virtual APIC enabled
[    0.652476] AMD-Vi: X2APIC enabled
[    0.652604] AMD-Vi: Lazy IO/TLB flushing enabled

I have also just 5 IOMMO Groups with the newest BIOS:

Code: 
root@xxx:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
IOMMU Group 0 01:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 0 01:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 0 01:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 0 01:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 1 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 1 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 1 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ec]
IOMMU Group 1 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU Group 1 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU Group 1 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 1 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 1 03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 1 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 1 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 1 05:00.0 Network controller [0280]: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] [8086:24fb] (rev 10)
IOMMU Group 1 08:00.0 Non-Volatile memory controller [0108]: Kingston Technologies Device [2646:2263] (rev 03)
IOMMU Group 2 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 2 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU Group 2 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [1002:1636] (rev da)
IOMMU Group 2 09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
IOMMU Group 2 09:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
IOMMU Group 2 09:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU Group 2 09:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU Group 3 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
IOMMU Group 3 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 4 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 [1022:1448]
IOMMU Group 4 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 [1022:1449]
IOMMU Group 4 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 [1022:144a]
IOMMU Group 4 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 [1022:144b]
IOMMU Group 4 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 [1022:144c]
IOMMU Group 4 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 [1022:144d]
IOMMU Group 4 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 [1022:144e]
IOMMU Group 4 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 [1022:144f]

When I tried to passthrough the m.2 E-Key Intel wifi module the system crashes because the M.2 SSD with the host system is in the same group. I used an old router for wifi for now but do you think with an old BIOS I could get better IOMMO Groups as well?
I try to passthrough the iGPU at the moment but even if the system does not crash the vm wont start at all.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back