AMD ASROCK X570 CREATOR Thunderbolt 3 Passthrough, IOMMU Groups splitted but can't pass all the controller to the VM

D

deliad

New Member
Jun 12, 2023
12
0
1
hi,

FIRST OF ALL,
sorry that i divided my post to several posts one after one, thats because my files exceeded the characters limit per post
don't point me to read other threads, I've already read all of them :)

THE PROBLEM:

I've successfully pass through my VGA Controller to the vm NVIDIA Corporation GK208B [GeForce GT 710]

Now I'm interested to pass through all the Intel Corporation JHL7540 Thunderbolt 3 [Titan Ridge 4C 2018] to a VM but it seems that not all the Thunderbolt devices are using the vfio-pci driver instead of it I keep seeing that in some Thunderbolt 3 devices it shows
Code: 
Kernel driver in use: pcieport
although I ignored it via
Code: 
vfio-pci ids

actually, I will be satisfied if I could passthrough only specific - thunderbolt 3-connected devices to the VM

but when I'm trying to only passthrough my RME UFX+ audio controller that connected via Thunderbolt 3 cable to the #1 of 2 total Thunderbolt ports I have, I'm getting this QEMU Error:
Code: 
kvm: ../hw/pci/pci.c:1613: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
after alot of threads reading i found out that other users encountered this error when another devices was in the same IOMMU Groups toghether with not passthrough-wanted devices, but it's not my issue because i'm already using the
Code: 
pcie_acs_override=downstream,multifunction
that splitted all my devices to it's own group.

if i pass the 2 thunderbolt card devices i see in the list together with my rme ufx+ device the system suddenly reboots

why it keep showing
Code: 
Kernel driver in use: pcieport
although I ignored it via
Code: 
vfio-pci ids
?

of course
Code: 
update-initramfs -u -k all
and
Code: 
update-grub
is done after all of that configurations
 
Last edited:
this is my /etc/default/grub file

Code: 
# If you change this file, run 'update-grub' afterwards to update

# /boot/grub/grub.cfg.

# For full documentation of the options in this file, see:

#   info -f grub -n 'Simple configuration'



GRUB_DEFAULT=0

GRUB_TIMEOUT=0

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`

#GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt nomodeset pci=realloc,assign-busses,hpbussize=0x33 pcie_port_pm=off initcall_blacklist=sysfb_init"

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt nomodeset pci=realloc,assign-busses,hpbussize=0x33 pcie_port_pm=off initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction video=efifb:off video=vesa:off vfio-pci.ids=10de:128b,19da:6326,8086:0000,8086:15ea,8086:15eb,8086:15ec,8086:1549,1d18:0001 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,rme"



GRUB_CMDLINE_LINUX=""



# Uncomment to enable BadRAM filtering, modify to suit your needs

# This works with Linux (no patch required) and with any kernel that obtains

# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)

#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"



# Uncomment to disable graphical terminal (grub-pc only)

#GRUB_TERMINAL=console



# The resolution used on graphical terminal

# note that you can use only modes which your graphic card supports via VBE

# you can see them in real GRUB with the command `vbeinfo'

#GRUB_GFXMODE=640x480



# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux

#GRUB_DISABLE_LINUX_UUID=true



# Uncomment to disable generation of recovery mode menu entries

#GRUB_DISABLE_RECOVERY="true"



# Uncomment to get a beep at grub start

#GRUB_INIT_TUNE="480 440 1"
 
Last edited:
This are my IOMMU Groups list (as you can see, all the devices splitted to separate groups):
Code: 
IOMMU Group 0:
-e      00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
-e      00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
-e      00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
-e      00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
-e      00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
-e      00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
-e      00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
-e      00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
-e      00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
-e      00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 10:
-e      00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11:
-e      00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12:
-e      00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
-e      00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
-e      00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
-e      00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
-e      00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
-e      00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
-e      00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
-e      00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
-e      00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
-e      00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 14:
-e      01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 15:
-e      02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 16:
-e      02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 17:
-e      02:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 18:
-e      02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 19:
-e      02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 20:
-e      02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 21:
-e      02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 22:
-e      02:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 23:
-e      03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 24:
-e      04:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03)
IOMMU Group 25:
-e      05:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU Group 26:
-e      06:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU Group 27:
-e      06:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU Group 28:
-e      06:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU Group 29:
-e      06:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU Group 30:
-e      07:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06)
IOMMU Group 31:
-e      3b:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06)
IOMMU Group 32:
-e      6f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 33:
-e      70:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 34:
-e      70:02.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 35:
-e      70:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 36:
-e      70:04.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 37:
-e      70:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 38:
-e      70:06.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 39:
-e      70:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU Group 40:
-e      71:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 41:
-e      72:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 42:
-e      73:00.0 Network controller [0280]: Broadcom Inc. and subsidiaries BCM4352 802.11ac Wireless Network Adapter [14e4:43b1] (rev 03)
IOMMU Group 43:
-e      74:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 44:
-e      78:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)
IOMMU Group 45:
-e      79:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 46:
-e      79:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 47:
-e      79:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 48:
-e      7a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 49:
-e      7b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 50:
-e      7c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 51:
-e      7d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 52:
-e      7d:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 53:
-e      7e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 54:
-e      7f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 55:
-e      7f:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 56:
-e      7f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 57:
-e      7f:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 58:
-e      08:00.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU Group 59:
-e      09:00.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU Group 60:
-e      09:01.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU Group 61:
-e      0a:00.0 Multimedia audio controller [0401]: RME Fireface UFX+ [1d18:0001] (rev 01)
 
Last edited:
deliad said:
FIRST OF ALL, don't point me to read other threads, I've already read all of them :)
Click to expand...
Sure, I won't provide any links and you can search them yourself.
deliad said:
I've successfully pass through my VGA Controller to the vm NVIDIA Corporation GK208B [GeForce GT 710]
Click to expand...
Was it connected via Thunderbolt? Or just an example that passthrough is possible on your system?
deliad said:
Now I'm interested to pass through all the Intel Corporation JHL7540 Thunderbolt 3 [Titan Ridge 4C 2018] to a VM but it seems that not all the Thunderbolt devices are using the vfio-pci driver instead of it I keep seeing that in some Thunderbolt 3 devices it shows
Code: 
Kernel driver in use: pcieport
although I ignored it via
Code: 
vfio-pci ids

actually, I will be satisfied if I could passthrough only specific - thunderbolt 3-connected devices to the VM
Click to expand...
I remember that people could passthrough connected Thuderbolt devices but not the Thunderbolt controller itself. There are at least two threads about it. And therefore not get hotplug of Thunderbolt devices in a VM working. I would suggest not blacklising thunderbold (or anything really) and just try passthrough of the separate devices (which are sometimes together) like the GT710.
deliad said:
but when I'm trying to only passthrough my RME UFX+ audio controller that connected via Thunderbolt 3 cable to the #1 of 2 total Thunderbolt ports I have, I'm getting this QEMU Error:
Code: 
kvm: ../hw/pci/pci.c:1613: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
Click to expand...
I have never seen anyone resolve that error, except that it sometimes was caused by a wrong configuration and disappears after a reboot.
deliad said:
after alot of threads reading i found out that other users encountered this error when another devices was in the same IOMMU Groups toghether with not passthrough-wanted devices, but it's not my issue because i'm already using the
Code: 
pcie_acs_override=downstream,multifunction
that splitted all my devices to it's own group.
Click to expand...
pcie_acs_override does not guarantee that passthrough will work, it just makes Proxmox ignore the actual IOMMU groups. Please try not to use it and show the complete IOMMU groups in a readable way (without pcie_acs_override) using cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done and use CODE-tags for better readability instead of empty lines.
deliad said:
why it keep showing
Code: 
Kernel driver in use: pcieport
although I ignored it via
Code: 
vfio-pci ids
?
Click to expand...
You cannot use vfio-pci for PCI(e) Brides and you can normally just ignore those, as vfio-pci takes care of that when passing the device under it.
 
leesteken said:
Sure, I won't provide any links and you can search them yourself.

Was it connected via Thunderbolt? Or just an example that passthrough is possible on your system?
Click to expand...
correct, it's just an example i successfully passthrough a pci-e device

the problem is that when i'm trying to passthrough the rme audio interface itself alone (without the thunderbolt devices)
Code: 
IOMMU Group 61:
-e      0a:00.0 Multimedia audio controller [0401]: RME Fireface UFX+ [1d18:0001] (rev 01)

the host's proxmox system reboots when powering the VM up, what crash log i should show you to know why it crashed?
 
Last edited:
deliad said:
the problem is that when i'm trying to passthrough the rme audio interface itself alone (without the thunderbolt devices)
Code: 
IOMMU Group 61:
-e      0a:00.0 Multimedia audio controller [0401]: RME Fireface UFX+ [1d18:0001] (rev 01)

the host's proxmox system reboots when powering the VM up, what crash log i should show you to know why it crashed?
Click to expand...
Please show the complete IOMMU groups without pcie_acs_override using cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done.
Do you know of anyone (on the internet) doing a successful passthrough of that particular device? Maybe it's a device or motherboard issue.
 
# cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n";

lspci -nns "${d##*/}"; done
BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt nomodeset pci=realloc,assign-busses,hpbussize=0x33 pcie_port_pm=off initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction video=efifb:eek:ff video=vesa:eek:ff vfio-pci.ids=10de:128b,19da:6326,8086:0000,8086:15ea,8086:15eb,8086:15ec,8086:1549,1d18:0001 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,rme
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
IOMMU group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
IOMMU group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
IOMMU group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
IOMMU group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
IOMMU group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
IOMMU group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
IOMMU group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU group 14 01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU group 15 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 16 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 17 02:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 18 02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 19 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 1 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20 02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 21 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 22 02:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 23 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU group 24 04:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03)
IOMMU group 25 05:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 26 06:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 27 06:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 28 06:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 29 06:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 30 07:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06)
IOMMU group 31 3b:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06)
IOMMU group 32 6f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 33 70:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 34 70:02.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 35 70:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 36 70:04.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 37 70:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 38 70:06.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 39 70:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 40 71:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 41 72:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU group 42 73:00.0 Network controller [0280]: Broadcom Inc. and subsidiaries BCM4352 802.11ac Wireless Network Adapter [14e4:43b1] (rev 03)
IOMMU group 43 74:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 44 78:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)
IOMMU group 45 79:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 46 79:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 47 79:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 48 7a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 49 7b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 50 7c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU group 51 7d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU group 52 7d:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU group 53 7e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 54 7f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 55 7f:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 56 7f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 57 7f:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 58 08:00.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU group 59 09:00.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 60 09:01.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU group 61 0a:00.0 Multimedia audio controller [0401]: RME Fireface UFX+ [1d18:0001] (rev 01)
IOMMU group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
 
Last edited:
deliad said:
it seems ive already provided the same but in different output:

with your command:
Code: 
cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt nomodeset pci=realloc,assign-busses,hpbussize=0x33 pcie_port_pm=off initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction video=efifb:off video=vesa:off vfio-pci.ids=10de:128b,19da:6326,8086:0000,8086:15ea,8086:15eb,8086:15ec,8086:1549,1d18:0001 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,rme
Click to expand...
pcie_acs_override=downstream,multifunction is active, so the information is useless to me, sorry.
 
cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done | sort -n

Code: 
BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt nomodeset pci=realloc,assign-busses,hpbussize=0x33 pcie_port_pm=off initcall_blacklist=sysfb_init video=efifb:off video=vesa:off vfio-pci.ids=10de:128b,19da:6326,8086:0000,8086:15ea,8086:15eb,8086:15ec,8086:1549,1d18:0001 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 1 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
IOMMU group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
IOMMU group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
IOMMU group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
IOMMU group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
IOMMU group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
IOMMU group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
IOMMU group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU group 14 01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU group 15 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 16 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 17 02:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 18 02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 19 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20 02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 20 79:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 20 79:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 20 79:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 21 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 21 7a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 22 02:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 22 7b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 23 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU group 24 04:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03)
IOMMU group 25 05:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 26 06:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 27 06:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 28 06:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 29 06:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 30 07:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06)
IOMMU group 31 3b:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06)
IOMMU group 32 6f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 33 70:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 33 71:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 34 70:02.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 34 72:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU group 35 70:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 35 73:00.0 Network controller [0280]: Broadcom Inc. and subsidiaries BCM4352 802.11ac Wireless Network Adapter [14e4:43b1] (rev 03)
IOMMU group 36 70:04.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 36 74:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 37 70:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 38 70:06.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 39 70:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1187e 7-Port PCIe x1 Gen2 Packet Switch [1b21:1187]
IOMMU group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 40 78:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)
IOMMU group 41 7c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU group 42 7d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU group 42 7d:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU group 43 7e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 44 7f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 45 7f:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 46 7f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 47 7f:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 48 08:00.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU group 49 09:00.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU group 49 0a:00.0 Multimedia audio controller [0401]: RME Fireface UFX+ [1d18:0001] (rev 01)
IOMMU group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 50 09:01.0 PCI bridge [0604]: Intel Corporation DSL2210 Thunderbolt Controller [Port Ridge 1C 2011] [8086:1549]
IOMMU group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]

So , it looks like that the RME Fireface UFX+ [1d18:0001] (rev 01) shares the same IOMMU Group with the thunderbolt pci bridge, which is make sense , i guess it will be the same with any thunderbolt attached device

the problem is that if i passthrough it , when powering up the VM , the proxmox host reboots

what log to look in for debug the cause?

lspci -vv shows the following on it:
Code: 
0a:00.0 Multimedia audio controller: RME Fireface UFX+ (rev 01)
        Subsystem: RME Fireface UFX+
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 255
        IOMMU group: 49
        Region 0: Memory at d4000000 (32-bit, non-prefetchable) [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Kernel driver in use: vfio-pci
 
Last edited:
deliad said:
So , it looks like that the RME Fireface UFX+ [1d18:0001] (rev 01) shares the same IOMMU Group with the thunderbolt pci bridge, which is make sense , i guess it will be the same with any thunderbolt attached device
Click to expand...
PCI(e) Bridges should not be a problem and indeed it is in a isolated group, so passthrough won't fail because of that. And you probably don't need pcie_acs_override.
deliad said:
the problem is that if i passthrough it , when powering up the VM , the proxmox host reboots

what log to look in for debug the cause?
Click to expand...
There probably is nothing in the logs (journalctl scroll to the time of starting the VM) due to the sudden system reset.
Maybe your pci=realloc,assign-busses,hpbussize=0x33 pcie_port_pm=off are interfering? Or maybe the device or motherboard/thunderbolt is just not compatible with passthrough. Or find someone on the internet that has success with it and see what special work-arounds are necessary.
Maybe the device supports different reset methods? Check with cat '/sys/bus/pci/devices/0000:0a:00.0/reset_method' and write the method that you want to try to it, before starting the VM.
 
Code: 
2023-07-04T17:02:11.161665+03:00 deliad1upve qm[3328]: <root@pam> starting task UPID:deliad1upve:00000D01:00012201:64A42663:qmstop:100:root@pam:
2023-07-04T17:02:11.162013+03:00 deliad1upve qm[3329]: stop VM 100: UPID:deliad1upve:00000D01:00012201:64A42663:qmstop:100:root@pam:
2023-07-04T17:02:11.169574+03:00 deliad1upve qm[3328]: <root@pam> end task UPID:deliad1upve:00000D01:00012201:64A42663:qmstop:100:root@pam: OK
2023-07-04T17:02:11.591940+03:00 deliad1upve qm[3330]: <root@pam> starting task UPID:deliad1upve:00000D03:0001222C:64A42663:qmstop:100:root@pam:
2023-07-04T17:02:11.592070+03:00 deliad1upve qm[3331]: stop VM 100: UPID:deliad1upve:00000D03:0001222C:64A42663:qmstop:100:root@pam:
2023-07-04T17:02:11.600285+03:00 deliad1upve qm[3330]: <root@pam> end task UPID:deliad1upve:00000D03:0001222C:64A42663:qmstop:100:root@pam: OK
2023-07-04T17:02:17.049621+03:00 deliad1upve qm[3334]: <root@pam> starting task UPID:deliad1upve:00000D07:0001244D:64A42669:qmstop:100:root@pam:
2023-07-04T17:02:17.049842+03:00 deliad1upve qm[3335]: stop VM 100: UPID:deliad1upve:00000D07:0001244D:64A42669:qmstop:100:root@pam:
2023-07-04T17:02:17.053481+03:00 deliad1upve qm[3334]: <root@pam> end task UPID:deliad1upve:00000D07:0001244D:64A42669:qmstop:100:root@pam: OK
2023-07-04T17:02:17.503577+03:00 deliad1upve qm[3337]: stop VM 100: UPID:deliad1upve:00000D09:0001247B:64A42669:qmstop:100:root@pam:
2023-07-04T17:02:17.503707+03:00 deliad1upve qm[3336]: <root@pam> starting task UPID:deliad1upve:00000D09:0001247B:64A42669:qmstop:100:root@pam:
2023-07-04T17:02:17.511594+03:00 deliad1upve qm[3336]: <root@pam> end task UPID:deliad1upve:00000D09:0001247B:64A42669:qmstop:100:root@pam: OK
2023-07-04T17:02:17.955117+03:00 deliad1upve qm[3338]: <root@pam> starting task UPID:deliad1upve:00000D1C:000124A7:64A42669:qmstart:100:root@pam:
2023-07-04T17:02:17.955255+03:00 deliad1upve qm[3356]: start VM 100: UPID:deliad1upve:00000D1C:000124A7:64A42669:qmstart:100:root@pam:
2023-07-04T17:02:18.178145+03:00 deliad1upve kernel: [  749.386946] vfio-pci 0000:0a:00.0: refused to change power state from D3hot to D0
2023-07-04T17:02:20.382071+03:00 deliad1upve kernel: [  751.631007] clocksource: timekeeping watchdog on CPU28: hpet wd-wd read-back delay of 201705174ns
2023-07-04T17:02:20.382077+03:00 deliad1upve kernel: [  751.631011] clocksource: wd-tsc-wd read-back delay of 201705384ns, clock-skew test skipped!
2023-07-04T17:02:21.792135+03:00 deliad1upve kernel: [  752.236123] xhci_hcd 0000:3b:00.0: can't change power state from D3cold to D0 (config space inaccessible)
2023-07-04T17:02:23.809192+03:00 deliad1upve kernel: [  754.656586] xhci_hcd 0000:3b:00.0: can't change power state from D3cold to D0 (config space inaccessible)
2023-07-04T17:02:24.616034+03:00 deliad1upve kernel: [  755.866816] xhci_hcd 0000:3b:00.0: Controller not ready at resume -19
2023-07-04T17:02:24.616039+03:00 deliad1upve kernel: [  755.866821] xhci_hcd 0000:3b:00.0: PCI post-resume error -19!
2023-07-04T17:02:24.616040+03:00 deliad1upve kernel: [  755.866824] xhci_hcd 0000:3b:00.0: HC died; cleaning up
2023-07-04T17:02:27.642116+03:00 deliad1upve kernel: [  758.489008] atlantic 0000:78:00.0 enp120s0: atlantic: link change old 1000 new 10000
2023-07-04T17:02:31.877538+03:00 deliad1upve kernel: [  762.724847] sched: RT throttling activated
2023-07-04T17:02:31.877545+03:00 deliad1upve kernel: [  762.724852] atlantic 0000:78:00.0 enp120s0: atlantic: link change old 1000 new 10000

[AFTER THAT THE SYSTEM REBOOTED!]

So here are the reason, any thoughts?
 
leesteken said:
PCI(e) Bridges should not be a problem and indeed it is in a isolated group, so passthrough won't fail because of that. And you probably don't need pcie_acs_override.

There probably is nothing in the logs (journalctl scroll to the time of starting the VM) due to the sudden system reset.
Maybe your pci=realloc,assign-busses,hpbussize=0x33 pcie_port_pm=off
are interfering?
Click to expand...
i try to remove it , but if i remember well that's was the first reason i tried to use it
leesteken said:
Or maybe the device or motherboard/thunderbolt is just not compatible with passthrough. Or find someone on the internet that has success with it and see what special work-arounds are necessary.
Maybe the device supports different reset methods? Check with cat '/sys/bus/pci/devices/0000:0a:00.0/reset_method'
Click to expand...
cat /sys/bus/pci/devices/0000\:0a\:00.0/reset_method shows:
Code: 
bus
 
Last edited:
deliad said:
can we think about some workaround?
Click to expand...
Maybe you can disable the bus reset and see if the system does not instantly reboot? Maybe patch the Linux kernel to not reset this device?
Is there a Linux driver for the device? Study the source code of it and see if there something in there for resetting the device to a clean state. Then use that part of the code to reset it instead of a bus reset, like vendor-reset does for some generations of AMD GPUs.
EDIT: Maybe contact the manufacturer and ask them about it?
 
i'm trying to find how to disable it but i don't have a clue

can you suggest me?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back