IOMMU groups get disconnected when starting VM that uses different group

P

Polyphemus

Member
Nov 18, 2021
46
9
13
44
Hi,

I just upgraded my server to a Ryzen 5 5600G on a Gigabyte A520M H.
Running kernel 5.15 because of errors (DID_BAD_TARGET) on my NVME SSD when shutting the server down, when I attach my Coral (in its own IOMMU group 04) to my Debian VM, other IOMMU groups (that aren't passed through to the VM) get disconnected when i start the VM:

Code: 
Feb  6 13:35:32 pve pvedaemon[1541]: start VM 104: UPID:pve:00000605:00002B64:61FFC094:qmstart:104:root@pam:
Feb  6 13:35:32 pve pvedaemon[1113]: <root@pam> starting task UPID:pve:00000605:00002B64:61FFC094:qmstart:104:root@pam:
Feb  6 13:35:32 pve kernel: [  111.116341] xhci_hcd 0000:02:00.0: remove, state 4
Feb  6 13:35:32 pve kernel: [  111.116346] usb usb2: USB disconnect, device number 1
Feb  6 13:35:32 pve kernel: [  111.116603] xhci_hcd 0000:02:00.0: USB bus 2 deregistered
Feb  6 13:35:32 pve kernel: [  111.116610] xhci_hcd 0000:02:00.0: remove, state 1
Feb  6 13:35:32 pve kernel: [  111.116613] usb usb1: USB disconnect, device number 1
Feb  6 13:35:32 pve kernel: [  111.116614] usb 1-3: USB disconnect, device number 2
Feb  6 13:35:32 pve kernel: [  111.116615] usb 1-3.2: USB disconnect, device number 4
Feb  6 13:35:32 pve kernel: [  111.327643] usb 1-9: USB disconnect, device number 3
Feb  6 13:35:32 pve kernel: [  111.385487] xhci_hcd 0000:02:00.0: USB bus 1 deregistered
Feb  6 13:35:32 pve kernel: [  111.655202] r8169 0000:05:00.0 enp5s0: Link is Down
Feb  6 13:35:32 pve kernel: [  111.662902] vmbr0: port 1(enp5s0) entered disabled state
Feb  6 13:35:32 pve kernel: [  111.663142] device enp5s0 left promiscuous mode
Feb  6 13:35:32 pve kernel: [  111.663147] vmbr0: port 1(enp5s0) entered disabled state
Feb  6 13:35:34 pve systemd[1]: Started 104.scope.
Feb  6 13:35:34 pve systemd-udevd[1548]: Using default interface naming scheme 'v247'.
Feb  6 13:35:34 pve systemd-udevd[1548]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  6 13:35:34 pve kernel: [  113.310128] device tap104i0 entered promiscuous mode
Feb  6 13:35:34 pve kernel: [  113.315441] vmbr1: port 2(tap104i0) entered blocking state
Feb  6 13:35:34 pve kernel: [  113.315444] vmbr1: port 2(tap104i0) entered disabled state
Feb  6 13:35:34 pve kernel: [  113.315518] vmbr1: port 2(tap104i0) entered blocking state
Feb  6 13:35:34 pve kernel: [  113.315520] vmbr1: port 2(tap104i0) entered forwarding state
Feb  6 13:35:35 pve kernel: [  113.997980] vfio-pci 0000:04:00.0: enabling device (0000 -> 0002)
Feb  6 13:35:36 pve kernel: [  115.023006] vfio-pci 0000:04:00.0: vfio_ecap_init: hiding ecap 0x1e@0x110

My
Code: 
lspci -nnk
is as follows:

Code: 
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex [1022:1630]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex [1022:1630]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU [1022:1631]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU [1022:1631]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
        Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
        Kernel driver in use: pcieport
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
        Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
        Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
        Subsystem: Gigabyte Technology Co., Ltd FCH SMBus Controller [1458:5001]
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
        Subsystem: Gigabyte Technology Co., Ltd FCH LPC Bridge [1458:5001]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d]
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671]
01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
        Subsystem: Fujitsu Technology Solutions I350 Gigabit Network Connection [1734:11d1]
        Kernel driver in use: igb
        Kernel modules: igb
01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
        Subsystem: Fujitsu Technology Solutions I350 Gigabit Network Connection [1734:11d1]
        Kernel driver in use: igb
        Kernel modules: igb
01:00.2 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
        Subsystem: Fujitsu Technology Solutions I350 Gigabit Network Connection [1734:11d1]
        Kernel driver in use: igb
        Kernel modules: igb
01:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
        Subsystem: Fujitsu Technology Solutions I350 Gigabit Network Connection [1734:11d1]
        Kernel driver in use: igb
        Kernel modules: igb
02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ec]
        Subsystem: ASMedia Technology Inc. Device [1b21:1142]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
        Subsystem: ASMedia Technology Inc. Device [1b21:1062]
        Kernel driver in use: ahci
        Kernel modules: ahci
02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
        Kernel driver in use: pcieport
03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
        Kernel driver in use: pcieport
03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
        Kernel driver in use: pcieport
04:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]
        Subsystem: Global Unichip Corp. Coral Edge TPU [1ac1:089a]
        Kernel driver in use: vfio-pci
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 16)
        Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet [1458:e000]
        Kernel driver in use: r8169
        Kernel modules: r8169
06:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5019] (rev 01)
        Subsystem: Sandisk Corp Device [15b7:5019]
        Kernel driver in use: nvme
        Kernel modules: nvme
07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev c9)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
07:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
        Kernel driver in use: ccp
        Kernel modules: ccp
07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
        Subsystem: Gigabyte Technology Co., Ltd Renoir USB 3.1 [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
07:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
        Subsystem: Gigabyte Technology Co., Ltd Renoir USB 3.1 [1458:5007]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci

Using
Code: 
pcie_acs_override=downstream
in grub does not solve the issue.

pveversion:

Code: 
proxmox-ve: 7.1-1 (running kernel: 5.15.17-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-5.15: 7.1-9
pve-kernel-helper: 7.1-9
pve-kernel-5.13: 7.1-7
pve-kernel-5.15.17-1-pve: 5.15.17-1
pve-kernel-5.13.19-4-pve: 5.13.19-8
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-1
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

Can anybody give me a hint?
Thanks!
 
Last edited:
Can you please show your IOMMU groups using this command? for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
And can you attach the VM configuration file from the /etc/pve/qemu-server/ directory?
 
Hi @avw

Code: 
root@pve:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 10 01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 11 01:00.2 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 12 01:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 13 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ec]
IOMMU group 13 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU group 13 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU group 13 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 13 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 13 04:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]
IOMMU group 13 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 16)
IOMMU group 14 06:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5019] (rev 01)
IOMMU group 15 07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev c9)
IOMMU group 16 07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
IOMMU group 17 07:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
IOMMU group 18 07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 19 07:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
IOMMU group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 3 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 4 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 6 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 7 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
IOMMU group 7 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 8 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a]
IOMMU group 8 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b]
IOMMU group 8 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c]
IOMMU group 8 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d]
IOMMU group 8 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e]
IOMMU group 8 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f]
IOMMU group 8 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670]
IOMMU group 8 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671]
IOMMU group 9 01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)

Code: 
root@pve:/etc/pve/qemu-server# cat 104.conf
agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 2
cpu: host
hostpci0: 0000:04:00.0
ide2: none,media=cdrom
machine: q35
memory: 6144
meta: creation-qemu=6.1.0,ctime=1637912757
name: argus
net0: virtio=DE:9F:A2:A7:3A:4C,bridge=vmbr1
numa: 0
ostype: l26
scsi0: local-lvm:vm-104-disk-0,size=160G
scsihw: virtio-scsi-pci
smbios1: uuid=77df0e76-31f2-4be1-937e-59f86a8fddda
sockets: 2
vmgenid: 0f320cab-1f12-4049-97f8-14b3c906d394
 
Your PCIe device 04:00.0 is in IOMMU group 13, which also contains USB and SATA controllers. A IOMMU group cannot be shared between VMs or between a VM and the Proxmox host. Therefore, when you pass one device a group to a VM, the host loses connection to all devices in that group. This explains the behavior you are experiencing.
The motherboard and BIOS determine the IOMMU groups and you can only move device between groups by plugging it into another PCIe slot.
On Ryzen CPU and motherboards only some PCIe x16 slots, one M.2 and sometimes audio and one USB controller are in isolated groups (which are provided by the CPU). Everything else is in a single "chipset" group. Only the X570 chipset is much more flexible in its IOMMU groups.

EDIT: If you don't mind that the VM can access all host memory using DMA via PCI(e) devices, you could consider adding pcie_acs_override=downstream,multifunction to the kernel parameters.
 
Last edited:
avw said:
Your PCIe device 04:00.0 is in IOMMU group 13, which also contains USB and SATA controllers. A IOMMU group cannot be shared between VMs or between a VM and the Proxmox host. Therefore, when you pass one device a group to a VM, the host loses connection to all devices in that group. This explains the behavior you are experiencing.
The motherboard and BIOS determine the IOMMU groups and you can only move device between groups by plugging it into another PCIe slot.
On Ryzen CPU and motherboards only some PCIe x16 slots, one M.2 and sometimes audio and one USB controller are in isolated groups (which are provided by the CPU). Everything else is in a single "chipset" group. Only the X570 chipset is much more flexible in its IOMMU groups.

EDIT: If you don't mind that the VM can access all host memory using DMA via PCI(e) devices, you could consider adding pcie_acs_override=downstream,multifunction to the kernel parameters.
Click to expand...
adding pcie_acs_override=downstream,multifunction does not resolve it.

I will try if it works in a different PCIe slot :)
 
Polyphemus said:
adding pcie_acs_override=downstream,multifunction does not resolve it.

I will try if it works in a different PCIe slot :)
Click to expand...
Unless you can use the x16 slot or the M.2 slot, it won't help on your motherboard.
The pcie_acs_override=downstream,multifunction always changes the IOMMU groups. If it did nothing, then it is not correctly applied. What is the output of cat /proc/cmdline when adding it?
 
avw said:
Unless you can use the x16 slot or the M.2 slot, it won't help on your motherboard.
The pcie_acs_override=downstream,multifunction always changes the IOMMU groups. If it did nothing, then it is not correctly applied. What is the output of cat /proc/cmdline when adding it?
Click to expand...
I'm trying the x16 slot now, having to reinstall again because now the onboard LAN isn't recognized :rolleyes:
 
Polyphemus said:
I'm trying the x16 slot now, having to reinstall again because now the onboard LAN isn't recognized :rolleyes:
Click to expand...
When you add, remove or move PCI(e) devices, the PCI bus addresses (like 05:00.0) tend to shift by 1 (like 06:00.0 or 04:00.0). Unfortunately, the name of the network devices is determined by the PCI bus address (like enp5s0 for 05:00.0) and that will change by 1 also. You'll need to manually change all occurrences of the old name to the new name in the /etc/network/interfaces file.
EDIT: You can use ip a or journalctl -b 0 | grep renamed to find the new name.
 
Last edited:
  • Like
Reactions: Polyphemus
I'm restoring my VMs now, the Coral now sits in its own group (checked with the command you gave me). I had a 4 port NIC in the x16 slot, but I will manage to live without that (was intended for future use).

Before the upgrade I had a 4 core i3-9100 on an H310 chipset, which worked flawlessly with all IOMMU groups. I have promoted that hardware to a storage server. Silly me, I thought I wouldn't be limited by the A520 chipset...

Many thanks for your directions :)
 
I have a simliar Problem, AMD 5600G but I get the Error:

TASK ERROR: Cannot open iommu_group: No such file or directory

If I open the PCI Device on the webinterface the Device has no iommu group, any idea?
 
Tim1 said:
I have a simliar Problem, AMD 5600G but I get the Error:

TASK ERROR: Cannot open iommu_group: No such file or directory

If I open the PCI Device on the webinterface the Device has no iommu group, any idea?
Click to expand...
Please show your IOMMU groups (for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done) and tell us which devices you are passing through in the VM (or maybe even your VM configuration file).
 
leesteken said:
Please show your IOMMU groups (for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done) and tell us which devices you are passing through in the VM (or maybe even your VM configuration file).
Click to expand...
root@pve:/var/lock/qemu-server# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; doneIOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 10 02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 11 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev c9)
IOMMU group 12 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
IOMMU group 13 03:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
IOMMU group 14 03:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 15 03:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU group 16 03:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
IOMMU group 17 04:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
IOMMU group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 3 00:02.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 4 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 6 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 7 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
IOMMU group 7 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 8 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a]
IOMMU group 8 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b]
IOMMU group 8 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c]
IOMMU group 8 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d]
IOMMU group 8 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e]
IOMMU group 8 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f]
IOMMU group 8 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670]
IOMMU group 8 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671]
IOMMU group 9 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a809]
root@pve:/var/lock/qemu-server#
 
leesteken said:
Please show your IOMMU groups (for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done) and tell us which devices you are passing through in the VM (or maybe even your VM configuration file).
Click to expand...
I can add it to the vm, but it doesn't works

1670501228646.png
 
Please show the VM configuration file (from the /etc/pve/qemu-server/ directory) as it looks like you are passing all functions of 03:00 and that won't work. You need to pass the VGA and Audio function but the other functions need to stay at the host.
 
leesteken said:
Please show the VM configuration file (from the /etc/pve/qemu-server/ directory) as it looks like you are passing all functions of 03:00 and that won't work. You need to pass the VGA and Audio function but the other functions need to stay at the host.
Click to expand...


agent: 1,fstrim_cloned_disks=1
args: -device intel-hda,id=sound5,bus=pci.0,addr=0x18 -device hda-micro,id=sound5-co>
balloon: 0
bios: ovmf
boot: order=ide0;net0
cores: 2
cpu: host
efidisk0: VMStorage:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:03:00
hotplug: disk,network,usb,memory,cpu
ide0: VMStorage:vm-100-disk-1,size=100G,ssd=1
machine: pc-q35-5.2
memory: 8192
meta: creation-qemu=6.2.0,ctime=1652681312
name: WIN11-Desktop
net0: rtl8139=06:01:B5:A6:97:A2,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=c115f238-9a8f-4c30-a144-d290b61df3de
sockets: 2
tpmstate0: VMStorage:vm-100-disk-2,size=4M,version=v2.0
usb0: host=1-1,usb3=1
usb1: host=0586:3426,usb3=1
vga: none
vmgenid: 885b4191-c916-4805-822c-1cb92e99a35d

I see the igpu in the webinterface, but as not mediated device. (Info)
 
Last edited:
Tim1 said:
args: -device intel-hda,id=sound5,bus=pci.0,addr=0x18 -device hda-micro,id=sound5-co>
Click to expand...
A virtual audio device can nowadays be done via Proxmox (GUI: intel-hda) instead of args. (and the line was cut off)
Tim1 said:
hostpci0: 0000:03:00
Click to expand...
You are passing all functions (as I expected). Don't do that and only pass VGA and Audio functions.
Tim1 said:
hotplug: disk,network,usb,memory,cpu
Click to expand...
Don't hotplug memory unless you configured the VM to handle memory hotplug, otherwise it'll only have 1GB.
Tim1 said:
I see the igpu in the webinterface, but as not mediated device. (Info)
Click to expand...
I don't think you can use an AMD APU as a mediated device. It looks like you are doing PCIe passthrough and not mediated graphics like Intel GVT or NVidia vGPU.
 
leesteken said:
A virtual audio device can nowadays be done via Proxmox (GUI: intel-hda) instead of args. (and the line was cut off)

You are passing all functions (as I expected). Don't do that and only pass VGA and Audio functions.

Don't hotplug memory unless you configured the VM to handle memory hotplug, otherwise it'll only have 1GB.

I don't think you can use an AMD APU as a mediated device. It looks like you are doing PCIe passthrough and not mediated graphics like Intel GVT or NVidia vGPU.
Click to expand...
Ok, thanks

You mean it is not possible with a AMD CPU?
 
I have a Win11 VM, here I want to project some streams to my tv.

Because of no graphics it is at the moment not possible to project the screen.
 
You must log in or register to reply here.
Top Bottom