Ryzen 7000 - Internal GPU Passtrough

[Hativ3]

I'm tyring to passtrough the internal GPU of my Ryzen 7950X to a Linux VM.

Passthrough is configured like described here:

1. IOMMU enabled in BIOS
2. Added to /etc/modules:
   

   vfio
   vfio_iommu_type1
   vfio_pci
   vfio_virqfd

3. That's it.

So amdgpu is not blacklisted and handles the takeover to vfio. I tried the "old method" as well, with blacklisting amdgpu etc., but had the same result.

I see all boot messages and the login screen. After starting the VM the screen just get's black. I don't see any output from the VM.

Any ideas?

Proxmox VE 7.3, Linux kernel 5.19

Mainboard has HDMI and DisplayPort output, I use DisplayPort. But also checked if HDMI outputs anything.

journalctl -r after starting the VM:

Dec 02 12:09:02 proxmox-host pvedaemon[2438]: <root@pam> end task UPID:proxmox-host:00000CB8:00001F2B:6389DCCC:qmstart:100:root@pam: OK
Dec 02 12:09:02 proxmox-host kernel: vfio-pci 0000:13:00.0: vfio_ecap_init: hiding ecap 0x27@0x450
Dec 02 12:09:02 proxmox-host kernel: vfio-pci 0000:13:00.0: vfio_ecap_init: hiding ecap 0x26@0x410
Dec 02 12:09:02 proxmox-host kernel: vfio-pci 0000:13:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
Dec 02 12:09:02 proxmox-host kernel: vfio-pci 0000:13:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Dec 02 12:09:01 proxmox-host kernel: device fwln100i0 entered promiscuous mode
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Dec 02 12:09:01 proxmox-host kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Dec 02 12:09:01 proxmox-host kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Dec 02 12:09:01 proxmox-host kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Dec 02 12:09:01 proxmox-host kernel: device fwpr100p0 entered promiscuous mode
Dec 02 12:09:01 proxmox-host kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Dec 02 12:09:01 proxmox-host kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Dec 02 12:09:01 proxmox-host systemd-udevd[3264]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 02 12:09:01 proxmox-host systemd-udevd[3262]: Using default interface naming scheme 'v247'.
Dec 02 12:09:01 proxmox-host systemd-udevd[3262]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 02 12:09:01 proxmox-host systemd-udevd[3264]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 02 12:09:01 proxmox-host kernel: device tap100i0 entered promiscuous mode
Dec 02 12:09:01 proxmox-host systemd-udevd[3264]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 02 12:09:01 proxmox-host systemd-udevd[3264]: Using default interface naming scheme 'v247'.
Dec 02 12:09:01 proxmox-host systemd[1]: Started 100.scope.
Dec 02 12:09:01 proxmox-host systemd[1]: Created slice qemu.slice.
Dec 02 12:09:01 proxmox-host systemd[1]: Stopped target Sound Card.
Dec 02 12:09:00 proxmox-host kernel: vfio-pci 0000:13:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:eek:wns=none
Dec 02 12:09:00 proxmox-host kernel: [drm] amdgpu: ttm finalized
Dec 02 12:09:00 proxmox-host kernel: [drm] free PSP TMR buffer
Dec 02 12:09:00 proxmox-host kernel: amdgpu 0000:13:00.0: amdgpu: amdgpu: finishing device.
Dec 02 12:09:00 proxmox-host kernel: Console: switching to colour dummy device 80x25
Dec 02 12:09:00 proxmox-host pvedaemon[2438]: <root@pam> starting task UPID:proxmox-host:00000CB8:00001F2B:6389DCCC:qmstart:100:root@pam:
Dec 02 12:09:00 proxmox-host pvedaemon[3256]: start VM 100: UPID:proxmox-host:00000CB8:00001F2B:6389DCCC:qmstart:100:root@pam:

lspci -v output on the host:

[..]
13:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 164e (rev c1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd Device d000
        Flags: bus master, fast devsel, latency 0, IRQ 99, IOMMU group 20
        Memory at fce0000000 (64-bit, prefetchable) [size=256M]
        Memory at fcf0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at f000
        Memory at fcd00000 (32-bit, non-prefetchable) [size=512K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [64] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [270] Secondary PCI Express
        Capabilities: [2a0] Access Control Services
        Capabilities: [2b0] Address Translation Service (ATS)
        Capabilities: [2c0] Page Request Interface (PRI)
        Capabilities: [2d0] Process Address Space ID (PASID)
        Capabilities: [410] Physical Layer 16.0 GT/s <?>
        Capabilities: [450] Lane Margining at the Receiver <?>
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

13:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1640
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1640
        Flags: bus master, fast devsel, latency 0, IRQ 136, IOMMU group 21
        Memory at fcd88000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [64] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [2a0] Access Control Services
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

[..]

13:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
        DeviceName: Realtek ALC1220
        Subsystem: Gigabyte Technology Co., Ltd Family 17h (Models 10h-1fh) HD Audio Controller
        Flags: bus master, fast devsel, latency 0, IRQ 137, IOMMU group 25
        Memory at fcd80000 (32-bit, non-prefetchable) [size=32K]
       Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [64] Express Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [2a0] Access Control Services
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
[..]

VM config:

agent: 0
bios: ovmf
boot: order=ide2
cores: 8
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:13:00.0,pcie=1,x-vga=1
hostpci1: 0000:13:00.1,pcie=1
hostpci2: 0000:13:00.6,pcie=1
ide2: local:iso/manjaro-kde-21.3.7-220816-linux515.iso,media=cdrom,size=3487954K
machine: q35
memory: 16384
meta: creation-qemu=7.1.0,ctime=1669914299
name: linux
net0: virtio=52:7C:EB:47:D3:48,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-zfs:vm-100-disk-1,iothread=1,size=250G
scsihw: virtio-scsi-single
smbios1: uuid=7d084749-34ed-426e-81ad-9e93b755b10b
sockets: 1
vmgenid: 07a3c7f5-00f0-4d4c-b811-bf64cecf19d2

lspci -v from the VM:

 

[Hativ3]

Tried Linux kernel 6.0, but nothing changed.

Does it possibly need a romfile?

 

[AndrwHmmr]

Is it even possible to passthrough the only gpu in your system?

 

[leesteken]

Is it even possible to passthrough the only gpu in your system?

Yes but you probably need this work-around when passing through the boot GPU.
Note that passthrough of integrated graphics (APU, iGPU) can be more complicated than discrete GPUs and is not always possible.

 

[isc30]

Seems like this guide works
https://github.com/isc30/ryzen-7000-series-proxmox/tree/main