[SOLVED] AMD GPU Passthrough

C

compiz

Member
Dec 29, 2022
31
1
13
Hi and Happy New Year everybody!
I have a mini pc ASRock DeskMeet B660 with an intel i3 13100 CPU (with iGPU) and a RX6600 GPU. Also a coral.ai TPU on NVMe slot 2.
I want to make 2 VMs, 1 to have the coral.ai TPU running ubuntu 22.04, this I made it with cloud image and it works no problem.
I have problem now with the 2nd VM that is Windows 11. I have enabled the CSM mode in the BIOS which for one made me lose the iGPU entirely and I am talking with ASRock to if and how I can get also the iGPU to work.
So where I need help, before I had unraid, in unraid I had to enable CSM and my GPU was working with a windows 11 vm without problems. But I couldn't make the coral TPU to work, it could see it with their driver, I could pass it to VM but I was having crashes and couldn't get logs to see what was wrong.
This is where I said ok, let's try proxmox for GPU passthrough!
Do I need to have CSM enabled in proxmox? Or can I do a bios reset and use UEFI? Always talking about passthrough of hardware.
I have seen a few videos in youtube and read a few guides but it doesn't seem to help me.

I am posting my current configs:
Code: 
$nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

$update-grub

$nano /etc/kernel/cmdline
%> intel_iommu=on
[ICODE]
$proxmox-boot-tool refresh

$nano /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

$nano /etc/modprobe.d/blacklist.conf
blacklist amdgpu
blacklist radeon
blacklist nouveau
blacklist nvidia

$lspci
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller

$lspci -n -s 03:00 -v
03:00.0 0300: 1002:73ff (rev c7) (prog-if 00 [VGA controller])
        Subsystem: 1da2:e451
        Flags: fast devsel, IRQ 16, IOMMU group 15
        Memory at 90000000 (64-bit, prefetchable) [size=256M]
        Memory at a0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at 3000 [size=256]
        Memory at 72100000 (32-bit, non-prefetchable) [size=1M]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [64] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [200] Physical Resizable BAR
        Capabilities: [240] Power Budgeting <?>
        Capabilities: [270] Secondary PCI Express
        Capabilities: [2a0] Access Control Services
        Capabilities: [2d0] Process Address Space ID (PASID)
        Capabilities: [320] Latency Tolerance Reporting
        Capabilities: [410] Physical Layer 16.0 GT/s <?>
        Capabilities: [440] Lane Margining at the Receiver <?>
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu


03:00.1 0403: 1002:ab28
        Subsystem: 1002:ab28
        Flags: fast devsel, IRQ 17, IOMMU group 16
        Memory at 72220000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [64] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [2a0] Access Control Services
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

$echo "options vfio-pci ids=1002:73ff,1002:ab28 disable_vga=1"> /etc/modprobe.d/vfio.conf
$update-initramfs -u -k all
$reboot

After that I have tried to make a mapped device that includes the GPU and its audio
For the VM, I use Q35 and OVMF but no matter what I use in the GPU passthrough, I always get noVNC and no video signal on the monitor attached to the GPU.

Obviously I have done something wrong, can someone guide me how fix this?
Last resort for me will be to use windows as host OS but really don't want to xD
 
compiz said:
I have enabled the CSM mode in the BIOS which for one made me lose the iGPU entirely and I am talking with ASRock to if and how I can get also the iGPU to work.
Click to expand...
If the Radeon 6600 is used during boot, you probably need this work-around: https://forum.proxmox.com/posts/478351/
compiz said:
Do I need to have CSM enabled in proxmox? Or can I do a bios reset and use UEFI?
Click to expand...
You might need to fix the Proxmox boot if you switch between Legacy Mode and UEFI boot. Just try it? Or maybe reinstall might be easier?
compiz said:
I have seen a few videos in youtube and read a few guides but it doesn't seem to help me.
Click to expand...
Here is the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough . But there are also related Wiki pages.
 
leesteken said:
If the Radeon 6600 is used during boot, you probably need this work-around: https://forum.proxmox.com/posts/478351/

You might need to fix the Proxmox boot if you switch between Legacy Mode and UEFI boot. Just try it? Or maybe reinstall might be easier?

Here is the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough . But there are also related Wiki pages.
Click to expand...
I will try the work-around thanks!
Does the hardware passthrough works in UEFI boot? Because as far as I can tell in unraid it didn't work, so assumed the same here
 
compiz said:
Does the hardware passthrough works in UEFI boot? Because as far as I can tell in unraid it didn't work, so assumed the same here
Click to expand...
Usually PCI(e) passthrough is unrelated to the boot mode. My system boots in UEFI mode and does passthrough just fine. I cannot comment on why unraid stuff works or does not.

EDIT: Make sure to disable Resizable BAR (Smart Memory Access/SMA/CMA). Maybe disabling Above 4G Decoding fixes your Coral TPU issue?
 
Last edited:
  • Like
Reactions: compiz
leesteken said:
Usually PCI(e) passthrough is unrelated to the boot mode. My system boots in UEFI mode and does passthrough just fine. I cannot comment on why unraid stuff works or does not.

EDIT: Make sure to disable Resizable BAR (Smart Memory Access/SMA/CMA). Maybe disabling Above 4G Decoding fixes your Coral TPU issue?
Click to expand...
I have the rebar off and all those options, the coral tpu plays ok (having issues with get video feed from dahua camera to the frigate but that's not the current issue), coral tpu gets detected and powers on ok in the ubuntu 22.04 vm with passthrough just that.
Main issue is the GPU now for the 2nd vm, which i will try to fix now with bios reset
 
I have connected a monitor now and did a reboot, i see in the monitor:
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Measured initrd data into PCR 9

And stay like that forever
 
compiz said:
I have connected a monitor now and did a reboot, i see in the monitor:
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Measured initrd data into PCR 9

And stay like that forever
Click to expand...
Maybe that's because you blacklisted the 6600 and used the work-around? Does it start and can you connect via SSH and web GUI?
 
I am in the webgui and ssh works, this is just what i see in the screen connected to it.
I have closed CSM and enabled UEFI.
Made sure all rebar options are disabled(4g etc) and now I have also enabled the iGPU which is nice!
Let me retry!
 
Now it seems to work in both vms!
I can get video output from the GPU on the windows 11 vm and I can also use the TPU in the ubuntu vm!
So I guess in proxmox UEFI is needed for it to work while on unraid not (at least in my case, can't tell for all)
Thanks for the help!
 
You must log in or register to reply here.
Top Bottom