GPU can't passthrough to vm successfully with RX470

absolutracy

New Member
Aug 17, 2022
3
0
1
Hello everyone, recently I read a post written by Nicholas Sherlock. The post showed a way to install macOS on proxmox. I followed the instructions and was able to install the macOS Monterey on proxmox successfully. When I came to the gpu passing-through part, I tried a lot of methods posted on the Internet and spent hours and hours in the last few days, but I failed. I hope someone can save me from this terrible situation. Thanks!

Here is my basic info of my PC BIOS:
https://photos.app.goo.gl/pK6FbuUjgx69Uvhj8

CPU: i3 8100
RAM: 16GB
MOTHERBOARD: MSI H310M FIRE
GPU: Sapphire-RX470

Here are the /etc/default/grub settings:
Code:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on i915.enable_gvt=1 iommu=pt pcie_acs_override=downstream,multifunction video=efifb:off video=vesa:off vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb textonly nomodeset video=efifb:off"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream initcall_blacklist=sysfb_init"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb textonly nomodeset video=efifb:off video=vesafb:off video=simplefb:off vfio-pci.ids=1002:67df,1002:aaf0"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on nomodeset video=vesafb:off video=efifb:off video=simplefb:off"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt textonly video=astdrmfb pcie_acs_override=downstream,multifunction video=efifb:off video=vesafb:off vfio-pci.ids=1002:67df,1002:aaf0 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream multifunction nofb nomodeset video=vesafb:off efifb:off"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on rootdelay=10"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

/etc/modprobe.d/blacklist.conf:
Code:
blacklist radeon
blacklist nouveau
blacklist nvidia

/etc/modprobe.d/kvm.conf:
Code:
options kvm ignore_msrs=Y

/etc/modprobe.d/kvm-intel.conf:
Code:
options kvm-intel nested=Y

/etc/modprobe.d/vfio.conf:
Code:
options vfio-pci ids=1002:67df,1002:aaf0 disable_vga=1

/etc/modules:
Code:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# kvmgt
vendor-reset

Here are the /etc/pve/qemu-server/100.conf settings:
Code:
agent: 1
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc -global nec-usb-xhci.msi=off -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
balloon: 0
bios: ovmf
boot: order=virtio0;net0
cores: 4
cpu: Penryn
efidisk0: mac:vm-100-disk-0,efitype=4m,size=4M
hostpci0: 0000:01:00.0;0000:01:00.1,pcie=1,x-vga=1
machine: q35
memory: 8192
meta: creation-qemu=6.2.0,ctime=1658682475
name: macos
net0: virtio=22:06:52:E5:2D:41,bridge=vmbr0,firewall=1
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=d0c2e9f7-2d21-45b9-b38c-ef6c908cf759
sockets: 1
usb0: host=1-8.1,usb3=1
usb1: host=1-9,usb3=1
vga: none
virtio0: mac:vm-100-disk-1,cache=unsafe,discard=on,size=100G
vmgenid: e12d225e-2dc7-4216-9b1a-21647edeec00

dmesg | grep -e DMAR -e IOMMU
Code:
root@xx:/etc/modprobe.d# dmesg | grep -e DMAR -e IOMMU
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.008108] ACPI: DMAR 0x000000008EAADFA8 000070 (v01 INTEL  EDK2     00000002      01000013)
[    0.008131] ACPI: Reserving DMAR table memory at [mem 0x8eaadfa8-0x8eaae017]
[    0.028543] DMAR: IOMMU enabled
[    0.086759] DMAR: Host address width 39
[    0.086760] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.086764] DMAR: dmar0: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.086766] DMAR: RMRR base: 0x0000008f1c9000 end: 0x0000008f412fff
[    0.086769] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 0
[    0.086770] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.086771] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.088276] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.395673] DMAR: No ATSR found
[    0.395674] DMAR: No SATC found
[    0.395675] DMAR: dmar0: Using Queued invalidation
[    0.395855] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    4.562521] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

dmesg | grep 'remapping'
Code:
[    0.086771] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.088276] DMAR-IR: Enabled IRQ remapping in x2apic mode

find /sys/kernel/iommu_groups/ -type l
Code:
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/3/devices/0000:00:12.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.5
/sys/kernel/iommu_groups/8/devices/0000:00:1f.3
/sys/kernel/iommu_groups/8/devices/0000:00:1f.4
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/2/devices/0000:00:08.0
/sys/kernel/iommu_groups/10/devices/0000:02:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:01:00.0
/sys/kernel/iommu_groups/9/devices/0000:01:00.1

lspci -nnv
Code:
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev cf) (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited / Sapphire Technology Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [174b:353e]
        Flags: bus master, fast devsel, latency 0, IRQ 11, IOMMU group 9
        Memory at 90000000 (64-bit, prefetchable) [size=256M]
        Memory at a0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at 4000 [size=256]
        Memory at a0400000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [200] Physical Resizable BAR
        Capabilities: [270] Secondary PCI Express
        Capabilities: [2b0] Address Translation Service (ATS)
        Capabilities: [2c0] Page Request Interface (PRI)
        Capabilities: [2d0] Process Address Space ID (PASID)
        Capabilities: [320] Latency Tolerance Reporting
        Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [370] L1 PM Substates
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
        Subsystem: PC Partner Limited / Sapphire Technology Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [174b:aaf0]
        Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 9
        Memory at a0460000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

update-initramfs -u -k all
Code:
update-initramfs: Generating /boot/initrd.img-5.15.39-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

update-grub
Code:
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.39-3-pve
Found initrd image: /boot/initrd.img-5.15.39-3-pve
Found linux image: /boot/vmlinuz-5.15.30-2-pve
Found initrd image: /boot/initrd.img-5.15.30-2-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done

I followed official documents about the gpu-passthrough part and go to this step:

How to know if a Graphics Card is UEFI (OVMF) compatible​

Get and compile the software "rom-parser":

git clone https://github.com/awilliam/rom-parser
cd rom-parser
make

Then dump the rom of you vga card:

cd /sys/bus/pci/devices/0000:01:00.0/
echo 1 > rom
cat rom > /tmp/image.rom
echo 0 > rom

and test it with:

./rom-parser /tmp/image.rom

Output should look like this:

Valid ROM signature found @0h, PCIR offset 190h
PCIR: type 0, vendor: 10de, device: 1280, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found @f400h, PCIR offset 1ch
PCIR: type 3, vendor: 10de, device: 1280, class: 030000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid
Last image

To be UEFI compatible, you need a "type 3" in the result.

Unluckily, I can't get the 'type 3' expected output, instead, I get this:

./rom-parser /tmp/image.rom
Code:
Valid ROM signature found @0h, PCIR offset 258h
        PCIR: type 0 (x86 PC-AT), vendor: 1002, device: 67df, class: 030000
        PCIR: revision 0, vendor revision: f32
Error, ran off the end

Does it mean that my graphic card is not UEFI compatible?

I tried to boot the vm of macOS, nothing was displayed on the screen. The screen was black and turned into energy saving mode. So I shutdown the vm of macOS and tried to pass through the graphic card to another vm of Ubuntu to which I successfully passed through my UHD 630, I met the same situation. There was nothing on the screen, the screen was black and went to energy saving mode after starting the vm.

Here are my questions:
  1. Are my settings correct? (grub/modules/vfio/blacklist...) If not, please kindly show me the correct settings.
  2. Does output of rom-parser above mean my graphic card can't be used for hackintosh environment?
  3. Why my screen turned black when I started my vm?
The full version of dmesg info is in the attchment. If I miss any information please let me know. Thanks!
 

Attachments

try to add to your configuration VM file to string with args
-cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

cpu: host

machine: q35 - here u can use 3.1 or 5.1 version, sometimes it helps
 
try to add to your configuration VM file to string with args
-cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

cpu: host

machine: q35 - here u can use 3.1 or 5.1 version, sometimes it helps
Thank you for your reply. I added these args but with no luck. The screen turned black after booting the vm. It seems like it's not the problem of the vm's configuration. I think maybe the grub configuration matters, but I don't know what exactly the problem is. Maybe hackintosh with gpu passing- through is too complex for me... sad.
 
Thank you for your reply. I added these args but with no luck. The screen turned black after booting the vm. It seems like it's not the problem of the vm's configuration. I think maybe the grub configuration matters, but I don't know what exactly the problem is. Maybe hackintosh with gpu passing- through is too complex for me... sad.
if u see a logo of ProxMox at your display, after VM start - this means that all ok and the problems can be only with confugurations VM or/and Proxmox. IMHO, try to install windows to guest. if this would be ok - try do do with the our OS.

options kvm ignore_msrs=Y
options kvm-intel nested=Y

as i remember right:
options kvm ignore_msrs=1
options kvm-intel nested=1

try to add to /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

in my case i use this settings at VM config:
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' … bios: ovmf … cpu: host,hidden=1,flags=+pcid … machine: pc-q35-3.1 … hostpci0: 01:00.0,pcie=1,x-vga=on
 
Is PROXMOX going to ever fix this problem?

Which topic do you exactly mean?

PCI(e)-passthrough which is, at least on the older wiki page [1], still considered experimental; probably because there are so many different individual things that come to play with it? [2]
or
Virtualizing MacOS which most likely (hopefully) nobody in a production/enterprise environment does and therefore only a hobbyist/homelab thing and possibly even against Apple's terms is?

If you want a proper MacOS experience, get a Mac.

[1] https://pve.proxmox.com/wiki/Pci_passthrough#Introduction
[2] https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_general_requirements
 
Neobin, passthrough of course// just seems to be experimental for a long time now...