GPU can't passthrough to vm successfully with RX470

A

absolutracy

New Member
Aug 17, 2022
3
0
1
Hello everyone, recently I read a post written by Nicholas Sherlock. The post showed a way to install macOS on proxmox. I followed the instructions and was able to install the macOS Monterey on proxmox successfully. When I came to the gpu passing-through part, I tried a lot of methods posted on the Internet and spent hours and hours in the last few days, but I failed. I hope someone can save me from this terrible situation. Thanks!

Here is my basic info of my PC BIOS:
https://photos.app.goo.gl/pK6FbuUjgx69Uvhj8

CPU: i3 8100
RAM: 16GB
MOTHERBOARD: MSI H310M FIRE
GPU: Sapphire-RX470

Here are the /etc/default/grub settings:
Code: 
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on i915.enable_gvt=1 iommu=pt pcie_acs_override=downstream,multifunction video=efifb:off video=vesa:off vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb textonly nomodeset video=efifb:off"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream initcall_blacklist=sysfb_init"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb textonly nomodeset video=efifb:off video=vesafb:off video=simplefb:off vfio-pci.ids=1002:67df,1002:aaf0"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on nomodeset video=vesafb:off video=efifb:off video=simplefb:off"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt textonly video=astdrmfb pcie_acs_override=downstream,multifunction video=efifb:off video=vesafb:off vfio-pci.ids=1002:67df,1002:aaf0 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream multifunction nofb nomodeset video=vesafb:off efifb:off"
# GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on rootdelay=10"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

/etc/modprobe.d/blacklist.conf:
Code: 
blacklist radeon
blacklist nouveau
blacklist nvidia

/etc/modprobe.d/kvm.conf:
Code: 
options kvm ignore_msrs=Y

/etc/modprobe.d/kvm-intel.conf:
Code: 
options kvm-intel nested=Y

/etc/modprobe.d/vfio.conf:
Code: 
options vfio-pci ids=1002:67df,1002:aaf0 disable_vga=1

/etc/modules:
Code: 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# kvmgt
vendor-reset

Here are the /etc/pve/qemu-server/100.conf settings:
Code: 
agent: 1
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc -global nec-usb-xhci.msi=off -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
balloon: 0
bios: ovmf
boot: order=virtio0;net0
cores: 4
cpu: Penryn
efidisk0: mac:vm-100-disk-0,efitype=4m,size=4M
hostpci0: 0000:01:00.0;0000:01:00.1,pcie=1,x-vga=1
machine: q35
memory: 8192
meta: creation-qemu=6.2.0,ctime=1658682475
name: macos
net0: virtio=22:06:52:E5:2D:41,bridge=vmbr0,firewall=1
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=d0c2e9f7-2d21-45b9-b38c-ef6c908cf759
sockets: 1
usb0: host=1-8.1,usb3=1
usb1: host=1-9,usb3=1
vga: none
virtio0: mac:vm-100-disk-1,cache=unsafe,discard=on,size=100G
vmgenid: e12d225e-2dc7-4216-9b1a-21647edeec00

dmesg | grep -e DMAR -e IOMMU
Code: 
root@xx:/etc/modprobe.d# dmesg | grep -e DMAR -e IOMMU
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.008108] ACPI: DMAR 0x000000008EAADFA8 000070 (v01 INTEL  EDK2     00000002      01000013)
[    0.008131] ACPI: Reserving DMAR table memory at [mem 0x8eaadfa8-0x8eaae017]
[    0.028543] DMAR: IOMMU enabled
[    0.086759] DMAR: Host address width 39
[    0.086760] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.086764] DMAR: dmar0: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.086766] DMAR: RMRR base: 0x0000008f1c9000 end: 0x0000008f412fff
[    0.086769] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 0
[    0.086770] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.086771] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.088276] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.395673] DMAR: No ATSR found
[    0.395674] DMAR: No SATC found
[    0.395675] DMAR: dmar0: Using Queued invalidation
[    0.395855] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    4.562521] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

dmesg | grep 'remapping'
Code: 
[    0.086771] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.088276] DMAR-IR: Enabled IRQ remapping in x2apic mode

find /sys/kernel/iommu_groups/ -type l
Code: 
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/3/devices/0000:00:12.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.5
/sys/kernel/iommu_groups/8/devices/0000:00:1f.3
/sys/kernel/iommu_groups/8/devices/0000:00:1f.4
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/2/devices/0000:00:08.0
/sys/kernel/iommu_groups/10/devices/0000:02:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:01:00.0
/sys/kernel/iommu_groups/9/devices/0000:01:00.1

lspci -nnv
Code: 
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev cf) (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited / Sapphire Technology Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [174b:353e]
        Flags: bus master, fast devsel, latency 0, IRQ 11, IOMMU group 9
        Memory at 90000000 (64-bit, prefetchable) [size=256M]
        Memory at a0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at 4000 [size=256]
        Memory at a0400000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [200] Physical Resizable BAR
        Capabilities: [270] Secondary PCI Express
        Capabilities: [2b0] Address Translation Service (ATS)
        Capabilities: [2c0] Page Request Interface (PRI)
        Capabilities: [2d0] Process Address Space ID (PASID)
        Capabilities: [320] Latency Tolerance Reporting
        Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [370] L1 PM Substates
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
        Subsystem: PC Partner Limited / Sapphire Technology Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [174b:aaf0]
        Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 9
        Memory at a0460000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

update-initramfs -u -k all
Code: 
update-initramfs: Generating /boot/initrd.img-5.15.39-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

update-grub
Code: 
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.39-3-pve
Found initrd image: /boot/initrd.img-5.15.39-3-pve
Found linux image: /boot/vmlinuz-5.15.30-2-pve
Found initrd image: /boot/initrd.img-5.15.30-2-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done

I followed official documents about the gpu-passthrough part and go to this step:

How to know if a Graphics Card is UEFI (OVMF) compatible​

Get and compile the software "rom-parser":

git clone https://github.com/awilliam/rom-parser
cd rom-parser
make

Then dump the rom of you vga card:

cd /sys/bus/pci/devices/0000:01:00.0/
echo 1 > rom
cat rom > /tmp/image.rom
echo 0 > rom

and test it with:

./rom-parser /tmp/image.rom

Output should look like this:

Valid ROM signature found @0h, PCIR offset 190h
PCIR: type 0, vendor: 10de, device: 1280, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found @f400h, PCIR offset 1ch
PCIR: type 3, vendor: 10de, device: 1280, class: 030000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid
Last image

To be UEFI compatible, you need a "type 3" in the result.
Click to expand...

Unluckily, I can't get the 'type 3' expected output, instead, I get this:

./rom-parser /tmp/image.rom
Code: 
Valid ROM signature found @0h, PCIR offset 258h
        PCIR: type 0 (x86 PC-AT), vendor: 1002, device: 67df, class: 030000
        PCIR: revision 0, vendor revision: f32
Error, ran off the end

Does it mean that my graphic card is not UEFI compatible?

I tried to boot the vm of macOS, nothing was displayed on the screen. The screen was black and turned into energy saving mode. So I shutdown the vm of macOS and tried to pass through the graphic card to another vm of Ubuntu to which I successfully passed through my UHD 630, I met the same situation. There was nothing on the screen, the screen was black and went to energy saving mode after starting the vm.

Here are my questions:
  1. Are my settings correct? (grub/modules/vfio/blacklist...) If not, please kindly show me the correct settings.
  2. Does output of rom-parser above mean my graphic card can't be used for hackintosh environment?
  3. Why my screen turned black when I started my vm?
The full version of dmesg info is in the attchment. If I miss any information please let me know. Thanks!
 

Attachments

try to add to your configuration VM file to string with args
-cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

cpu: host

machine: q35 - here u can use 3.1 or 5.1 version, sometimes it helps
 
LuchS said:
try to add to your configuration VM file to string with args
-cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

cpu: host

machine: q35 - here u can use 3.1 or 5.1 version, sometimes it helps
Click to expand...
Thank you for your reply. I added these args but with no luck. The screen turned black after booting the vm. It seems like it's not the problem of the vm's configuration. I think maybe the grub configuration matters, but I don't know what exactly the problem is. Maybe hackintosh with gpu passing- through is too complex for me... sad.
 
absolutracy said:
Thank you for your reply. I added these args but with no luck. The screen turned black after booting the vm. It seems like it's not the problem of the vm's configuration. I think maybe the grub configuration matters, but I don't know what exactly the problem is. Maybe hackintosh with gpu passing- through is too complex for me... sad.
Click to expand...
if u see a logo of ProxMox at your display, after VM start - this means that all ok and the problems can be only with confugurations VM or/and Proxmox. IMHO, try to install windows to guest. if this would be ok - try do do with the our OS.

options kvm ignore_msrs=Y
options kvm-intel nested=Y
Click to expand...

as i remember right:
options kvm ignore_msrs=1
options kvm-intel nested=1

try to add to /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

in my case i use this settings at VM config:
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' … bios: ovmf … cpu: host,hidden=1,flags=+pcid … machine: pc-q35-3.1 … hostpci0: 01:00.0,pcie=1,x-vga=on
 
KenF said:
Is PROXMOX going to ever fix this problem?
Click to expand...

Which topic do you exactly mean?

PCI(e)-passthrough which is, at least on the older wiki page [1], still considered experimental; probably because there are so many different individual things that come to play with it? [2]
or
Virtualizing MacOS which most likely (hopefully) nobody in a production/enterprise environment does and therefore only a hobbyist/homelab thing and possibly even against Apple's terms is?

If you want a proper MacOS experience, get a Mac.

[1] https://pve.proxmox.com/wiki/Pci_passthrough#Introduction
[2] https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_general_requirements
 
Neobin, passthrough of course// just seems to be experimental for a long time now...
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom