Solved - Single RX570 GPU passthrough

[William C]

Could I have some help passing through my AMD Radeon RX 570 graphics card to an OSX virtual machine?
I cannot get further than the Proxmox boot screen

Update

I have finally after three weeks resolved my GPU passthrough issues.

I will added details to last message on this thread

 

[noel.]

Hi,

GPU passthrough is notoriously finnicky. There Is a Tutorial on the Forum that got a RX 570 to work, maybe that helps?

 

[William C]

I am getting this error in dmesg

65.101987] vfio-pci 0000:07:00.0: BAR 0: can't reserve [mem 0xffe0000000-0xffefffffff 64bit pref]

Update

This was resolved by using just the line initcall_blacklist=sysfb_init in /etc/default/grub and running update-grub to apply the change

The problem is caused by the Framebuffer reserving memory which prevents the GPU passthrough from working.

You can check for this issue by running cat /proc/iomem and making sure no mention of Framebuffer,simplefb efifb etc

 

[leesteken]

Looks like the same GPU is used to boot and show Proxmox. Try this work-around that should resolved the BAR can't reserve problem.
I don't have time for details now but this works perfect for my RX570 (used for boot): upgrade to pve-kernel-5.19, install vendor-reset, enable device_specific reset_method and don't blacklist amdgpu or bind the GPU to vfio-pci.

 

[William C]

Yes same gpu

I can confirm that adding initcall_blacklist=sysfb_init to grub resolved the BAR issue

 

[William C]

I created a windows vm for testing and managed to get GPU passthrough working however the graphics are low quality - not sure if this is normal.



Update

Low quality graphics is normal and will often get resolved either when you install a driver inside the vm or the vm itself loads the appropriate graphics driver bumping up the resolution

 

[leesteken]

Yes, many options and work-around no longer apply for the new kernel where the hand-off from amdgpu to vfio-pci works well again (and much better than simplefb to vfio-pci) for this type of AMD GPU (and most are for NVidia or motherboard issues). With an modern AMD CPU (I didn't see much information about your system in this thread), you don't really need any kernel parameters.
Resolution is always low until drivers are installed, except Linux with drivers in the kernel. I'm sorry but I can't really help with Windows or MacOS specific issues. Fortunately, the passthrough appears to be working.
If you don't have issues with starting the VMs after stopping it, then vendor-reset must be working fine (you need to enable device_specific every reboot of the host).

 

[William C]

Thank you for your help to get this going

My setup is i have a gigabyte a320m motherboard with ryzen 7-2700 8 core cpu / rx570 8gb graphics.

I have lots of questions i hope you can answer

1. Are there any steps to make the boot up process look prettier …ie less garbage on screen and just nice logos?
Ideally i would want just the gigabyte logo and a blank screen and no proxmox logos. i would of course expect to see the boot logo screen for windows/mac.

I note that one of the grub options seemed to affect both the font and some of the stuff that was coming through to screen. It also broke my boot loadino when i added lots of options, but all good again after just having ‘quiet’. Are there any others you would recommend useful still?

2. I’m a bit unclear when seletting cpu type eg ‘host’ or pentryn ( whatever its called), whether i am trying to marry this to the guest os or the proxmox host?

3. I have found that to get further than just the proxmox logo on screen, i have to set display to off/or set the graphics card as primary. Although even this hasnt fixed the issue with mac loading up. That’s tonights project !

4. do i have any concerns over Ssd wearing with proxmox ? I ran this command which a forum suggested to turn off unnecessary logs . systemctl disable corosync pve-ha-crm pve-ha-lrm. my understanding is that due to the way that Ssds work , by rewriting an entire cluster , even though only a few bytes needs writing, that intensive log writing can wear down the Ssd?

5. Is there anything you’d recommend doing to backup my proxmox server configuration so say i wanted to try and upgrade in future i can do so without the stress of possibly breaking what is working! I suppose i could backup the drive that has the proxmox server installed on with a 3rd party partition/cloning tool but just wondering if anything else recommended.

thats all i can think of off top of my head.

 

[leesteken]

My setup is i have a gigabyte a320m motherboard with ryzen 7-2700 8 core cpu / rx570 8gb graphics.

Note that, except for X570 (and maybe some of the new AM5?), all Ryzen motherboards are limited in what they can passthrough (especially the lowest end A320).

1. Are there any steps to make the boot up process look prettier …ie less garbage on screen and just nice logos?
Ideally i would want just the gigabyte logo and a blank screen and no proxmox logos. i would of course expect to see the boot logo screen for windows/mac.

I note that one of the grub options seemed to affect both the font and some of the stuff that was coming through to screen. It also broke my boot loadino when i added lots of options, but all good again after just having ‘quiet’. Are there any others you would recommend useful still?

I don't know what you consider garbage. I was very happy to get to see boot messages again with this kernel version in case something goes wrong. Given the target audience of enterprise servers, I don't think anyone spends time on this. I think trying to hide the Proxmox logo's is disrespectful of such a nice and free piece of software.

2. I’m a bit unclear when seletting cpu type eg ‘host’ or pentryn ( whatever its called), whether i am trying to marry this to the guest os or the proxmox host?

I'm not sure what you mean. Select host for best performance usually. Select something else when you need to migrate VMs between systems or when the operating system inside the VM needs that.

3. I have found that to get further than just the proxmox logo on screen, i have to set display to off/or set the graphics card as primary. Although even this hasnt fixed the issue with mac loading up. That’s tonights project !

I don't know what you mean and I can't help with MacOS.

4. do i have any concerns over Ssd wearing with proxmox ? I ran this command which a forum suggested to turn off unnecessary logs . systemctl disable corosync pve-ha-crm pve-ha-lrm. my understanding is that due to the way that Ssds work , by rewriting an entire cluster , even though only a few bytes needs writing, that intensive log writing can wear down the Ssd?

Please search this forum and you'll find that enterprise SSDs are recommanded for Proxmox VE (which is an enterprise server oriented hypervisor).

5. Is there anything you’d recommend doing to backup my proxmox server configuration so say i wanted to try and upgrade in future i can do so without the stress of possibly breaking what is working! I suppose i could backup the drive that has the proxmox server installed on with a 3rd party partition/cloning tool but just wondering if anything else recommended.

Please search this forum. Personally, I really enjoy Proxmox Backup Server for VMs and use its client to backup /etc on my Proxmox hosts.

thats all i can think of off top of my head.

Feel free to search the forum if you think of anything else.

 

[William C]

I just wanted to post an update for the benefit of anyone else trying to get macOS running on a similar setup and getting nowhere

It can work on the RX570 but in my experience this was not possible at this time with Proxmox.

Its absolutely critical to make sure the frame buffer is not reserving memory which you can check by running "cat /proc/iomem" without parenthesis.
Look for your GPU and make sure you don't see anything about SimpleFB or EFIFB or something similar. If you do you need to deal with that before GPU passthrough is going to work. In my case adding "video=efifbff" to grub sorted but there other commands that can achieve the same result.



Something else I found essential was using dracut to force the binding of vfio_pci early before amdgpu had a chance to get it!



Follow steps also for installing vendor-reset, apart from anything else its reassuring to see the nice messages you get in dmesg showing everything is ticketyboo

I managed to get Windows 10 loading fine but macOS refusing to get past the apple logo when GPU passthrough enabled.

I decided to try my luck on Debian directly with Qemu and had a lot more success. I then decided to give Proxmox another shot, seeing that I had got it working on Debian so in theory if I carefully mirrored my settings over to Proxmox it should work right? Still no joy though

I have now gone back to using Debian and QEMU.

Working so far :

GPU passthrough

• Network bridge to host ethernet adapter
• Keyboard and mouse (once I figured out how to add these in config)
• Automatic Qemu VM launch using Cronos with @reboot job

Work in progress

• Passing through other USB devices like webcam/digital piano
• Passing through Sound card
• Network bridge so my vm gets an IP directly from router rather than setting up a private 10.x.x.x. network between itself and host
• Correct CPU configuration so utilizing all cores

 

[William C]

I decided to try proxmox again this evening /morning now that I have a deeper understanding of Linux from many hours of trial and error.

I have managed amazingly for first time to get macOS installer booting

I wanted to document here to save anyone else the grief

I added initcall_blacklist=sysfb_init to grub which prevents framebuffer from making memory reservation
I have not got any other options or vendor reset installed

i also added the pci without selecting ‘all functions’ or pcie. I have always previously selected one or both of these

this is also mentioned in https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/page-2

By the way, I also found better solution on a reddit post. It is adding "initcall_blacklist=sysfb_init" to kernel parameter. No need "video=efifbff" or "video=simplefbff" in kernel parameter. I also tested, it does solve the problem!

Reference:
https://www.reddit.com/r/VFIO/comme...let_simplefb_stay_away_from_the_gpu/?sort=old
https://www.reddit.com/r/Proxmox/comments/vc9hw3/latest_proxmox_7_the_kernel_breaks_my_gpu/?sort=old

 

[William C]

Ok so final update on this thread

I managed to get GPU passthrough working for Graphics by unticking PCI-E however audio not working



Two weeks later and after many hours of trying everything I could possibly think to try I can reveal that what got it working was setting the q35 to version 6.0 or lower.

When I did so I found that the VM boots AND HDMI audio finally freaking working!!!

I came to this solution by realizing the pci-e was working with my windows vm so I thought what's causing it not to work on the OSX vm. I went through and tried many of the cpu hosts Penny etc but still not loading. I then tried an older q35 - what have I got to loose and sat back on sofa and to my surprise saw it had booted!

I should not that the OSX progress bar does not move when loading VM but it does after about 5-10 seconds when using older q35.

I had come across this q35 issue mentioned on a forum before but it didn't seem to make any difference but I think this was because I was assuming the frozen progress bar meant it wasn't going to get further so I just reset or stopped the vm at this point. I also assumed that PCIE just wasn't going to work for my graphics card.

It could well be that this works with just the older q35 but haven't tested yet.

I will spin up another copy of my vm and see just what exactly did or did not need to be changed to get this working



I tried turning PCI-e off to see if it still worked and nope it doesn't. You need PCI-E ticked and Q35 v6.0 or earlier. I haven't tested every single version prior to 6.0 but the ones I tested worked. Somethings broken it after version 6.0

Also worth noting too that this worked both with my model set as macpro7,1 and imacpro1,1. I don't think the version much mattered in respect of the audio not working. it was just because of the q35 version and PCI-e not ticked. The only kexts I have are whatever green and lilu.

https://www.reddit.com/r/VFIO/comments/q8juo3/macos_vm_qemu610_problems/

This solution is also mentioned at end of this thread fyi

https://forum.proxmox.com/threads/a...-work-anymore-since-upgrade-to-7-2-11.117377/

 

[biskero]

Hello,

I have problem with Audio and GPU.
Thanks to aldariz that helped me I got GPU passthrough working in the sense that macos Ventura knows there is a GPU (RX580, not showing 8Gb!) but is not using it even though I specify to use it in GRUD and VM Hardware. Audio is not working as Ventura does not see an Audio device, but it sees an Audio device in PCI.

In Hardware for VM, PCI Device I had to ROM-Bar unchecked else would not be able to boot via VMWare.
If I tried to set Display to none cannot see anything on the Display (34") since macos is using the onboard vga.
Btw I cannot test it since my motherboard does not have a physical video output since it's a workstation/server Asus Z10PE D8 WS.
The system has Dual Xeon E5 20 cores with 128 Gb RAM.

I tried the q35 6.0 but did nothing, any suggestions on what I can try?
Thx

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio-pci.ids=1002:67df,1002:aaf0 initcall_blacklist=sysfb_init"

Here are the 2 graphics video:

lspci -nnv | grep VGA
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7) (prog-if 00 [VGA controller])
0a:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30) (prog-if 00 [VGA controller])

# dmesg | grep -e DMAR -e IOMMU
[    0.013418] ACPI: DMAR 0x000000007AED1F60 0000F8 (v01 ALASKA A M I    00000001 INTL 20091013)
[    0.013460] ACPI: Reserving DMAR table memory at [mem 0x7aed1f60-0x7aed2057]
[    0.329002] DMAR: IOMMU enabled
[    0.811818] DMAR: Host address width 46
[    0.811820] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.811826] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    0.811829] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    0.811833] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    0.811835] DMAR: RMRR base: 0x0000007db86000 end: 0x0000007db95fff
[    0.811837] DMAR: ATSR flags: 0x0
[    0.811839] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
[    0.811840] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
[    0.811843] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
[    0.811845] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
[    0.811846] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
[    0.811848] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    0.811849] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.811850] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.812567] DMAR-IR: Enabled IRQ remapping in xapic mode
[    1.428387] DMAR: [Firmware Bug]: RMRR entry for device 02:00.0 is broken - applying workaround
[    1.428395] DMAR: No SATC found
[    1.428399] DMAR: dmar1: Using Queued invalidation
[    1.440757] DMAR: Intel(R) Virtualization Technology for Directed I/O

And VM configuration file 100.conf:

#February 9 2023 kernel 5.15
agent: 1
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -global nec-usb-xhci.msi=off -globa
l ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off -cpu host,vendor=GenuineIntel,+invtsc,+hypervisor,kvm=on,vmware-cpuid-freq=on
balloon: 0
bios: ovmf
boot: order=virtio0
cores: 8
cpu: Haswell
cpuunits: 1000
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,size=4M
hostpci0: 0000:03:00,pcie=1,rombar=0,x-vga=1
machine: q35
memory: 122880
meta: creation-qemu=7.1.0,ctime=1675598005
name: macos-ventura
net0: virtio=62:01:EB:AF:56:76,bridge=vmbr0,firewall=1
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=xxxxxxxxxxxxxxxxxxxxxxxx
snaptime: 1675971443
sockets: 2
usb0: host=1c4f:0063,usb3=1
usb1: host=046d:c062,usb3=1
usb2: host=0a12:0001,usb3=1
usb3: host=046d:c52b,usb3=1
vga: vmware
virtio0: local-lvm:vm-100-disk-1,cache=unsafe,discard=on,iothread=1,size=850G
vmgenid: xxxxxxxxxxxxxxxxxxxxxxxx

 

[William C]

I might be barking up wrong tree but i would personally be setting vga to none.

You’d only be setting to VMware compatible if you are wanting to see the host remotely inside the web gui . I presume you are trying to pass through the gpu and for the output to be the screen in front of you.

id have thought that by setting it as other than none, OSX is using this as the primary display

how many GPUs do you have ? Do you have IGPU as well? Have you set the appropriate primary graphics card in bios ?

i presume proxmox is installed as the main os? You could try spinning up a windows or Linux vm with same settings and establish it can see the gpu because if it can’t then could help narrow down the problem

i flitted between using proxmox and linux/qemu before I got proxmox fully working as I wanted.

I use proxmox with a similar gpu without issue now.

just as an update to my earlier posts . The solution to the rx570 audio not working in my last post was that you either have to use an older version of q35 v6.0 or have that line ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off in the args

 

[biskero]

Thanks for the answer.
So if I set the Display to none in VM Hardware no video on the monitor is showing.

I have only one GPU and the on board mb video graphics, which is not accessible via hdmi since there is none.
In the BIOS is set to use the off board video card (GPU).

Proxmox is the primary OS.
As per Audio I do have those settings ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off as I posted above 100.conf:.

I need to do more testing I guess!

 

[toomanylogins]

I passthrough my 2 nvidia gt730 cards to osx without any issues. I think the problem is the bios should be set to use the onboard graphics otherwise proxmox host will use the off board card. Connect hdmi to onbaord and make sure you get the proxmox host. Then sort the pass through. I use a kvm to switch between all three.

 

[biskero]

I do not have hdmi on the motherboard is a workstation/server type, no physical video output.
BTW I can get into the macos Ventura VM via VMWare display and the macos see the 580 (I posted) but is not using it even if I set as primary GPU in the Hardware config. If I set Display to none, I do not have any signal to the monitor.

 

[biskero]

Just noticed that my mother board has VGA jumper




I need to test this since it would disable on board VGA even if it's selected via Bios.

 

[biskero]

I tried but did not helped.
Looks like the vga is still used instead of GPU even though everything is set to use GPU for video!

 

[flognort]

I know this thread is old, but to anyone coming here disabling above 4g decoding fixed it for me