Solved - Single RX570 GPU passthrough

[William C]

Could I have some help passing through my AMD Radeon RX 570 graphics card to an OSX virtual machine?
I cannot get further than the Proxmox boot screen

Update

I have finally after three weeks resolved my GPU passthrough issues.

I will added details to last message on this thread

 

[noel.]

Hi,

GPU passthrough is notoriously finnicky. There Is a Tutorial on the Forum that got a RX 570 to work, maybe that helps?

 

[William C]

I am getting this error in dmesg

65.101987] vfio-pci 0000:07:00.0: BAR 0: can't reserve [mem 0xffe0000000-0xffefffffff 64bit pref]

Update

This was resolved by using just the line initcall_blacklist=sysfb_init in /etc/default/grub and running update-grub to apply the change

The problem is caused by the Framebuffer reserving memory which prevents the GPU passthrough from working.

You can check for this issue by running cat /proc/iomem and making sure no mention of Framebuffer,simplefb efifb etc

 

[leesteken]

Looks like the same GPU is used to boot and show Proxmox. Try this work-around that should resolved the BAR can't reserve problem.
I don't have time for details now but this works perfect for my RX570 (used for boot): upgrade to pve-kernel-5.19, install vendor-reset, enable device_specific reset_method and don't blacklist amdgpu or bind the GPU to vfio-pci.

 

[William C]

Yes same gpu

I can confirm that adding initcall_blacklist=sysfb_init to grub resolved the BAR issue

 

[William C]

I created a windows vm for testing and managed to get GPU passthrough working however the graphics are low quality - not sure if this is normal.



Update

Low quality graphics is normal and will often get resolved either when you install a driver inside the vm or the vm itself loads the appropriate graphics driver bumping up the resolution

 

[leesteken]

Yes, many options and work-around no longer apply for the new kernel where the hand-off from amdgpu to vfio-pci works well again (and much better than simplefb to vfio-pci) for this type of AMD GPU (and most are for NVidia or motherboard issues). With an modern AMD CPU (I didn't see much information about your system in this thread), you don't really need any kernel parameters.
Resolution is always low until drivers are installed, except Linux with drivers in the kernel. I'm sorry but I can't really help with Windows or MacOS specific issues. Fortunately, the passthrough appears to be working.
If you don't have issues with starting the VMs after stopping it, then vendor-reset must be working fine (you need to enable device_specific every reboot of the host).

 

[William C]

Thank you for your help to get this going

My setup is i have a gigabyte a320m motherboard with ryzen 7-2700 8 core cpu / rx570 8gb graphics.

I have lots of questions i hope you can answer

1. Are there any steps to make the boot up process look prettier …ie less garbage on screen and just nice logos?
Ideally i would want just the gigabyte logo and a blank screen and no proxmox logos. i would of course expect to see the boot logo screen for windows/mac.

I note that one of the grub options seemed to affect both the font and some of the stuff that was coming through to screen. It also broke my boot loadino when i added lots of options, but all good again after just having ‘quiet’. Are there any others you would recommend useful still?

2. I’m a bit unclear when seletting cpu type eg ‘host’ or pentryn ( whatever its called), whether i am trying to marry this to the guest os or the proxmox host?

3. I have found that to get further than just the proxmox logo on screen, i have to set display to off/or set the graphics card as primary. Although even this hasnt fixed the issue with mac loading up. That’s tonights project !

4. do i have any concerns over Ssd wearing with proxmox ? I ran this command which a forum suggested to turn off unnecessary logs . systemctl disable corosync pve-ha-crm pve-ha-lrm. my understanding is that due to the way that Ssds work , by rewriting an entire cluster , even though only a few bytes needs writing, that intensive log writing can wear down the Ssd?

5. Is there anything you’d recommend doing to backup my proxmox server configuration so say i wanted to try and upgrade in future i can do so without the stress of possibly breaking what is working! I suppose i could backup the drive that has the proxmox server installed on with a 3rd party partition/cloning tool but just wondering if anything else recommended.

thats all i can think of off top of my head.

 

[leesteken]

My setup is i have a gigabyte a320m motherboard with ryzen 7-2700 8 core cpu / rx570 8gb graphics.

Note that, except for X570 (and maybe some of the new AM5?), all Ryzen motherboards are limited in what they can passthrough (especially the lowest end A320).

1. Are there any steps to make the boot up process look prettier …ie less garbage on screen and just nice logos?
Ideally i would want just the gigabyte logo and a blank screen and no proxmox logos. i would of course expect to see the boot logo screen for windows/mac.

I note that one of the grub options seemed to affect both the font and some of the stuff that was coming through to screen. It also broke my boot loadino when i added lots of options, but all good again after just having ‘quiet’. Are there any others you would recommend useful still?

I don't know what you consider garbage. I was very happy to get to see boot messages again with this kernel version in case something goes wrong. Given the target audience of enterprise servers, I don't think anyone spends time on this. I think trying to hide the Proxmox logo's is disrespectful of such a nice and free piece of software.

2. I’m a bit unclear when seletting cpu type eg ‘host’ or pentryn ( whatever its called), whether i am trying to marry this to the guest os or the proxmox host?

I'm not sure what you mean. Select host for best performance usually. Select something else when you need to migrate VMs between systems or when the operating system inside the VM needs that.

3. I have found that to get further than just the proxmox logo on screen, i have to set display to off/or set the graphics card as primary. Although even this hasnt fixed the issue with mac loading up. That’s tonights project !

I don't know what you mean and I can't help with MacOS.

4. do i have any concerns over Ssd wearing with proxmox ? I ran this command which a forum suggested to turn off unnecessary logs . systemctl disable corosync pve-ha-crm pve-ha-lrm. my understanding is that due to the way that Ssds work , by rewriting an entire cluster , even though only a few bytes needs writing, that intensive log writing can wear down the Ssd?

Please search this forum and you'll find that enterprise SSDs are recommanded for Proxmox VE (which is an enterprise server oriented hypervisor).

5. Is there anything you’d recommend doing to backup my proxmox server configuration so say i wanted to try and upgrade in future i can do so without the stress of possibly breaking what is working! I suppose i could backup the drive that has the proxmox server installed on with a 3rd party partition/cloning tool but just wondering if anything else recommended.

Please search this forum. Personally, I really enjoy Proxmox Backup Server for VMs and use its client to backup /etc on my Proxmox hosts.

thats all i can think of off top of my head.

Feel free to search the forum if you think of anything else.

 

[William C]

I just wanted to post an update for the benefit of anyone else trying to get macOS running on a similar setup and getting nowhere

It can work on the RX570 but in my experience this was not possible at this time with Proxmox.

Its absolutely critical to make sure the frame buffer is not reserving memory which you can check by running "cat /proc/iomem" without parenthesis.
Look for your GPU and make sure you don't see anything about SimpleFB or EFIFB or something similar. If you do you need to deal with that before GPU passthrough is going to work. In my case adding "video=efifbff" to grub sorted but there other commands that can achieve the same result.



Something else I found essential was using dracut to force the binding of vfio_pci early before amdgpu had a chance to get it!



Follow steps also for installing vendor-reset, apart from anything else its reassuring to see the nice messages you get in dmesg showing everything is ticketyboo

I managed to get Windows 10 loading fine but macOS refusing to get past the apple logo when GPU passthrough enabled.

I decided to try my luck on Debian directly with Qemu and had a lot more success. I then decided to give Proxmox another shot, seeing that I had got it working on Debian so in theory if I carefully mirrored my settings over to Proxmox it should work right? Still no joy though

I have now gone back to using Debian and QEMU.

Working so far :

GPU passthrough

• Network bridge to host ethernet adapter
• Keyboard and mouse (once I figured out how to add these in config)
• Automatic Qemu VM launch using Cronos with @reboot job

Work in progress

• Passing through other USB devices like webcam/digital piano
• Passing through Sound card
• Network bridge so my vm gets an IP directly from router rather than setting up a private 10.x.x.x. network between itself and host
• Correct CPU configuration so utilizing all cores

 

[William C]

I decided to try proxmox again this evening /morning now that I have a deeper understanding of Linux from many hours of trial and error.

I have managed amazingly for first time to get macOS installer booting

I wanted to document here to save anyone else the grief

I added initcall_blacklist=sysfb_init to grub which prevents framebuffer from making memory reservation
I have not got any other options or vendor reset installed

i also added the pci without selecting ‘all functions’ or pcie. I have always previously selected one or both of these

this is also mentioned in https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/page-2

By the way, I also found better solution on a reddit post. It is adding "initcall_blacklist=sysfb_init" to kernel parameter. No need "video=efifbff" or "video=simplefbff" in kernel parameter. I also tested, it does solve the problem!

Reference:
https://www.reddit.com/r/VFIO/comme...let_simplefb_stay_away_from_the_gpu/?sort=old
https://www.reddit.com/r/Proxmox/comments/vc9hw3/latest_proxmox_7_the_kernel_breaks_my_gpu/?sort=old

 

[William C]

Ok so final update on this thread

I managed to get GPU passthrough working for Graphics by unticking PCI-E however audio not working



Two weeks later and after many hours of trying everything I could possibly think to try I can reveal that what got it working was setting the q35 to version 6.0 or lower.

When I did so I found that the VM boots AND HDMI audio finally freaking working!!!

I came to this solution by realizing the pci-e was working with my windows vm so I thought what's causing it not to work on the OSX vm. I went through and tried many of the cpu hosts Penny etc but still not loading. I then tried an older q35 - what have I got to loose and sat back on sofa and to my surprise saw it had booted!

I should not that the OSX progress bar does not move when loading VM but it does after about 5-10 seconds when using older q35.

I had come across this q35 issue mentioned on a forum before but it didn't seem to make any difference but I think this was because I was assuming the frozen progress bar meant it wasn't going to get further so I just reset or stopped the vm at this point. I also assumed that PCIE just wasn't going to work for my graphics card.

It could well be that this works with just the older q35 but haven't tested yet.

I will spin up another copy of my vm and see just what exactly did or did not need to be changed to get this working



I tried turning PCI-e off to see if it still worked and nope it doesn't. You need PCI-E ticked and Q35 v6.0 or earlier. I haven't tested every single version prior to 6.0 but the ones I tested worked. Somethings broken it after version 6.0

Also worth noting too that this worked both with my model set as macpro7,1 and imacpro1,1. I don't think the version much mattered in respect of the audio not working. it was just because of the q35 version and PCI-e not ticked. The only kexts I have are whatever green and lilu.

https://www.reddit.com/r/VFIO/comments/q8juo3/macos_vm_qemu610_problems/

This solution is also mentioned at end of this thread fyi

https://forum.proxmox.com/threads/a...-work-anymore-since-upgrade-to-7-2-11.117377/