Support for changing device apertures?

[blackangus]

I've been trying to get resizable bar (SAM for me) enabled in a Windows guest using a passed-through 6950 XT. While this is mostly working (I think) there are some snags I'm running into. See https://github.com/qemu/qemu/commit/3412d8ec9810b819f8b79e8e0c6b87217c876e32#commitcomment-112438130

TLDR:

1. I'm not able to get BAR 2 any higher than 8MB without running into code 43
2. I don't really know if SAM is being enabled/utilized in any way since AMD's Adrenaline software is reporting that SAM is unavailable, but at the same time indicates that 256MB is being used

I noticed you can configure the aperture with the more native way of configuring things as reported here https://forums.unraid.net/topic/133...-in-61-kernel/?do=findComment&comment=1221718 I've seen that it is very important that the domain is set right (<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>), otherwise it won't work. I'm not seeing how this maps to Proxmox, or if it's even possible to configure this. Are there any Proxmox devs lurking that would know how to accomplish this?

 

[cheiss]

Hi,

1. I guess the domain type setting is already a given and thus irrelevant. According to the documentation, this specifies the hypervisor - and Proxmox is using KVM anyway (as long as "KVM hardware virtualization" is set in the options).
2. The config can be set using "args", either by editing the VM config file manually or using the qm tool:

qm set <vmid> -args '-fw_cfg opt/ovmf/X-PciMmio64Mb,string=65536'

No gurantees that any of this works in the end of course, as anything involving PCI-passthrough as very hardware dependent in general. But that is the gist of how the linked instructions from the Unraid forums would translate to Proxmox VE.

 

[blackangus]

Thanks for the info. I've tried the args approach you mentioned a few days ago but it didn't have any effect. My educated guess was that the xmlns was not taking effect since it was not set anywhere. Sorry, I should have been more explicit in my post and said the xmlns attribute is the key part of the domain element.

Is there a way to specify the xmlns somewhere? I can't find the doc right now but I had seen something in qemu where they said the arg you mentioned has no effect unless the correct xmlns is in place.

 

[cheiss]

Is there a way to specify the xmlns somewhere? I can't find the doc right now but I had seen something in qemu where they said the arg you mentioned has no effect unless the correct xmlns is in place.

This is all libvirt-specific, which uses an XML document to describe VM configs. As Proxmox VE does not use libvirt (or any XML for that matter), there's no way to specify it.
Again, is this describes the type of hypervisor to use, it is also redundant on Proxmox VE - as it uses KVM/QEMU anyway, as long has hardware virtualization is enabled correctly.

I suppose you correctly set up all the other things mentioned in the thread? (E.g. BIOS ReBAR, Above 4G Decoding, boot host and VM in UEFI mode)

Are there any (error) messages relating to this in the kernel log? (dmesg -H, both host and guest)
It's also possible that the hardware/firmware simply doesn't play along, especially on consumer hardware anything relating to PCI passthrough is only lightly tested (if at all) and unfortunately often broken in some ways.

 

[blackangus]

I suppose you correctly set up all the other things mentioned in the thread? (E.g. BIOS ReBAR, Above 4G Decoding, boot host and VM in UEFI mode)

Yes that's right. I have things mostly working, the only issue is resizing bar 2 to it's max size. I'm able to successfully boot and use the VM (Windows 11 FWIW) with the gpu passed through when bar 0 is set to its maximum size.

I'll check the kernel log later this week when I get a chance to try the fw_cfg arg again. When I tried it last everything still worked with bar 2 being at 8MB, it was just that the behavior around code 43 error that's given inside the guest when bar 2 is set higher was the same as without using fw_cfg.

 

[blackangus]

Sorry for the delay on this, I've been dealing with an issue for a while that turned out to be related to this topic FWIW anyone seeing this thread or wondering if/how to use resizable bar: watch out using it (at least with AMD cards) for now https://github.com/qemu/qemu/commit/3412d8ec9810b819f8b79e8e0c6b87217c876e32#commitcomment-121804805

Separate question: assuming this https://github.com/qemu/qemu/commit/b5048a4cbfa0362abc720b5198fe9a35441bf5fe gets pulled into a release soon how far out do you think it'd be until it came in to the repos? Do you just pull directly from upstream (Debian) for this stuff or is qemu tailored by the Proxmox devs?