Reduced BAR on 8.1 with Intel Arc A380

[rubylaser]

Hello, I have been able to pass through my Intel Arc GPU to an Ubuntu 22.04 64-bit server virtual machine. The host in an AMD Epyc 7402 on a Supermicro H12SSL-I running Proxmox 8.1. On the PVE host I have IOMMU working. Here are the steps that I've done.

root@hyperion:~# dmesg | grep -e DMAR -e IOMMU
[    0.981461] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported
[    0.986972] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported
[    0.993833] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[    1.001974] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.009509] pci 0000:c0:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.009540] pci 0000:80:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.009562] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.009584] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.010523] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.010536] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    1.010550] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[    1.010562] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).

With remapping working.

root@hyperion:~# dmesg | grep 'remapping'                                                                                                                                                                                        
[    1.009606] AMD-Vi: Interrupt remapping enabled

I have this in /etc/modules and I've updated initramfs and rebooted.

root@hyperion:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I have this in /etc/modprobe.d/vfio.conf. I'm passing through an Adaptec HBA, and the two Arc VGA and Audio devices.

root@hyperion:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=9005:028c,8086:56a5,8086:4f92 disable_vga=1

And, this in /etc/modprobe.d/blacklist.conf (I have rebooted again afterward)

root@hyperion:~# cat /etc/modprobe.d/blacklist.conf
blacklist i915
blacklist aacraid

In my VM, I have passed through all three devices (41:00 is the Adaptec, and 83:00/84:00 are the Arc)


Finally, to the problem. The VM takes minutes to boot. When it finally does, I see part of the dmesg, then it eventually stops at the part about the Resizeable BAR (bolded) message. I can read the rest when viewing dmesg once it's done booting.

[    2.560771] Console: switching to colour dummy device 80x25
[    2.560863] bochs-drm 0000:00:01.0: vgaarb: deactivate vga console
[    2.560972] [drm] Found bochs VGA, ID 0xb0c5.
[    2.560978] [drm] Framebuffer size 16384 kB @ 0x80000000, mmio @ 0x8320b000.
[    2.562147] [drm] Found EDID data blob.
[    2.562311] [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:01.0 on minor 0
[    2.563209] fbcon: bochs-drmdrmfb (fb0) is primary device
[    2.627281] Console: switching to colour frame buffer device 160x50
[    2.646579] bochs-drm 0000:00:01.0: [drm] fb0: bochs-drmdrmfb frame buffer device
[    3.100622] i915 0000:02:00.0: [drm] VT-d active for gfx access
[    3.106060] i915 0000:02:00.0: BAR 0: releasing [mem 0x81000000-0x81ffffff 64bit]
[    3.106119] i915 0000:02:00.0: BAR 2: releasing [mem 0x380800000000-0x38080fffffff 64bit pref]
[    3.106196] i915 0000:02:00.0: [drm] Failed to resize BAR2 to 8192M (-EINVAL)
[    3.107188] i915 0000:02:00.0: BAR 2: assigned [mem 0x380800000000-0x38080fffffff 64bit pref]
[    3.107299] i915 0000:02:00.0: BAR 0: assigned [mem 0x81000000-0x81ffffff 64bit]
[    3.138102] i915 0000:02:00.0: [drm] Local memory IO size: 0x0000000010000000
[    3.138141] i915 0000:02:00.0: [drm] Local memory available: 0x000000017c800000
[    3.138162] i915 0000:02:00.0: [drm] Using a reduced BAR size of 256MiB. Consider enabling 'Resizable BAR' or similar, if available in the BIOS.
[    3.156772] i915 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    3.164443] i915 0000:02:00.0: [drm] Finished loading DMC firmware i915/dg2_dmc_ver2_08.bin (v2.8)
[    3.175225] i915 0000:02:00.0: [drm] GT0: GuC firmware i915/dg2_guc_70.bin version 70.5.1
[    3.175254] i915 0000:02:00.0: [drm] GT0: HuC firmware i915/dg2_huc_gsc.bin version 7.10.3
[    3.187195] i915 0000:02:00.0: [drm] GT0: GUC: submission enabled
[    3.187215] i915 0000:02:00.0: [drm] GT0: GUC: SLPC enabled
[    3.187505] i915 0000:02:00.0: [drm] GT0: GUC: RC enabled
[    3.291626] [drm] Initialized i915 1.6.0 20201103 for 0000:02:00.0 on minor 1
[    3.326960] i915 0000:02:00.0: [drm] fb1: i915drmfb frame buffer device

I also see this error a little later that repeats a few times during boot about the linux hwe 6.5 - 6.5.0 kernel. This probably is just an issue with the latest hardware enablement kernel with my Adaptec HBA (aacraid).

[    5.802285] workqueue: drm_fb_helper_damage_work [drm_kms_helper] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[    5.807061]  sdb: sdb1
[    5.808204] sd 1:1:10:0: [sdb] Attached SCSI removable disk
[    5.814270]  sdc: sdc1
[    5.815451] sd 1:1:13:0: [sdc] Attached SCSI removable disk
[    5.823667] ses 1:3:0:0: Attached Enclosure device
[    5.936813] ================================================================================
[    5.937681] UBSAN: array-index-out-of-bounds in /build/linux-hwe-6.5-34pCLi/linux-hwe-6.5-6.5.0/drivers/scsi/aacraid/aachba.c:3855:10
[    5.938328] index 1 is out of range for type 'sgentry64 [1]'

All three devices do show up in the VM and the Intel Arc even registers with intel_gpu_top, but I'm hoping that some of these errors can be resolved and that the machine can boot faster.

root@loki:~# lspci -v
# ... skipped rest
01:00.0 RAID bus controller: Adaptec Series 7 6G SAS/PCIe 3 (rev 01)
        Subsystem: Adaptec Series 7 - ASR-7805 - 8 internal 6G SAS Port/PCIe 3.0
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at 82e00000 (64-bit, non-prefetchable) [size=1M]
        Memory at 82f00000 (64-bit, non-prefetchable) [size=1K]
        I/O ports at 8000 [size=256]
        Expansion ROM at 82f80000 [disabled] [size=512K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [b0] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: aacraid
        Kernel modules: aacraid

02:00.0 VGA compatible controller: Intel Corporation Device 56a5 (rev 05) (prog-if 00 [VGA controller])
        Subsystem: ASRock Incorporation Device 6006
        Physical Slot: 0-2
        Flags: bus master, fast devsel, latency 0, IRQ 65
        Memory at 81000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 380800000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: i915
        Kernel modules: i915

03:00.0 Audio device: Intel Corporation Device 4f92
        Subsystem: ASRock Incorporation Device 6006
        Physical Slot: 0-3
        Flags: bus master, fast devsel, latency 0
        Memory at 82c00000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intell, latency 0, IRQ 16

# skipped the rest ...

Any help or guidance is greatly appreciated!

 

[leesteken]

Proxmox (KVM/QEMU) does not currently support Resizable BAR (or all the other names it's called). Try disabling it in the motherboard BIOS. Disabling can hurt performance a lot on Intel ARC but passthrough won't work (yet) with it enabled.

 

[rubylaser]

Thanks for the idea! Passthrough is technically working. The GPU is in the VM and shows up in the lspci with the proper driver loaded, it's just not getting the re-size of the BAR (this is probably what you meant). My motherboard unfortunately doesn't seem to have an option disable resizable BAR. I also looked for SAM (Smart Access Memory), but couldn't find that either as options beyond the supporting option of Above 4G Decoding (this is enabled).

This old Reddit post led me to believe that this should be able to work

 

[leesteken]

A Proxmox Wiki page mentions it but the link it provides is not working for me:

If your GPU supports resizable BAR/SAM and you have this option set in your BIOS, you might need to deactivate it or manually tweak your BAR using an udev rule

Maybe try to convert this (AMD Radeon not Intel ARC but probably similar) into a hookscript and args: in your VM configuration file: https://angrysysadmins.tech/index.p...eable-bar-rebar-in-your-vfio-virtual-machine/

EDIT: Someone here got it working: https://forum.proxmox.com/threads/c...ype-q35-and-intel-arc-a380.127324/post-556937

 

[rubylaser]

First, thanks for the helpful suggestions! I've already tried the GRUB option and didn't notice any difference, so I've removed that again.

GRUB_CMDLINE_LINUX_DEFAULT="quiet video=efifb:off"

My virtual machine is running a 6.5 kernel with the latest Intel drivers. I tried to swap it to an i1440fx machine type, but it fails to boot with that option. I just get this error (I thought that you had to use a q35 machine for PCIe passthrough?).

TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer/PCI.pm line 486.

Edit: What I needed was the latest version of the ARC drivers for Ubuntu 22.04. I've installed those and now I don't get any BAR errors and the boot process is MUCH faster.

https://dgpu-docs.intel.com/driver/installation.html#ubuntu-package-repository

 

[leesteken]

First, thanks for the helpful suggestions! I've already tried the GRUB option and didn't notice any difference, so I've removed that again.

GRUB_CMDLINE_LINUX_DEFAULT="quiet video=efifb:off"

That does nothing on modern Proxmox. Use this work-around IF you need it: https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/page-2#post-478351

My virtual machine is running a 6.5 kernel with the latest Intel drivers. I tried to swap it to an i1440fx machine type, but it fails to boot with that option. I just get this error (I thought that you had to use a q35 machine for PCIe passthrough?).

TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer/PCI.pm line 486.

i440fx is also fine, but that (old) motherboard does not support PCIe and you cannot select PCI-Express when doing passthrough unless you use q35. Since it is virtual, the performance usually does not change but the (virtual) PCI(e) layout is different (and it differs with Machine Versions), which is mostly a problem for Windows drivers and Linux does not care (usually).

 

[ksullivan86]

Proxmox (KVM/QEMU) does not currently support Resizable BAR (or all the other names it's called). Try disabling it in the motherboard BIOS. Disabling can hurt performance a lot on Intel ARC but passthrough won't work (yet) with it enabled.

any Idea if this resizable bar support is in the works? I'm on unraid and want to move to a unraid VM but I want to makes sure i'm not going to waste my arc a770. I'm a bit bummed to hear its not currently supported.

 

[rubylaser]

any Idea if this resizable bar support is in the works? I'm on unraid and want to move to a unraid VM but I want to makes sure i'm not going to waste my arc a770. I'm a bit bummed to hear its not currently supported.

Yes, it is definitely supported. I have it working now in Ubuntu on a 6.5 kernel (see post #5 in Edit portion this thread). You just need a new kernel and drivers (installed in the VM) and to pass through the card properly to your VM.

 

[M0053KNUCKLE]

I have been trying for days to get this working. I don't even get any display on my a380. Not even the proxmox bios. (Which is weird as I was expecting at least that to show) Let alone rbar. I do see the card in the vm.

i have tried everything. Including machine types and pcie, rebar on/off. 22.04 lts, 23.10 etc.

Passthrough seems ok, but not sure. My nvidia gpu is passed through to another vm fine. This a380 is driving me bonkers. Everything is blacklisted as needed, using the right kernel options, etc.

Any help would be appreciated

 

[bond_bbs]

I ran into a similar issue about a month ago - everything should have worked but didn't.

What I ended up needing to do was changing the "Display" under hardware from the "Default" or any other "Virtual" type device to be "none" or (what I went with) "Serial terminal". I'm only passing through the Display device, but I blacklisted both the display and HDMI audio device.

Edit: I also have Rebar turned on in the bios, and the VM detects and it seems to be functioning as it should.

For some reason (at least for me), my Asrock A380 _refused_ to actually work if it wasn't the primary display on the Ubuntu VM. It would be detected, and lspci looked good - but intel_gpu_top would error, and transcoding would fail/fallback to CPU.

Now, YMMV - I'm only using my A380 for hardware transcode in Jellyfin, so cannot comment on 3d acceleration.

Screenshot for reference. The PCI `0000:2d:00` device is the Asrock A380


Edit: Spelling.

 

[neail]

@bond_bbs

Hi, thank you for your solution, I am thinking about the same setup, could you please let me know your system configuration? ( I am thinking about a i3-12100, but if a Pentium G7400 works , it will save more energy)

And does the A380 card need any dummy HDMI to function?
How did you pass through the PCIe device - as raw?
What is the avg CPU and GPU usage for 4K transcoding?