Issue with Intel Arc A380 PCIe passthrough

[Dark18]

Hello,

I have a proxmox server with an AMD Ryzen 9 5900X and (previously) a RTX 2070 Super for encoding. PCIe passthrough works fine with it.
The host itself uses a second GPU (Nvidia Quadro P600).

Now I would like to switch to a new Arc A380 (because of av1 support).
I already tried the A380 with my Ubuntu desktop pc, where it works fine.

On my proxmox server, I upgraded the kernel to 6.2.16-11-bpo11-pve and modified the configuration.
The A380 is shown as a PCIe device on the proxmox host without model name.

But when doing a passthrough to an Ubuntu VM, it freezes at boot with the following errors for...
...an already installed Ubuntu:

...an Ubuntu live image:
:


I managed to boot using recovery mode, where the A380 is shown by lspci -nnv:

While it is displayed correctly, I was not able to utilize it in any way. I tried using it for QSV encoding and main GPU.

A Windows VM does boot, but it only shows an unkown PCIe device in the device manager. Installing the Intel Arc driver fails.

There are no errors in dmesg or journalctl.

cat /etc/modprobe.d/vfio.conf

alias pci:v00008086d000056A5sv00001849sd00006004bc03sc00i00 vfio-pci
alias pci:v00008086d00004F92sv00001849sd00006004bc04sc03i00 vfio-pci

options vfio-pci ids=8086:56a5,8086:4f92 disable_vga=1

cat /etc/modprobe.d/blacklist.conf

blacklist i915

Grub cmdline: GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

cat /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

lspci on the proxmox host shows it as an Intel device, but without the model (like on the Ubuntu VM)

lspci -nnv

...
0f:00.0 VGA compatible controller [0300]: Intel Corporation Device [8086:56a5] (rev 05) (prog-if 00 [VGA controller])
    Subsystem: ASRock Incorporation Device [1849:6004]
    Flags: bus master, fast devsel, latency 0, IRQ 139, IOMMU group 19
    Memory at ec000000 (64-bit, non-prefetchable) [size=16M]
    Memory at d0000000 (64-bit, prefetchable) [size=256M]
    Expansion ROM at ed000000 [disabled] [size=2M]
    Capabilities: [40] Vendor Specific Information: Len=0c <?>
    Capabilities: [70] Express Endpoint, MSI 00
    Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
    Capabilities: [d0] Power Management version 3
    Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [420] Physical Resizable BAR
    Capabilities: [400] Latency Tolerance Reporting
    Kernel driver in use: vfio-pci
    Kernel modules: i915

10:00.0 Audio device [0403]: Intel Corporation Device [8086:4f92]
    Subsystem: ASRock Incorporation Device [1849:6004]
    Flags: fast devsel, IOMMU group 20
    Memory at ed300000 (64-bit, non-prefetchable) [disabled] [size=16K]
    Capabilities: [50] Power Management version 3
    Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [70] Express Endpoint, MSI 00
    Capabilities: [100] Latency Tolerance Reporting
    Kernel driver in use: vfio-pci
    Kernel modules: snd_hda_intel

...

It is shown as an IOMMU device:

ls /sys/kernel/iommu_groups/*/devices

...

/sys/kernel/iommu_groups/20/devices:
0000:10:00.0

...

cat /etc/pve/qemu-server/109.conf


acpi: 0
boot: order=scsi0
cores: 4
cpuunits: 100
hostpci1: 0000:0f:00,pcie=1
machine: q35
memory: 4000
meta: creation-qemu=7.0.0,ctime=1737917780
name: ubuntu
net0: virtio=82:12:34:56:78:56,bridge=vmbr1
numa: 0
ostype: l26
scsi0: m2:109/vm-109-disk-1.qcow2,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=db79c6ef-fdc6-438f-ac87-2e5900789c4f
sockets: 1
startup: order=4
vga: memory=96
vmgenid: 1c...d2

dmesg | grep -E "IOMMU|DMAR|vfio|vga"

[    0.739070] pci 0000:09:00.0: vgaarb: setting as boot VGA device
[    0.739070] pci 0000:09:00.0: vgaarb: bridge control possible
[    0.739070] pci 0000:09:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.739070] pci 0000:0f:00.0: vgaarb: bridge control possible
[    0.739070] pci 0000:0f:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    0.739070] vgaarb: loaded
[    0.750581] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.751608] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.760701] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[   42.131127] vfio-pci 0000:0f:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[   42.131176] vfio_pci: add [8086:56a5[ffffffff:ffffffff]] class 0x000000/00000000
[   42.131311] vfio_pci: add [8086:4f92[ffffffff:ffffffff]] class 0x000000/00000000
[  151.439712] vfio-pci 0000:0f:00.0: enabling device (0000 -> 0002)

I would be grateful for any help.

 

[leesteken]

I have no experience with Intel ARC and I don't know if Intel ARC resets properly, so you might need to make sure no driver touches it before the VM starts (blacklist or early bind to vfio-pci).
Make sure to disable Resizable BAR (SAM, CAM) because it is not yet supported with passthrough. Unfortunately, that will impact the performance of Intel ARC more than other GPUs.

EDIT: Your post disappeared but I see now that you already boot with another GPU, so I updatesd my reply.

 

[Dark18]

I have no experience with Intel ARC and I don't know if Intel ARC resets properly, so you might need to make sure no driver touches it before the VM starts (blacklist or early bind to vfio-pci).

I blacklisted blacklist i915 in modprobe.d and according to lspci it is used by vfio-pci.

Make sure to disable Resizable BAR (SAM, CAM) because it is not yet supported with passthrough. Unfortunately, that will impact the performance of Intel ARC more than other GPUs.

I disabled it in the host UEFI, but it changed nothing, unfortunately.

 

[leesteken]

I blacklisted blacklist i915 in modprobe.d and according to lspci it is used by vfio-pci.
I disabled it in the host UEFI, but it changed nothing, unfortunately.

I fear you'll need to find out if anyone (on the internet) has been able to passthrough a Intel ARC with QEMU/KVM and what work-arounds they needed. I could not find any success stories on this forum, unfortunately. The Primary GPU option does not make sense for ARC (as it is a work-around for NVidia). You could try dumping the ROM-file and adding that manually to the hostpci0 setting.

 

[Dark18]

Well, I found 2 or 3 reports where it worked with some kind of ARC GPU.
But they don't have the same issue as I have.

I still tried their solutions, like the script from this thread, without success:
https://forum.proxmox.com/threads/i...-pci-not-ready-after-flr-or-bus-reset.130667/

 

[blebo]

What version of Ubuntu are you using ? You need at least 22.04.3 LTS for kernel 6.2 with ARC support (maybe try upgrading your existing Ubuntu VM to the HWE kernel).

 

[Dark18]

What version of Ubuntu are you using ? You need at least 22.04.3 LTS for kernel 6.2 with ARC support (maybe try upgrading your existing Ubuntu VM to the HWE kernel).

I am using Ubuntu 23.04 with kernel 6.2

 

[the1corrupted]

Hi,

I hope this isn't necroposting but I am curious about a couple more things.

I am a little new but have you been able to verify the passthrough is working correctly from the host?

Did you do this for a new VM or pre existing? Did you preinstall the ARC drivers on the Ubuntu instance? Have you tried another OS or clones instance?

Do you have ECC ram or standard unbuffered desktop memory? This may not actually make a difference I am just curious.

 

[Dark18]

I am a little new but have you been able to verify the passthrough is working correctly from the host?

On the host, I could run the intel driver tools when the drivers were loaded and the passthrough disabled.
And also when enabling the passthrough, I could not find any errors on the host.

Did you do this for a new VM or pre existing? Did you preinstall the ARC drivers on the Ubuntu instance? Have you tried another OS or clones instance?

I tried with an already existing Ubuntu VM that used a 2070 Super before. I installed all possible drivers and also tried multiple options, like free and non-free. I tried a new Ubuntu and Arch Linux installation as well, which did not work.

Do you have ECC ram or standard unbuffered desktop memory

I have Kingston UDIMM modules with ECC (KSM26ED8/32ME) on a B550 AORUS ELITE V2.
ECC works fine according to memtest86.



I am now using a LXC container, which can easily access the GPU.

 

[skittlebrau]

I've been able to successfully passthrough an AsRock Arc A380 Challenger ITX GPU to Linux VMs running kernel 6.2 and higher.

However, you have to disable the kvm virtual GPU (set Display to 'none') and set the Arc GPU as 'Primary GPU' in your VM's Hardware settings, therefore you need to rely on using SSH with your VM.

If you don't do that, then you'll get the same apparent 'freeze' (it's not actually frozen as you can SSH in still) on VM startup.

I haven't had this problem with NVIDIA or AMD GPUs before, so there's just something about the Intel driver that demands it be 'primary' which causes the behaviour you've seen.

 

[skittlebrau]

I've also just tested out using the serial console and that works great too when you've got the Intel Arc GPU set as primary, so at least you can get console output from the Proxmox UI.

https://devopstales.github.io/virtualization/proxmox-xtermjs-enable/

 

[sarbian]

Adding i915.modeset=0 to the VM boot options prevents the intel card from taking over the console. I have my display set to SPICE and it works fine after booting with that option.

 

[skittlebrau]

Adding i915.modeset=0 to the VM boot options prevents the intel card from taking over the console. I have my display set to SPICE and it works fine after booting with that option.

After adding that to

/etc/default/grub

and doing

update-grub

I've got the Proxmox Web UI console back. Thanks!

 

[skittlebrau]

Adding i915.modeset=0 to the VM boot options prevents the intel card from taking over the console. I have my display set to SPICE and it works fine after booting with that option.

I spoke too soon.

Disabling modesetting for i915 killed off transcoding ability for my Arc GPU, so I've had to revert to using serial console.