PCI Passthrough Arc A380 Into VM

T

trademarkej

New Member
Mar 6, 2024
7
0
1
I recently setup a new instance of Proxmox 8.2.2 after my Intel i5 10500 died on an AMD 5700UG based system and haven't had much luck getting iGPU transcoding working (so much easier on my old Core i5) so I opted to install an ASROCK Intel Arc A380 to do hardware transcoding in a kubernetes pod running on an Ubuntu 23.10 cloud image VM. I believe I have everything configured correctly and can add the A380 devices into the test VM I spun up as PCI devices without any errors but I can't seem to get drivers to load (weren't they supposed to be native in ~> 6.5 kernels). Any help on how to proceed with getting this card working would be greatly appreciated!

/boot/grub/grub.cfg
Bash: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

/etc/modules
Bash: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/blacklist.conf
Bash: 
blacklist i915

/etc/modprobe.d/vfio.conf
Bash: 
options vfio-pci ids=8086:56a5,8086:4f92 disable_vga=1

100.conf
Bash: 
agent: enabled=1
balloon: 0
boot: c
bootdisk: scsi0
cores: 2
hostpci0: 0000:09:00,pcie=1
hostpci1: 0000:0a:00,pcie=1
ide2: local-lvm:vm-100-cloudinit,media=cdrom,size=4M
machine: q35
memory: 4096
name: test-pci
ostype: l26
scsi0: local-lvm:vm-100-disk-0,discard=on,size=10752M,ssd=1
scsihw: virtio-scsi-pci
serial0: socket
vga: serial0

Bash: 
$ lspci -nnv
09:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: ASRock Incorporation DG2 [Arc A380] [1849:6004]
        Flags: fast devsel, IOMMU group 19
        Memory at fb000000 (64-bit, non-prefetchable) [size=16M]
        Memory at b0000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at fc000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

0a:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: ASRock Incorporation DG2 Audio Controller [1849:6004]
        Flags: fast devsel, IOMMU group 20
        Memory at fc300000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

$ lspci -n -s 09:00
09:00.0 0300: 8086:56a5 (rev 05)
$ lspci -n -s 0a:00
0a:00.0 0403: 8086:4f92
 
I /just/ got the a380 passed through to my win10 vm today, so this pain is fresh for me. I can't solve your exact problem but I can provide my settings for you to compare.

cat /etc/default/grub (my proxmox install is using legacy bios)
Bash: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet and intel_iommu=on iommu=pt"

cat /etc/modules
Bash: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

cat /etc/modprobe.d/blacklist.conf
Bash: 
blacklist i915
blacklist xe

cat /etc/modprobe.d/vfio.conf - I got the alias bit from here: https://null-src.com/posts/qemu-vfio-pci/post.php - is it needed? unknown but I have been throwing everything I can find to a solution.
Bash: 
alias pci:v00008086d000056A5sv00001849sd00006004bc03sc00i00 vfio-pci
alias pci:v00008086d00004F92sv00001849sd00006004bc04sc03i00 vfio-pci
options vfio-pci ids=8086:56a5,8086:4f92 disable_vga=1

lspci -nnv (notice the two kernel modules - xe and i915 this is why I put them both in the blacklist - needed ? I really dont know just content with my win for the day.
Bash: 
05:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: ASRock Incorporation DG2 [Arc A380] [1849:6004]
        Flags: bus master, fast devsel, latency 0, IRQ 249, NUMA node 0, IOMMU group 53
        Memory at 94000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 3bfe0000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

06:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: ASRock Incorporation DG2 Audio Controller [1849:6004]
        Flags: bus master, fast devsel, latency 0, IRQ 250, NUMA node 0, IOMMU group 54
        Memory at 95000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

My VM config to assist someone in the future or me in case I forget to back it up before doing something stupid.
Bash: 
agent: 1
bios: ovmf
boot: order=scsi0;ide0;net0
cores: 12
cpu: host
efidisk0: local-lvm:vm-115-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
--> DG2 Video --> hostpci0: 0000:05:00,pcie=1
--> DG2 Audio --> hostpci1: 0000:06:00,pcie=1
ide0: NFS-ISO:iso/virtio-win-0.1.240.iso,media=cdrom,size=612812K
machine: pc-q35-8.1
memory: 32000
meta: creation-qemu=8.1.5,ctime=1715857038
name: BI
net0: virtio=BC:24:11:BE:1C:51,bridge=vmbr1,firewall=1,mtu=9000,tag=20
numa: 0
ostype: win11
scsi0: local-lvm:vm-115-disk-0,cache=writeback,discard=on,iothread=1,size=1000G
scsihw: virtio-scsi-single
smbios1: uuid=(id here)
sockets: 2
vga: none
vmgenid: (id here)
 
Last edited:
  • Like
Reactions: trademarkej
SoggyCrouton said:
I /just/ got the a380 passed through to my win10 vm today, so this pain is fresh for me. I can't solve your exact problem but I can provide my settings for you to compare.

cat /etc/default/grub (my proxmox install is using legacy bios)
Bash: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet and intel_iommu=on iommu=pt"

cat /etc/modules
Bash: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

cat /etc/modprobe.d/blacklist.conf
Bash: 
blacklist i915
blacklist xe

cat /etc/modprobe.d/vfio.conf - I got the alias bit from here: https://null-src.com/posts/qemu-vfio-pci/post.php - is it needed? unknown but I have been throwing everything I can find to a solution.
Bash: 
alias pci:v00008086d000056A5sv00001849sd00006004bc03sc00i00 vfio-pci
alias pci:v00008086d00004F92sv00001849sd00006004bc04sc03i00 vfio-pci
options vfio-pci ids=8086:56a5,8086:4f92 disable_vga=1

lspci -nnv (notice the two kernel modules - xe and i915 this is why I put them both in the blacklist - needed ? I really dont know just content with my win for the day.
Bash: 
05:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: ASRock Incorporation DG2 [Arc A380] [1849:6004]
        Flags: bus master, fast devsel, latency 0, IRQ 249, NUMA node 0, IOMMU group 53
        Memory at 94000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 3bfe0000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

06:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: ASRock Incorporation DG2 Audio Controller [1849:6004]
        Flags: bus master, fast devsel, latency 0, IRQ 250, NUMA node 0, IOMMU group 54
        Memory at 95000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

My VM config to assist someone in the future or me in case I forget to back it up before doing something stupid.
Bash: 
agent: 1
bios: ovmf
boot: order=scsi0;ide0;net0
cores: 12
cpu: host
efidisk0: local-lvm:vm-115-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
--> DG2 Video --> hostpci0: 0000:05:00,pcie=1
--> DG2 Audio --> hostpci1: 0000:06:00,pcie=1
ide0: NFS-ISO:iso/virtio-win-0.1.240.iso,media=cdrom,size=612812K
machine: pc-q35-8.1
memory: 32000
meta: creation-qemu=8.1.5,ctime=1715857038
name: BI
net0: virtio=BC:24:11:BE:1C:51,bridge=vmbr1,firewall=1,mtu=9000,tag=20
numa: 0
ostype: win11
scsi0: local-lvm:vm-115-disk-0,cache=writeback,discard=on,iothread=1,size=1000G
scsihw: virtio-scsi-single
smbios1: uuid=(id here)
sockets: 2
vga: none
vmgenid: (id here)
Click to expand...

Ok, so I checked my VM using lscpi -nnv and it appears that both the video and audio devices are listed. I don't see any listing for the drivers being loaded though.

Bash: 
$ uname -srm
Linux 6.8.0-31-generic x86_64

$ lscpi -nnv
01:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: ASRock Incorporation DG2 [Arc A380] [1849:6004]
        Physical Slot: 0
        Flags: fast devsel
        Memory at fb000000 (64-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at fc000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting

02:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: ASRock Incorporation DG2 Audio Controller [1849:6004]
        Physical Slot: 0-2
        Flags: fast devsel
        Memory at fde00000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting

What am I still missing?
 
Last edited:
trademarkej said:
Ok, so I checked my VM using lscpi -nnv and it appears that both the video and audio devices are listed. I don't see any listing for the drivers being loaded though.

Bash: 
$ uname -srm
Linux 6.8.0-31-generic x86_64

$ lscpi -nnv
01:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: ASRock Incorporation DG2 [Arc A380] [1849:6004]
        Physical Slot: 0
        Flags: fast devsel
        Memory at fb000000 (64-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at fc000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting

02:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: ASRock Incorporation DG2 Audio Controller [1849:6004]
        Physical Slot: 0-2
        Flags: fast devsel
        Memory at fde00000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting

What am I still missing?
Click to expand...
After going over my configuration several times, I ended up updating the VM I was running from Ubuntu 23.10 to 24.04 LTS and suddenly the kernel drivers started working in the VM / pod! Very frustrating that I assumed the drivers were working correctly this whole time
 
Hey everyone :) first of all thanks for sharing your configs. With that I finally also got it working.
Just thought I would share some Points that I specifically had to look out for to get my Asrock A380 (the low profile Version) passed through to the VM since I was also struggling a bit:
  • As a OS Image I used the Ubuntu 24.04. LTS Server iso right away
  • Also when creating the VM I chose to stick with the Default BIOS (so for me it was SeaBIOS; UEFI wasn´t working for me before)
  • Important thing was to choose q35 Machine Type
  • Boot the VM without GPU PCI Devices passed through and run update / upgrade
  • Install qemu-guest-agent (not sure if that is needed but I installed it anyway)
  • Shutdown the VM
  • Add PCI Devices (VGA + Audio Device) and check the "PCI-Express" Checkbox
One thing is that when starting the VM at some Point the Web Console in Proxmox seems to hang (log actually shows a connection timeout error) but I could still log in over SSH and test the pass-through by installing intel-gpu-tools and then running intel_gpu_top.

Also what I noticed right after installing the GPU and booting into Proxmox that the FAN was basically pinned at 100%. And it still is even after passing it to the VM. Now the placement inside my chassis is far from ideal since right now the only possible solution for me was to use the outermost pci slot. So the Card has basically no breathing room in front of the fans. Also currently temperatures in the room itself are pretty warm as well, so not sure if it is just so warm that the fans have to spin this fast even when idling or if there is another issue.

Anyway, thanks againg :) will proceed to install jellyfin now :D
 
Since the performance of Intel ARC depends a lot on Resizable BAR
Click to expand...
Yes, that is also what I read. Thankfully for my Setup (with a Supermicro X11DPI Mainboard) I could patch my BIOS and enable that feature with the ReBarUEFI Tool (check the wiki -> Adding ffs Module). Important to note is that I used the MMTool Method (apparently the UEFITool Method led to some non-booting Supermicro boards). Also since it took me a while to find the correct Download Link for the MMTool, here it is: link.

lspci output before ReBar Patch (notice the Memory at ... [size=256M] line):
Code: 
lspci -n -s 67:00 -v
67:00.0 0300: 8086:56a5 (rev 05) (prog-if 00 [VGA controller])
        Subsystem: 1849:6006
        Flags: fast devsel, NUMA node 0, IOMMU group 9
        Memory at df000000 (64-bit, non-prefetchable) [disabled] [size=16M]
        Memory at 38bfe0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Expansion ROM at e0000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel modules: i915, xe


After Patch:
Code: 
67:00.0 0300: 8086:56a5 (rev 05) (prog-if 00 [VGA controller])
        Subsystem: 1849:6006
        Flags: bus master, fast devsel, latency 0, IRQ 200, NUMA node 0, IOMMU group 9
        Memory at df000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 38bc00000000 (64-bit, prefetchable) [size=8G]
        Expansion ROM at e0000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

Also I wanted to follow up on my issue with the Fan constantly Spinning at a high rate: First thing I tried was with a fresh Ubuntu 22 VM and the Intel out-of-tree kernel modules: link but that didn´t resolve the issue. Second I tried enforcing ASPM with the help of the ASPM tuning script: link but that also didn´t get the FAN speed under control. Then I found an article from Intel regarding the issue: link. Unfortunately it is not that easy to update the firmware on Intel ARC cards under Linux (here is a guide though). Now since I wasn´t sure if I could use the OPROM from the Asrock Challenger A380 (which is the full-height version of this card, and the only one that is mentioned in the firmware compilation matrix linked in the mentioned guide) for my Low-Profile Version I decided to just go the "official" route and install the Card in a Windows Desktop and install the Intel Driver (which apparently during Driver installation also does a Firmware Update). Even though I couldn´t find any evidence that the Windows Driver install actually updated the firmware on the card it is now back in my Server and running super quiet, with a very moderate fan-speed (as I would expect) for a few hours now :)
 
How can i blacklist just my DS2? my iGPU (00:02.0) uses "i915" and "xe" also..

Code: 
lspci -nnv

00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-P [Iris Xe Graphics] [8086:a7a0] (rev 04) (prog-if 00 [VGA controller])
        DeviceName: Onboard - Video
        Flags: bus master, fast devsel, latency 0, IRQ 270, IOMMU group 0
        Memory at 6246000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 3000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: i915
        Kernel modules: i915, xe

03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A310] [8086:56a6] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: Device [172f:4019]
        Flags: fast devsel, IOMMU group 17
        Memory at 6d000000 (64-bit, non-prefetchable) [disabled] [size=16M]
        Memory at 6000000000 (64-bit, prefetchable) [disabled] [size=4G]
        Expansion ROM at 6e000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

04:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: Device [172f:4019]
        Flags: fast devsel, IOMMU group 18
        Memory at 6e300000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
 
DigitalLF said:
How can i blacklist just my DS2? my iGPU (00:02.0) uses "i915" and "xe" also..

Code: 
lspci -nnv

00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-P [Iris Xe Graphics] [8086:a7a0] (rev 04) (prog-if 00 [VGA controller])
        DeviceName: Onboard - Video
        Flags: bus master, fast devsel, latency 0, IRQ 270, IOMMU group 0
        Memory at 6246000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 3000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: i915
        Kernel modules: i915, xe

03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A310] [8086:56a6] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: Device [172f:4019]
        Flags: fast devsel, IOMMU group 17
        Memory at 6d000000 (64-bit, non-prefetchable) [disabled] [size=16M]
        Memory at 6000000000 (64-bit, prefetchable) [disabled] [size=4G]
        Expansion ROM at 6e000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

04:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: Device [172f:4019]
        Flags: fast devsel, IOMMU group 18
        Memory at 6e300000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
Click to expand...
Your can early bind to vfio-pci 8086:56a6 (which is the ARC) and not 8086:a7a0 (which is the integrated graphics). No need to blacklist drivers. But you might want to early bind the other functions of the ARC GPU also (8086:4f92).
 
  • Like
Reactions: alkaizer
DigitalLF said:
How can i blacklist just my DS2? my iGPU (00:02.0) uses "i915" and "xe" also..

Code: 
lspci -nnv

00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-P [Iris Xe Graphics] [8086:a7a0] (rev 04) (prog-if 00 [VGA controller])
        DeviceName: Onboard - Video
        Flags: bus master, fast devsel, latency 0, IRQ 270, IOMMU group 0
        Memory at 6246000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 3000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: i915
        Kernel modules: i915, xe

03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A310] [8086:56a6] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: Device [172f:4019]
        Flags: fast devsel, IOMMU group 17
        Memory at 6d000000 (64-bit, non-prefetchable) [disabled] [size=16M]
        Memory at 6000000000 (64-bit, prefetchable) [disabled] [size=4G]
        Expansion ROM at 6e000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [d0] Power Management version 3
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [420] Physical Resizable BAR
        Capabilities: [400] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i915, xe

04:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
        Subsystem: Device [172f:4019]
        Flags: fast devsel, IOMMU group 18
        Memory at 6e300000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Latency Tolerance Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
Click to expand...
Like leesteken stated, edit the /etc/modprobe.d/vfio.conf file and include "options vfio-pci ids=8086:56a6,8086:4f92 disable_vga=1"
 
Just to add to this thread in case it helps anyone or I forget, I took similar steps as above for my Supermicro X10SRL-F except for systemd-boot instead of GRUB since my pve system is using UEFI BIOS. All of this is in the official docs https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough too.

First I ensured Intel VT-d was enabled in BIOS. Mine was under Advanced > Chipset Configuration > North Bridge > IIO Configuration > Intel VT for Directed I/O (VT-d).

Next I added intel_iommu=on to the end of my /etc/kernel/cmdline file:
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

And this to /etc/modules:
Code: 
vfio
vfio_iommu_type1
vfio_pci

To make sure the host doesn't use the GPU I only needed to blacklist the i915 driver. In /etc/modprobe.d/blacklist.conf
Code: 
blacklist i915

Note: you may also need to pass the device IDs to the options of the vfio-pci modules (as noted in the docs) to get passthrough to work. I didn't, but I just wanted to add this here in case you still have issues.

I then ran these two commands to update those changes, followed by a reboot.
Code: 
proxmox-boot-tool refresh
update-initramfs -u -k all

To verify IOMMU is enabled I ran this command
Code: 
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
You should see a line that looks like "DMAR: IOMMU enabled" indicating IOMMU is working.

To make sure the host wasn't using the GPU, I ran these two commands to see which kernel drivers were being used.
Code: 
lspci | grep Arc # To get the device ID at the start of the line
lspci -n -s 07:00 -v
If you see Kernel driver in use: vfio-pci (or the in use line is missing entirely) you should be good to go for passthrough.

Now this was the critical part for me to get this working. I tried doing these steps for a Debian 12 VM and installing intel-gpu-tools but even after all this I was getting errors when running "sudo intel_gpu_top". As far as I understand it, Intel ARC gpus don't work well for some of the older linux kernels, and Debian 12 uses 6.1.0. So I spun up an Ubuntu 24.04 VM which uses 6.8.0 and it worked! So I am using Ubuntu now and the GPU passthrough is working great.

After installing Ubuntu, I installed these packages then shutdown the VM
Code: 
sudo apt update && sudo apt upgrade -y
sudo apt install intel-gpu-tools
sudo apt install qemu-guest-agent # This is not needed for GPU passthrough but I install it to enable the guest agent
In the VM config Add > PCI Device and select Raw Device with the device ID. I checked All Functions and PCIe Device checkboxes as well. Then start up the VM and run sudo intel_gpu_top to verify passthrough is working.

The only other thing to do would be to enable ReBAR in the BIOS as mentioned above, but I'm a little hesitant right now to risk bricking my motherboard unless I start to see some performance issues. It would probably be best to do it but for now it's working for me.
 
You must log in or register to reply here.
Top Bottom