igd Passthru on Intel Sandy Bridge

tkffaul

New Member
Apr 8, 2021
16
9
3
45
I know this is older hardware... but i'm trying to get a nice little LXC/Windows setup going for a friend. I want to pass the igd into windows so that when the system boots, my friend will see windows and not linux/proxmox. I have been at this for a week... reading and talking to people... and I have followed every guide on the forums that I can find, but everyone results the same:

Code:
kvm: -device vfio-pci,host=0000:00:02.0,id=hostpci0,bus=pci.0,addr=0x2: vfio 0000:00:02.0: error getting device from group 1: Cannot allocate memory
Verify all devices in group 1 are bound to vfio-<bus> or pci-stub and not already in use
TASK ERROR: start failed: QEMU exited with code 1

I ready a bunch over at qemu, I've looked at TONS of kernel options... I guess I need help where to start. I know i have the vga broke out into its own IOMMU group and that group is 1 (as the error points out). My grub config currently looks like:

Code:
GRUB_CMDLINE_LINUX_DEFAULT="nomodeset audit=0 loglevel=3 pcie_acs_override=downstream,multifunction intel_iommu=on video=vesafb:off,efifb:off,simplefb:off nofb vfio-pci.ids=1028:0498,8086:010a,8086:244e vfio_iommu_type1.allow_unsafe_interrupts=1"
GRUB_TERMINAL=console
I still get the proxmox boot screen, even after all these settings are appied. I can confirm the vfio drive is loading on the device. I have tried multiple different things for qemu config:
Code:
hostpci0: 00:02.0,legacy-igd=1
args: -device vfio-pci,host=00:02.0 -nographic
args: -device vfio-pci,host=00:02.0,addr=0x02,x-igd-gms=10,x-igd-opregion=on,romfile=hd2000.rom

Obviously not all of those at the same time, but i wanted to show the different options. I also created a rom file from the bios. Followed the steps and re-ID'ed the file.

Again everything results in the same message above.
VIFO has a message too:
Code:
vfio-pci 0000:00:02.0: vfio_pci: Failed to setup Intel IGD regions

I've read that grub can cause issues with hooking the frame buffer, which is why i said the proxmox does display the boot screen and part of the boot up then the screen freezes and the system finishes booting. When i start the VM the screen goes blank like its going to pass it, but then qemu crashes. Any help would be appreciated.

** Sorry this is a E3-1260L SandyBridge and if i left out info you want to see just let me know what you want to see
 
Last edited:
So progress... I have changed my qemu entries to:
args: -device vfio-pci,host=00:02.0,id=hostpci0,bus=pci.0,addr=0x02,x-igd-gms=1,x-igd-opregion=on,romfile=/usr/share/kvm/i915ovmf.rom

This removed most of my errors, so I'm down to:
pve-test kernel: vfio-pci 0000:00:02.0: vfio_pci: Failed to setup Intel IGD regions

Still the exact same output from proxmox as above though... Any ideas? I've been searching the forums and I've found a few things to try but nothing has fixed the issue yet.
 
I cant believe no one has any suggestions... I found an write up on arch talking about needing to define the memory region for the IGD but it requires patching vfio with vfio_pci_igd.c but the posts are from 2019... but they are hard coding the device id's and the memory regions this way... Is this required for what I'm doing or has this been handled by the qemu quirks? because i found those as well.
 
I am experiencing this problem too: there's obviously some kind of quirk in HP firmware: ALL HP workstations/machines with Sandy Bridge have this problem, and I've tried three of those:

Elite 8200 usff pc
Z210 sff workstation
hp 8100

All same error in VFIO "Failed to setup Intel IGD regions".

Tried ANY kind of option in kernel/qemu, still got this error. Passtrough of a PCIe video card worked flawlessly, but IGD was a no-go.

Struggling with those old machines was truly frustrating: IGD passtrough worked in all Dell pcs, in most Asus/asrock/msi Motherboards, but HP workstations are a dead end, at least for me.

I suspect it has something to do with the vPro/Intel ME/AMT stuff: it's a computer-in-the-computer "ghost machine" that can access the video buffer/network card etc to remotely control the machine at low-level. The AMT/remote display works only with the IGD and not with a discrete graphics card: I think the firmware locks the IGD memory buffer in some obscure way, preventing VFIO to take full control of it.
 
So I've solved this problem... with help from Alex Williamson! So the issue for me was a bad memory map in the bios for the location of the IGD memory space. We will have to reset it in software for the remap to work correctly.

Assuming your IGD is at 00:02.0 enter the following to get the Memory Region that the bios is currently putting out:
Code:
setpci -s 0000:00:02.0 fc.l

Then confirm that the region exists in ACPI non-volatile storage:
Code:
cat /proc/iomem

Run this command to see the size of the opregion (replace 0xbacfa018 with your region address from above):
Code:
dd bs=1 skip=$((0xbacfa018 + 16)) count=4 if=/dev/mem 2>/dev/null | xxd

If your output is: 00000000: 0800 0000 your done and this isnt your issue. If not then keep following along.

Install a hex editor:
Code:
apt-get install hexedit

Export the above to a file for editing (replace 0xbacfa018 with your region address from above):
Code:
dd bs=1 skip=$((0xbacfa018 + 16)) count=4 if=/dev/mem of=opregion-size-patch

Edit your Patch File:
Code:
hexedit opregion-size-patch
Make the value: 08 00 00 00

Then Run (replace 0xbacfa018 with your region address from above):
Code:
dd bs=1 seek=$((0xbacfa018 + 16)) count=4 if=opregion-size-patch of=/dev/mem
This will reset the opregion in software and tell your machine where the memory region really is.

Verify with (replace 0xbacfa018 with your region address from above):
Code:
dd bs=1 skip=$((0xbacfa018 + 16)) count=4 if=/dev/mem 2>/dev/null | xxd

You should now be able to start the vm and pass the IGD off. The second to last line that fixes this issue. You will need run after every reboot or the bios map will take back over. I created a script at boot that runs this every time. I also needed to use an oprom file in the VM to get the IGD to come up. I ended up pulling apart a driver from asus to make this work. I have attached the rom file i use (place this in /usr/share/kvm), but you will probably need to get Alex's rom-parser to fix the file to load correctly. They are making modifications to the virtio driver so that this is automatically caught in the future but till that works its way down, this has worked on every sandybridge system that I've had issues with. Let me know if it works for you, or if your issue is different.
 

Attachments

Last edited:
  • Like
Reactions: Wyk72 and leesteken
I am trying your solution, but it seems I ran into another problem:


root@immortality-d01b5b:~# setpci -s 0000:00:02.0 fc.l
78d0d018

"78d0d018" is in the ACPI region:


cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009ffff : System RAM
000a0000-000bffff : PCI Bus 0000:00
000c0000-000dffff : PCI Bus 0000:00
000c0000-000cdbff : Video ROM
000ce000-000cefff : Adapter ROM
000f0000-000fffff : System ROM
00100000-0258d017 : System RAM
0258d018-0259d057 : System RAM
0259d058-0259e017 : System RAM
0259e018-025adc57 : System RAM
025adc58-1fffffff : System RAM
20000000-201fffff : Reserved
20200000-3fffffff : System RAM
40000000-401fffff : Reserved
40200000-78bcafff : System RAM
68000000-68a01cb6 : Kernel code
68c00000-68f27fff : Kernel rodata
69000000-6913ed3f : Kernel data
69604000-69bfffff : Kernel bss
78bcb000-78c13fff : ACPI Non-volatile Storage
78c14000-78c21fff : ACPI Tables
78c22000-78c22fff : ACPI Non-volatile Storage
78c23000-78c50fff : Reserved
78c51000-78c72fff : ACPI Non-volatile Storage
78c73000-78cf9fff : Reserved
78cfa000-78cfafff : ACPI Non-volatile Storage
78cfb000-78d0cfff : Reserved
78d0d000-78d1afff : ACPI Non-volatile Storage
78d1b000-78d2cfff : Reserved
78d2d000-78d6ffff : ACPI Non-volatile Storage
78d70000-78ffffff : System RAM
79000000-797fffff : RAM buffer
79800000-7d9fffff : Reserved
7da00000-ffffffff : PCI Bus 0000:00
d0000000-dfffffff : 0000:00:02.0
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
e0000000-efffffff : pnp 00:00
fe000000-fe3fffff : 0000:00:02.0
fe400000-fe41ffff : 0000:00:19.0
fe400000-fe41ffff : e1000e
fe420000-fe423fff : 0000:00:1b.0
fe424000-fe4240ff : 0000:00:1f.3
fe425000-fe4257ff : 0000:00:1f.2
fe425000-fe4257ff : ahci
fe426000-fe4263ff : 0000:00:1d.0
fe426000-fe4263ff : ehci_hcd
fe427000-fe4273ff : 0000:00:1a.0
fe427000-fe4273ff : ehci_hcd
fe428000-fe428fff : 0000:00:19.0
fe428000-fe428fff : e1000e
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed08000-fed08fff : pnp 00:07
fed10000-fed19fff : pnp 00:00
fed1c000-fed3ffff : Reserved
fed1c000-fed1ffff : pnp 00:07
fed1f410-fed1f414 : iTCO_wdt.1.auto
fed1f410-fed1f414 : iTCO_wdt.1.auto iTCO_wdt.1.auto
fed20000-fed3ffff : pnp 00:00
fed40000-fed44fff : 00:06 TPM
fed90000-fed90fff : dmar0
fed91000-fed91fff : dmar1
fee00000-fee0ffff : pnp 00:00
fee00000-fee00fff : Local APIC
ff000000-ffffffff : Reserved
ff000000-ffffffff : pnp 00:07
100000000-1005fffff : System RAM
100600000-103ffffff : RAM buffer
5.10.59-0-lts #1-Alpine

BUT:

dd bs=1 skip=$((0x78d0d018 + 16)) count=4 if=/dev/mem
dd: error reading '/dev/mem': Operation not permitted
0+0 records in
0+0 records out
0 bytes copied, 4.4191e-05 s, 0.0 kB/s

I have no idea why the "operation is not permitted". Running kernel 5.10.59-0-lts on apine Linux 3.14.

Any suggestions ?
 
I see, the kernel is configured, as most new kernels do, as

cat /boot/config-lts | grep CONFIG_STRICT_DEVMEM

CONFIG_STRICT_DEVMEM=y

That prevents access to /dev/mem .....seems I have to recompile it :(
 
Update: I re-configured my kernel, recompiled it, and now I've got access to /dev/mem

Here is my output, doing this in real time:

root@immortality-d01b5b:~# setpci -s 0000:00:02.0 fc.l
78d0d018

Ok, that's the OpRegion i guess

Now let's check it:


root@immortality-d01b5b:~# dd bs=1 skip=$((0x78d0d018 + 16)) count=4 if=/dev/mem 2>/dev/null | xxd
00000000: 0020 0000

mmh..looks the ouput is NOT 00000000: 0800 0000 as per your instructions.

Let's try to patch it.. (hexedited as per you instructions....and wrote it back with dd)

Now it looks better:

root@immortality-d01b5b:~# dd bs=1 skip=$((0x78d0d018 + 16)) count=4 if=/dev/mem 2>/dev/null | xxd
00000000: 0800 0000

Launching qemu.................

AND - AMAZINGLY- IT WORKS !!!

Omg, can't believe this thing is running. Wasted so much time in this issue.

Can't thank you enough for this: I had totally lost hope in seeing an IGD passthrough working in this USDT Hp 8200 Elite PC (ancient i5-2400 SandyBridge , Q67 chipset).

You deserve a medal for that.

The only problem is that the kernel is .. very unsecure, with root write access to /dev/mem !

Still, amazing achievement. Blame on HP for making ugly, ugly, fugly OEM BIOS for these machines.
 
I'm having issues with win10 though: every time I add the Sandy Bridge "HD Graphics" driver, it crashes. But maybe this machine does not have enough physical memory.

A quick boot of an Ubuntu 20.04 ISO worked flawlessly :)
 
Last edited:
I'm having issues with win10 though: every time I add the Sandy Bridge "HD Graphics" driver, it crashes. But maybe this machine does not have enough physical memory.

A quick boot of an Ubuntu 20.04 ISO worked flawlessly :)
When I was trying the iGPU passthrough I found the newer Intel Windows drivers failed, I can't remember which version, but an older version solved my problem.

DerekG
 
I'm having issues with win10 though: every time I add the Sandy Bridge "HD Graphics" driver, it crashes. But maybe this machine does not have enough physical memory.

A quick boot of an Ubuntu 20.04 ISO worked flawlessly :)
I'm so sorry, got caught up in putting a fence in at my house and just didn't get back to you in time! I'm really glad you got it working! I have it working on windows 10 with driver 9.17.10.4459. Im running 16gb total memory with 6gb in windows... no issues thus far! I hope others can get some use out of this!
 
So I'm trying to figure out if I'm also affected by the issue but I'm having trouble with checking if the region exists in ACPI non-volatile storage.

Code:
$ setpci -s 0000:00:02.0 fc.l
6d9d7018

But I don't find the region 6d9d7018 in /proc/iomem.

Code:
$ cat /proc/iomem

00000000-00000fff : Reserved
00001000-0003efff : System RAM
0003f000-0003ffff : Reserved
00040000-0009dfff : System RAM
0009e000-000fffff : Reserved
  000f0000-000fffff : System ROM
00100000-0fffffff : System RAM
10000000-12150fff : Reserved
12151000-60fc0017 : System RAM
60fc0018-60fd0857 : System RAM
60fd0858-60fd1017 : System RAM
60fd1018-60fe1857 : System RAM
60fe1858-60fe2017 : System RAM
60fe2018-60ff0057 : System RAM
60ff0058-60ff1017 : System RAM
60ff1018-61001057 : System RAM
61001058-6a11afff : System RAM
6a11b000-6a15bfff : Reserved
6a15c000-6b4a8fff : System RAM
6b4a9000-6d5c9fff : Reserved
6d5ca000-6d9bdfff : System RAM
6d9be000-6d9dcfff : ACPI Non-volatile Storage
6d9dd000-6dde0fff : Reserved
6dde1000-6e153fff : System RAM
6e154000-6e154fff : ACPI Non-volatile Storage
6e155000-6e17efff : Reserved
6e17f000-6e756fff : System RAM
6e757000-6e758fff : Reserved
6e759000-6effffff : System RAM
6f000000-7fffffff : Reserved
  6f800001-6fffffff : PCI Bus 0000:00
  70000001-7fffffff : PCI Bus 0000:00
80000000-cfffffff : PCI Bus 0000:00
  80000000-8fffffff : 0000:00:02.0
  90000000-90ffffff : 0000:00:02.0
  91000000-920fffff : PCI Bus 0000:04
    91000000-913fffff : 0000:04:00.1
    91400000-917fffff : 0000:04:00.1
      91400000-917fffff : igb
    91800000-91bfffff : 0000:04:00.0
    91c00000-91ffffff : 0000:04:00.0
      91c00000-91ffffff : igb
    92000000-9201ffff : 0000:04:00.1
      92000000-9201ffff : igb
    92020000-9203ffff : 0000:04:00.0
      92020000-9203ffff : igb
    92040000-92043fff : 0000:04:00.1
      92040000-92043fff : igb
    92044000-92047fff : 0000:04:00.0
      92044000-92047fff : igb
    92048000-92067fff : 0000:04:00.0
      92048000-9204bfff : 0000:04:10.0
      9204c000-9204ffff : 0000:04:10.2
      92050000-92053fff : 0000:04:10.4
      92054000-92057fff : 0000:04:10.6
    92068000-92087fff : 0000:04:00.0
      92068000-9206bfff : 0000:04:10.0
      9206c000-9206ffff : 0000:04:10.2
      92070000-92073fff : 0000:04:10.4
      92074000-92077fff : 0000:04:10.6
    92088000-920a7fff : 0000:04:00.1
      92088000-9208bfff : 0000:04:10.1
      9208c000-9208ffff : 0000:04:10.3
      92090000-92093fff : 0000:04:10.5
      92094000-92097fff : 0000:04:10.7
    920a8000-920c7fff : 0000:04:00.1
      920a8000-920abfff : 0000:04:10.1
      920ac000-920affff : 0000:04:10.3
      920b0000-920b3fff : 0000:04:10.5
      920b4000-920b7fff : 0000:04:10.7
  92100000-921fffff : 0000:00:0e.0
  92200000-922fffff : PCI Bus 0000:03
    92200000-922001ff : 0000:03:00.0
      92200000-922001ff : ahci
  92300000-923fffff : PCI Bus 0000:02
    92300000-92303fff : 0000:02:00.0
      92300000-92303fff : iwlwifi
  92400000-924fffff : PCI Bus 0000:01
    92400000-92403fff : 0000:01:00.0
    92404000-92404fff : 0000:01:00.0
      92404000-92404fff : r8169
  92500000-9250ffff : 0000:00:15.0
    92500000-9250ffff : xhci-hcd
      92508070-9250846f : intel_xhci_usb_sw
  92510000-92513fff : 0000:00:0e.0
  92514000-92515fff : 0000:00:12.0
    92514000-92515fff : ahci
  92516000-925160ff : 0000:00:1f.1
  92517000-925177ff : 0000:00:12.0
    92517000-925177ff : ahci
  92518000-925180ff : 0000:00:12.0
    92518000-925180ff : ahci
  9251b000-9251bfff : 0000:00:0f.0
    9251b000-9251bfff : mei_me
d0000000-d0ffffff : Reserved
  d0c00000-d0c00653 : INT3452:03
    d0c00000-d0c00653 : INT3452:03 INT3452:03
  d0c40000-d0c40763 : INT3452:01
    d0c40000-d0c40763 : INT3452:01 INT3452:01
  d0c50000-d0c5076b : INT3452:00
    d0c50000-d0c5076b : INT3452:00 INT3452:00
  d0c70000-d0c70673 : INT3452:02
    d0c70000-d0c70673 : INT3452:02 INT3452:02
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
  e0000000-efffffff : Reserved
    e0000000-efffffff : PCI Bus 0000:00
      e0000000-efffffff : pnp 00:03
fe042000-fe044fff : Reserved
fe900000-fe902fff : Reserved
fea00000-feafffff : pnp 00:03
fec00000-fec00fff : Reserved
  fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
  fed00000-fed003ff : PNP0103:00
fed01000-fed01fff : intel-spi
  fed01000-fed01fff : Reserved
    fed01000-fed01fff : pnp 00:03
fed03000-fed03fff : pnp 00:03
fed06000-fed06fff : pnp 00:03
fed08000-fed09fff : pnp 00:03
fed1c000-fed1cfff : pnp 00:03
fed64000-fed64fff : dmar0
fed65000-fed65fff : dmar1
fed80000-fedbffff : pnp 00:03
fee00000-fee00fff : Local APIC
  fee00000-fee00fff : Reserved
ff800000-ffffffff : Reserved
100000000-27fffffff : System RAM
  1ed400000-1ee402606 : Kernel code
  1ee600000-1eefebfff : Kernel rodata
  1ef000000-1ef43e23f : Kernel data
  1ef742000-1efbfffff : Kernel bss
root@servum-pve:~# cat /proc/iomem | grep 6d9d7018
root@servum-pve:~# cat /proc/iomem | grep '6d9d7018'
root@servum-pve:~# cat /proc/iomem | grep 6d9d7
root@servum-pve:~# cat /proc/iomem | grep 6d9
6d5ca000-6d9bdfff : System RAM
6d9be000-6d9dcfff : ACPI Non-volatile Storage
6d9dd000-6dde0fff : Reserved
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!