Alder Lake GVT-d integrated graphics passthrough

[jimjam]

I don’t blame you guys for doubting. When I had this setup originally it looked as though it was working, and then days later I would be upset by my server crashing. Look back at my posts in this thread, you can see the ups and downs I had.

That being said, my server has yet to crash with this setup (since December 27th 2022). I will report back on its status from time to time.

I haven’t heard back from anyone that has actually taken the time to compile and install the gmmlib, vaapi, and media driver repos. That could very well be the issue as those repos have specific fixes for the issues you guys are running into.

Some cherry picked changes in the media driver repo from the last few releases:

• Enhanced HEVC decoding picture & slice level parameters error handling
• Enabled Memory Decompression for ADLS and ADLN.
• Enhanced HEVC & AVC decode error handling robustness.
• Enabled HDR10 and HVS support
• Enabled encoding support including AV1/VP9/HEVC/AVC/JPEG codec
• Added Video Processing kernel source code
• Added ADL-N platform support

media-driver, libVA and gmmlib all go hand in hand. LibVA and gmmlib both have a decent number of changes but they seem to be at a different level than media-driver and would take more of my time to really understand.

I recommend following through the readme on each repo if you care to get this all working.

 

[Sapd]

I haven’t heard back from anyone that has actually taken the time to compile and install the gmmlib, vaapi, and media driver repos.

I did actually half a month ago. Compiled the whole latest user space stack and even the i915 kernel tip and verified that I really actually use them (even verified together with a jellyfin ffmpeg dev). Same results.

The only common thing with another guy with the 13th gen and same problem was, that we both used Gigabyte mainboard. Which do you have?

 

[jimjam]

I did actually half a month ago. Compiled the whole latest user space stack and even the i915 kernel tip and verified that I really actually use them (even verified together with a jellyfin ffmpeg dev). Same results.

The only common thing with another guy with the 13th gen and same problem was, that we both used Gigabyte mainboard. Which do you have?

Man, there sure are a lot of variables to consider. I wonder if that kernel you used was causing interference?

I have an Asus Z690-P motherboard.

 

[Sapd]

Man, there sure are a lot of variables to consider. I wonder if that kernel you used was causing interference?

I also tried multiple kernels. The drm-tip kernel, then a newer default linux kernel 6.x and two older 5.x kernels (also similar to your version).
There is really a lot of time to waste trying to find out whats the curlprit.
Actually it was suggested by jellyfin that I make a bug report in the i915 repo on freedesktop, but gathering all information again, turning on debug logs, also takes again more time and I lost a bit of motivation.

I have an Asus Z690-P motherboard.

Maybe its the reason; but its really just pure guessing.

 

[nyanmisaka]

I did actually half a month ago. Compiled the whole latest user space stack and even the i915 kernel tip and verified that I really actually use them (even verified together with a jellyfin ffmpeg dev). Same results.

The only common thing with another guy with the 13th gen and same problem was, that we both used Gigabyte mainboard. Which do you have?

That's probably me because I'm the only one coding for ffmpeg in Jellyfin team.

Can you give this jellyfin-ffmpeg version a try? You can download the deb package from artifacts and install using apt or dpkg.

There had an i915 KMD issue range from Linux 5.18 to 6.0.17/6.1.3 that may reset the i915 driver in OpenCL based tone-mapping. So make sure you are using Linux 6.2+ and drm-tip kernels.

And on the UMD side, there's a possible fix in media-driver(iHD) for the vcs* timeout under heavy load. I applied the patch to the iHD driver that shipped with this jellyfin-ffmpeg 5.1.2-8 deb package.

https://github.com/intel/media-driver/issues/1506
https://github.com/intel/media-driver/issues/1571

If it doesn't work then I'm afraid you have to file an issue to media-driver or drm-intel.

 

[Sapd]

Can you give this jellyfin-ffmpeg version a try? You can download the deb package from artifacts and install using apt or dpkg.

Yep. Compiled latest drm-tip, used the .deb from the artefacts and verified that its getting used.

However, still crashes, just different error messages. One time I even received a kernel panic.

[  327.755726] Fence expiration time out i915-0000:06:10.0:ffmpeg<14405>:53c!
[  357.486538] i915 0000:06:10.0: [drm] GT0: GUC: No register capture node found for 0x1033 / 0xFECB411D
[  357.486782] i915 0000:06:10.0: [drm] GPU HANG: ecode 12:4:00000000, in ffmpeg [14405]

I think I cannot avoid posting an issue with additional debug logs.

Note for everyone testing it, that the faulty drivers seems to also corrupt the RAM of the respective virtual machine. This can lead to file corruption.

 

[nyanmisaka]

Yep. Compiled latest drm-tip, used the .deb from the artefacts and verified that its getting used.

However, still crashes, just different error messages. One time I even received a kernel panic.

[  327.755726] Fence expiration time out i915-0000:06:10.0:ffmpeg<14405>:53c!
[  357.486538] i915 0000:06:10.0: [drm] GT0: GUC: No register capture node found for 0x1033 / 0xFECB411D
[  357.486782] i915 0000:06:10.0: [drm] GPU HANG: ecode 12:4:00000000, in ffmpeg [14405]

I think I cannot avoid posting an issue with additional debug logs.

Note for everyone testing it, that the faulty drivers seems to also corrupt the RAM of the respective virtual machine. This can lead to file corruption.

Better to file an issue to Intel. I can't reproduce this on my Tiger Lake i7-1165G7 iGPU and Arc A380 dGPU.

 

[jimjam]

Just checking to show the success I am having with this setup. I have restarted the server just a couple times since I set this up. I have intentionally tried to not restart it. The Plex server is now a solid 30 days of uptime. This server gets used daily by at least 2 users if not more. Regularly transcoding both video and audio. Working great!

 

[Dougiebabe]

This helped get me working. I haven't tested Plex yet but this is the first time I have got the q35, UEFI and PCI Express settings to work and have the VM boot. I also set the display to None which I hadn't done before. CPU is Intel(R) Pentium(R) Gold 7505.

Same PVE revision and Kernel, not sure what the VM kernel is but it's Ubuntu 22.04.

Currently have Frigate running within a docker and using the QSV hardware acceleration and showing 1% usage on the iGPU

Hopefully get Plex installed in the next few days and see if it works. Last time I tried with the BIOS and i440fx it would work but crash after a couple of streams started and the VM would need a reboot to function again. Fingers crossed it works this time and I can move Plex off my old server which is doing CPU transcoding.

I forgot to report back, this didn't work for me in the end.

The VM worked and I was able to run Frigate inside the VM (although that did crash a few times which could have been GPU related as I was using HW Accel) but as soon as I had Plex running a couple of transcodes the GPU would freeze and I would have to restart the VM.

I have since moved away from hosting Frigate and Plex in a VM and used LXC and have both working flawlessly. Frigate hasn't crashed once since moving across and I managed 6 transcodes with no issues. I haven't actually moved my Plex database and data across yet nor moved my users over as I want to see if I can get it working in an Unprivileged container first but need to find the time to do so.

I assume it works OK in LXC as it's talking direct to the iGPU instead of a virtualized layer? That's a total guess though...

@jimjam Maybe it would work if I installed those driver repos you talk about?

I'll post back here with a guide once I have tested it in an Unprivileged container, if I can't get it working then I will post instructions on how to do with it a privileged container.

 

[jimjam]

I happened to build another i9 12900k machine in that past month. I found the time over that past few days to run another Ubuntu setup on it. This time on bare metal.
If you have a PCIE graphics card installed, the i915 drivers will not load. Took me a while to muster up the desire to crack open the case and pull the graphics card to see if i915 drivers would load. I didn't have to do anything else special and this machine was pumping out 8x 1080p transcodes barely breaking a sweat.






link to source if this image is too low quality: https://imgur.com/a/oIyRYcj

 

[mati]

I switched to using LXC for Plex and bypassing the entire IOMMU/passthrough process altogether. It works perfectly now with much less configuration. I really do suspect that IOMMU is to blame as that's really the only notable difference in my setup.

Can you please share the tutorial that you used to install Plex on LXC? Thanks!

 

[Dougiebabe]

It took a while but I finally got Plex (and Frigate) working in a Unprivileged LXC with HW Transcoding and NAS Shares for media.

I've done limited testing as I haven't fully migrated my users over to the new Plex install in the LXC but I have had 9 transcodes with no issues where as I couldn't even get 2 to be stable when running in the VM.

I can post the guide on the forum if people would find that useful but I have made a blog post for my own records which can be found here

If anyone does try it let me know how you get on!

 

[mati]

For me updating to 6.2.8 kernel solved GPU HANG errors.

 

[Sandbo]

The only special configuration options were passing through the correct PCIE device, OVMF Bios, Q35 machine type, and display none.
View attachment 45768

Edit: Follow up, this virtual machine has been running without a reboot since I made the post about it working. It is very stable and hardware transcoding is working very well.

Edit: updated the importance of having display being set to none.

Your journey has been an interesting read, thanks for sharing your progress!

Now if I may ask a question, were you able to generate video output to the HDMI with the above setup?
I am currently trying to set a VM with an Intel N100 iGPU and hope to have it working as a HTPC, so being able to 1. pass the iGPU and 2. output video+audio through HDMI will be important to me.

 

[etnicor]

Coming back to report the success I had last night. I let this sit in my bucket of "wait for developers to fix" for quite some time. Last night I mustered up the strength to want to try again. It seems as though I have got it to work.

Proxmox Host Setup:

I am not sure how much of this is necessary, but I am not going to go back and remove stuff to see if it works, this is what it is, and its staying that way

Updated to 7.3-4

Grub command (/etc/default/grub):
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init"

my "/etc/modprob.d/blacklist.conf" file has this in it:
blacklist igb blacklist i915

my "/etc/modprob.d/vfio.conf" file has this in it:
options vfio-pci ids=8086:150e,8086:4680 disable_vga=1

Running kernel 5.15.83-1-pve

Ubuntu 22.04 LTS Guest:
Kernel:
5.15.0-56-generic

Grub command (/etc/default/grub):
GRUB_CMDLINE_LINUX_DEFAULT="i915.force_probe=4680 i915.enable_gvt=1"

Built and installed these three repos, but they might not be necessary:

• https://github.com/intel/gmmlib
• https://askubuntu.com/questions/1028521/installing-libva-and-intel-vaapi-driver
• https://github.com/intel/media-driver

Another interesting thread: https://github.com/intel/media-driver/issues/1371

-----------------------------------------------------------------------------------------------------
After doing that, I transferred my plexmediaserver directory from the old plex machine to the new one, enabled hardware encoding in settings and started multiple streams for an extended period of time. No crashes, kernel panics or other instability. Just very little CPU usage, and minimal power consumption:

View attachment 44914

and 1 transcode on my phone at the same time (weird clouds at the bottom are the PIP transcode intro to a movie), also showing the servers total power consumption (it runs more than just plex):
View attachment 44915

Edit: Updated guest OS kernel and version.

Have some hardware with a raptorlake igpu and a builtin gpu in BMC(ASPEED) on motherboard. Does the kernel options disable all vga on proxmox host?
Would like to pass the intel igpu, but have the BMC gpu to continue to work.

Also why is igb network driver disabled or is that unrelated?

 

[etnicor]

Coming back to report the success I had last night. I let this sit in my bucket of "wait for developers to fix" for quite some time. Last night I mustered up the strength to want to try again. It seems as though I have got it to work.

Proxmox Host Setup:

I am not sure how much of this is necessary, but I am not going to go back and remove stuff to see if it works, this is what it is, and its staying that way

Updated to 7.3-4

Grub command (/etc/default/grub):
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init"

my "/etc/modprob.d/blacklist.conf" file has this in it:
blacklist igb blacklist i915

my "/etc/modprob.d/vfio.conf" file has this in it:
options vfio-pci ids=8086:150e,8086:4680 disable_vga=1

Running kernel 5.15.83-1-pve

Ubuntu 22.04 LTS Guest:
Kernel:
5.15.0-56-generic

Grub command (/etc/default/grub):
GRUB_CMDLINE_LINUX_DEFAULT="i915.force_probe=4680 i915.enable_gvt=1"

Built and installed these three repos, but they might not be necessary:

• https://github.com/intel/gmmlib
• https://askubuntu.com/questions/1028521/installing-libva-and-intel-vaapi-driver
• https://github.com/intel/media-driver

Another interesting thread: https://github.com/intel/media-driver/issues/1371

-----------------------------------------------------------------------------------------------------
After doing that, I transferred my plexmediaserver directory from the old plex machine to the new one, enabled hardware encoding in settings and started multiple streams for an extended period of time. No crashes, kernel panics or other instability. Just very little CPU usage, and minimal power consumption:

View attachment 44914

and 1 transcode on my phone at the same time (weird clouds at the bottom are the PIP transcode intro to a movie), also showing the servers total power consumption (it runs more than just plex):
View attachment 44915

Edit: Updated guest OS kernel and version.

Followed this but on my VM when booting when added pci device I get message bellow and boot hangs. Only difference is that I run proxmox with the 6.2 kerel.

drm failed to find vbios tables while booting

 

[vctgomes]

Hey guys, how did you exported it to container?

My Alder Lake GPU. It's working and enable on Host, but when I try the command vainfo on container, it fails.

 

[isthevao]

Your journey has been an interesting read, thanks for sharing your progress!

Now if I may ask a question, were you able to generate video output to the HDMI with the above setup?
I am currently trying to set a VM with an Intel N100 iGPU and hope to have it working as a HTPC, so being able to 1. pass the iGPU and 2. output video+audio through HDMI will be important to me.

Hello,
Were you able to output the video to the HDMI port? I tried everything I found online, but could not get it working.

 

[patch]

were you able to generate video output to the HDMI with the above setup?
I am currently trying to set a VM with an Intel N100 iGPU and hope to have it working as a HTPC

Were you able to output the video to the HDMI port?

Interested in this also.

Similar application. Use custom built "all in one router" consisting of Proxmox, pfsense, 3cx, Unifi application. Similar at 2 sites. Have spare unit for lab & recovery from failure. Like to make more active use of the "spares" as HTPC. So need HDMI out from a VM on the spare hardware running Proxmox. Plan to disable the iGPU pass through when updating Proxmox just in case I need console access.

 

[vctgomes]

Hello,
Were you able to output the video to the HDMI port? I tried everything I found online, but could not get it working.

I think once the IMMOU is setup and GPU is able to do passthrough, you can add PCI and check the option Primary GPU
I try it sometimes ago and and display stop showing Proxmox host Shell and it shows a black screen. Maybe if my GPU was installed with the necessary drivers in the VM, the screen would shown.