iGPU & dGPU passed through to different VMs?

[johnflorin]

Hey all,

I searched all over the internet for a solution to this particular issue, but could not find one documented anywhere, just that it's supposed to work.

Thus, I ask you, how can I get it to work?

My use case is that I have 2 VMs in Proxmox for which I want to dedicate a GPU each, since GPU slicing seems to be a bad idea ( https://blog.ktz.me/why-i-stopped-using-intel-gvt-g-on-proxmox/ ).

One is a Home Assistant OS VM for which I got the iGPU passed thru just fine with this string:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init video=simplefb:off video=vesafb:off video=efifb:off video=vesa:off disable_vga=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,snd_hda_intel,snd_hda_codec_hdmi,i915"

The other is a Windows VM for which I want to pass through a GeForce for Plex transcoding.

Now, from what I understand, the only thing I'm missing (that is not documented already in guides like this one - https://3os.org/infrastructure/prox...vm/#proxmox-configuration-for-gpu-passthrough ) is the appropriate configuration of grub so that a single parameter list can cover BOTH graphic chips.

Can you help me? What should I add and/or remove? One thing that I know I should add is my dGPU IDs with "vfio-pci.ids=10de:1cb2,10de:0fb9", but what about the rest of the parameters? Also, if I add these IDs, should I also add the iGPU ID?

Thanks in advance for any help!

 

[AndroGen]

This is a bit another topic i am dealing with right now, but the "Home Assistant OS VM" sounds familiar.
Hence the question: Why do you need a dedicated GPU for this system?
There is almost no information on the system console and it can be easily managed using "standard" Proxmox console capabilities.
Or I am missing something?

BTW, there is a possibility to format text using 3 times ` - then your code will be nicely readable with no Emoji in it.

 

[mr44er]

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init video=simplefbff video=vesafbff video=efifbff video=vesaff disable_vga=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,snd_hda_intel,snd_hda_codec_hdmi,i915"

This isn't a passthrough 'string' whatsoever, these are boot parameters for the kernel. They all can help when different problems pop up, but pls start with small steps, not with all at once. That's a guarantee for havoc.

Where you edited this, please change it back to GRUB_CMDLINE_LINUX_DEFAULT="quiet"

Do you have an Intel or AMD machine?

First you need to run proxmox-boot-tool status to see if you are booting legacy or UEFI.

On Intel (CPU, not GPU!) legacy you edit /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet" -> GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

On Intel UEFI you edit /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet -> root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt

On AMD (CPU, not GPU!) legacy and UEFI you don't need to edit the kernel parameters.

Now run proxmox-boot-tool refresh and reboot.

Now for any guiding from here, some infos would be nice. On which gpu does proxmox has output on boot?
Please give output of lspci | grep VGA

 

[johnflorin]

This is a bit another topic i am dealing with right now, but the "Home Assistant OS VM" sounds familiar.
Hence the question: Why do you need a dedicated GPU for this system?
There is almost no information on the system console and it can be easily managed using "standard" Proxmox console capabilities.
Or I am missing something?

BTW, there is a possibility to format text using 3 times ` - then your code will be nicely readable with no Emoji in it.

Thanks, edited the OP

As for why I need a GPU passed through to a Home Assistant OS VM, it's actually for a Frigate addon in Docker within the OS, which makes use of the GPU for video stream decoding, thus not murdering the CPU for that task.

This isn't a passthrough 'string' whatsoever, these are boot parameters for the kernel. They all can help when different problems pop up, but pls start with small steps, not with all at once. That's a guarantee for havoc.

I used those params because they were mentioned in the guide at https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/#introduction

First you need to run proxmox-boot-tool status to see if you are booting legacy or UEFI.

Running that outputted this:

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. E: /etc/kernel/proxmox-boot-uuids does not exist.

As far as I know, though, I am booting legacy.

lspci | grep VGA outputs this:

00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)

On which gpu does proxmox has output on boot?

I have output on the Nvidia dGPU (in the BIOS I enabled 'multi-monitor via DGPU & iGPU, otherwise it would disable my iGPU with the dGPU inserted).

I have an Intel CPU, BTW.

Thanks for the input, hope this info helps!

 

[mr44er]

in the guide at https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/#introduction

I see, but I don't understand why. This tutorial looks good besides that.

On Intel (CPU, not GPU!) legacy you edit /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet" -> GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

These should be the only options you need.

 

[johnflorin]

So with these 2 options, I can passthrough both the iGPU and the Nvidia dGPU?

One thing mentioned in the tutorial was that the reason for blacklisting the Nvidia drivers was so that the Proxmox host console doesn't use it and it can be used in the VM, is that incorrect?

 

[mr44er]

So with these 2 options, I can passthrough both the iGPU and the Nvidia dGPU?

Yes, that should work.

One thing mentioned in the tutorial was that the reason for blacklisting the Nvidia drivers was so that the Proxmox host console doesn't use it and it can be used in the VM, is that incorrect?

No, but this blacklisting is done automatically in newer proxmox-versions.
See nano /etc/modprobe.d/pve-blacklist.conf

 

[johnflorin]

Ah, excellent, thanks a lot for the info! Will attempt it when the family's asleep, so that I don't kill Home Assistant for a while

 

[johnflorin]

Thanks again, this worked!

I now have Plex decoding with Nvidia in Windows, but unfortunately now something happened to my PCI-E Coral TPU in Frigate, which no longer detects it...Home Assistant OS shows it as being passed through in its hardware info, but Frigate (which saw it previously, with the same config) now doesn't see it anymore, giving the "No EdgeTPU was detected" log message.

So, it's back to the drawing board with this new problem, but super-happy to have both GPUs passed through properly to their respective VMs