I figured that I would share my deployment notes/guide for how to passthrough mediated devices (e.g. Intel iGPU (I'm using a 6700K, so it theory, it should work for anything 6th to 11th generation. For 12th to 14th generation, which uses the Intel Iris Xe graphics rather than (U)HD <<number>> Graphics (e.g. HD 530 Graphics in my case), there are instructions available online for how to enable SR-IOV for the Intel Iris Xe series, which is actually better because you can share that iGPU between more VMs than with my mediated device).
But at least it's better than only being able to pass said HD 530 Graphics only to one VM at a time (where as a mediated device, I can run two VMs using it), if someone needs it or has a use case for this.
I also have a pair of 3090s as well, so I've done the research and the testing on my system to make it so that both work nicely with each other (so that you can pass through 3090s for gaming and/or locally hosted AI workloads) and also be able to use the HD 530 Graphics at the same time.
When I was trying to troubleshoot my more recent issues (which ended up having nothing to do with my setup), you can find some stuff for mediated devices and some stuff for Nvidia dGPU passthrough, but never in one spot.
So, I am sharing my deployment notes/guide with the community, so that it's "one-stop shop" for those that might find this useful.
Without further ado:
Install the base operating system, by following through the prompts.
Disable the enterprise repositories (if you don't have a paid subscription/license) and add the no-subscription respository via the Proxmox GUI.
Update your system:
# apt update && apt upgrade -y
# apt install -y nfs-kernel-server lm-sensors pixz build-essential pve-headers-$(uname -r) wget curl gpg make
# pveam update
Create your local pve group and users as well your system user (which is authenticated using the system authentication rather than PVE PAM authentication.
Update grub:
Edit the grub config:
# vi /etc/default/grub
Change this line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
To
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt i915.enable_gvt=1 systemd.unified_cgroup_hierarchy=0 disable_vga=1 vfio-pci.ids=10de:2204,10de:1aef video=vesafbff,efifbff initcall_blacklist=sysfb_init nofb pcie_acs_override=downstream"
Note: If you include nomodeset with the kernel boot parameters, the Intel HD 530 Graphics will not work, as a mediated device. So don't include that in the kernel boot parameters.
# update-grub; reboot
Check to make sure that IOMMU has successfully been enabled.
# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.007731] ACPI: DMAR 0x000000008EA3BFB8 000078 (v01 INTEL KBL 00000001 INTL 00000001)
[ 0.007773] ACPI: Reserving DMAR table memory at [mem 0x8ea3bfb8-0x8ea3c02f]
[ 0.090674] DMAR: IOMMU enabled
[ 0.220361] DMAR: Host address width 39
[ 0.220362] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.220365] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.220367] DMAR: RMRR base: 0x0000008d962000 end: 0x0000008d981fff
[ 0.220369] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0
[ 0.220370] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[ 0.220370] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.221718] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.440174] DMAR: [Firmware Bug]: RMRR entry for device 03:00.0 is broken - applying workaround
[ 0.440176] DMAR: No ATSR found
[ 0.440176] DMAR: No SATC found
[ 0.440178] DMAR: dmar0: Using Queued invalidation
[ 0.440333] DMAR: Intel(R) Virtualization Technology for Directed I/O
You can use this script that I found to check the IOMMU Groups:
# vi check.sh
#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
save,quit
# chmod +x check.sh
# ./check.sh
Edit the following files to prepare for the Nvidia GPU passthrough:
(you can find your Nvidia PCI IDs via
# vi /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2531,10de:228e disable_vga=1
# vi /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
# vi /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
# vi /etc/modprobe.d/pve-blacklist.conf
add
blacklist nvidia
blacklist nouveau
blacklist radeon
save,quit
# vi /etc/modprobe.d/blacklist.conf
blacklist nvidiafb
blacklist nvidia
blacklist nouveau
blacklist radeon
save,quit
# vi /etc/modules
# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
# Modules required for Intel GVT
kvmgt
exngt
vfio-mdev
save,quit
# update-initramfs -u -k all; update-grub; reboot
Check to make sure that everything worked properly.
If it didn't, review and/or repeat the steps for the relevant section again, to make sure that it was executed properly.
But at least it's better than only being able to pass said HD 530 Graphics only to one VM at a time (where as a mediated device, I can run two VMs using it), if someone needs it or has a use case for this.
I also have a pair of 3090s as well, so I've done the research and the testing on my system to make it so that both work nicely with each other (so that you can pass through 3090s for gaming and/or locally hosted AI workloads) and also be able to use the HD 530 Graphics at the same time.
When I was trying to troubleshoot my more recent issues (which ended up having nothing to do with my setup), you can find some stuff for mediated devices and some stuff for Nvidia dGPU passthrough, but never in one spot.
So, I am sharing my deployment notes/guide with the community, so that it's "one-stop shop" for those that might find this useful.
Without further ado:
Install the base operating system, by following through the prompts.
Disable the enterprise repositories (if you don't have a paid subscription/license) and add the no-subscription respository via the Proxmox GUI.
Update your system:
# apt update && apt upgrade -y
# apt install -y nfs-kernel-server lm-sensors pixz build-essential pve-headers-$(uname -r) wget curl gpg make
# pveam update
Create your local pve group and users as well your system user (which is authenticated using the system authentication rather than PVE PAM authentication.
Update grub:
Edit the grub config:
# vi /etc/default/grub
Change this line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
To
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt i915.enable_gvt=1 systemd.unified_cgroup_hierarchy=0 disable_vga=1 vfio-pci.ids=10de:2204,10de:1aef video=vesafbff,efifbff initcall_blacklist=sysfb_init nofb pcie_acs_override=downstream"
Note: If you include nomodeset with the kernel boot parameters, the Intel HD 530 Graphics will not work, as a mediated device. So don't include that in the kernel boot parameters.
# update-grub; reboot
Check to make sure that IOMMU has successfully been enabled.
# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.007731] ACPI: DMAR 0x000000008EA3BFB8 000078 (v01 INTEL KBL 00000001 INTL 00000001)
[ 0.007773] ACPI: Reserving DMAR table memory at [mem 0x8ea3bfb8-0x8ea3c02f]
[ 0.090674] DMAR: IOMMU enabled
[ 0.220361] DMAR: Host address width 39
[ 0.220362] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.220365] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.220367] DMAR: RMRR base: 0x0000008d962000 end: 0x0000008d981fff
[ 0.220369] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0
[ 0.220370] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[ 0.220370] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.221718] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.440174] DMAR: [Firmware Bug]: RMRR entry for device 03:00.0 is broken - applying workaround
[ 0.440176] DMAR: No ATSR found
[ 0.440176] DMAR: No SATC found
[ 0.440178] DMAR: dmar0: Using Queued invalidation
[ 0.440333] DMAR: Intel(R) Virtualization Technology for Directed I/O
You can use this script that I found to check the IOMMU Groups:
# vi check.sh
#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
save,quit
# chmod +x check.sh
# ./check.sh
Edit the following files to prepare for the Nvidia GPU passthrough:
(you can find your Nvidia PCI IDs via
lspci -nn | grep -i nvidia# vi /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2531,10de:228e disable_vga=1
# vi /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
# vi /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
# vi /etc/modprobe.d/pve-blacklist.conf
add
blacklist nvidia
blacklist nouveau
blacklist radeon
save,quit
# vi /etc/modprobe.d/blacklist.conf
blacklist nvidiafb
blacklist nvidia
blacklist nouveau
blacklist radeon
save,quit
# vi /etc/modules
# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
# Modules required for Intel GVT
kvmgt
exngt
vfio-mdev
save,quit
# update-initramfs -u -k all; update-grub; reboot
Check to make sure that everything worked properly.
If it didn't, review and/or repeat the steps for the relevant section again, to make sure that it was executed properly.