Intel gvt-g not working with 6.2

everwisher · May 14, 2020

I just finished install the newly released ProxmoxVE 6.2 due to a inadvertent resize of the / which had led to my computer crumbled down. After a series of settings to enable pci-e passthrough, a few issues appeared as below:

pveproxy webUI reports "No IOMMU detected, please activate it.See Documentation for further information." when trying to add some device to guest vms. As can be seen in the screenshot,
Code:
```
lspci -Dnnk
```
shows the devices I'm gonna attach to vms are already bound to vfio-pci. And they actually can be passed through to vms and work well.
Intel gvt-g doesn't work at all. The /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types folder doesn't show up after rebooting the machine and pveproxy webUI reports a weird IOMMU Group number '-1' for this device. The i915 driver has been loaded with the Intel UHD 630 graphics unit.

Here's the key criteria in the configuration files:
/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="vfio-pci.ids=1002:67ff,1002:aae0,1106:3483,8086:a170 nomodeset iommu=pt pcie_acs_override=downstream,multifunction i915_enable.gvt=1 kvm.ignore_msrs=1 irqpoll drm.debug=0 intel_iommu=on,igfx_off"

/etc/modules
kvmgt vfio vfio_mdev vfio_pci vfio_iommu_type1 vfio_virqfd

/etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67ff,1002:aae0,1106:3483,8086:a170
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm_intel nested=1
options kvm ignore_msrs=Y

Anyone please help me out of this issue?
Thanks in advance.

UPDATE on Jun.1, 2020
After a series of attempts, I finally worked out that the issue was associated with the kernel parameter: nomodeset (or i915.modeset=0 as equivalent for Intel UHD)

t.lamprecht · May 14, 2020

everwisher said:
options vfio_iommu_type1 allow_unsafe_interrupts=1

This is the "issue", see the following thread: https://forum.proxmox.com/threads/i...nabled-still-error-message.67341/#post-302549

everwisher · May 14, 2020

t.lamprecht said:
This is the "issue", see the following thread: https://forum.proxmox.com/threads/i...nabled-still-error-message.67341/#post-302549

Thank you for replying. I did as told from the thread to transfer "allow_unsafe_interrupts=1" from mod option to grub commandline, and it is effective now as cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts changed from "N" to "Y".

But the non-effectiveness of gvt-g still persists along with the "No IOMMU detected, please activate it.See Documentation for further information." prompt in pveproxy webUI.

dcsapak · May 22, 2020

everwisher said:
But the non-effectiveness of gvt-g still persists along with the "No IOMMU detected, please activate it.See Documentation for further information." prompt in pveproxy webUI.

the prompt on the gui is simply because the first device has no iommu group (you can ignore this if you passthrough another device)

the reason why this happens is because of:

'intel_iommu=on,igfx_off' in you cmdline

i guess gvt-g does not work with that option, just set 'intel_iommu=on'

mushion22 · May 30, 2020

Hi there,

Have been trying to set this up for the first time. Am also on 6.2-4 but have not tried on previous versions.

The mdevs are not set up when only using iommu_on. This is on a Xeon E3-1245v6, C236 Chipset, Supermicro X11SSH-F board.

I ran into the same issue with IOMMU working but the mdevs not showing up.

Seems that switching them around worked -

change

intel_iommu=on,igfx_off

to

intel_iommu=igfx_off,on

However now I am finding my whole host locks up after trying to use one of the mdevs on a Win10 guest. I might be doing something else wrong though!

mushion22 · May 31, 2020

Slight aside - I've had a bunch of trouble with my LSI RAID card which I pass through to a FreeNAS VM after the upgrade from 5.4 to 6.2. The card seems to get reset as soon as any I/O starts happening to the drives, which sends ZFS into a grump and managed to cause some corruption. It's working fine if I don't pass through and mount the ZFS volume on the host. Have had it doing a scrub for several hours at ~8Gbps while mounted on the host and no trouble.

You might want to test your other pass-throughs thoroughly.

Will probably start another thread once I've had a change to test a bit more.

bolzerrr · Jun 1, 2020

Hi,

would you mind to share the configuration it finally made it work?

Jovan Yeo · Jun 3, 2020

i somehow got it to work. But would like to know actually the background of this technology?

my question is, do I expect a display when I connected a monitor to the onboard video output? or basically, just to have a better resolution?

bolzerrr · Jun 3, 2020

My intension would be the ability to use the /dev/dri device but i still can`t get it working.
I currently use:

/etc/default/grub

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on video=efifb:off,vesafb=off pcie_acs_override=downstream kvm.ignore_msrs=0 i915_enable.gvt=1  irqpoll drm.debug=0 nomodeset "

/etc/modprobe.d/vfio.conf

Code:

options vfio-pci ids=8086:3185
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm_intel nested=1
options kvm ignore_msrs=Y

But no device in
ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

doesn`t exsists.

lspci -Dnnk

Code:

0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3185] (rev 03)
        Subsystem: ASRock Incorporation Device [1849:2212]
        Kernel driver in use: vfio-pci
        Kernel modules: i915

Jovan Yeo · Jun 4, 2020

hi ,

i got the same error as you.

Are you able to use the Intel GPU as a PCIE instead?

bolzerrr · Jun 4, 2020

Not 100% sure what you mean. I tried the "classic" passthrough but have issues also, find more details here: https://forum.proxmox.com/threads/p...igd-graphics-passthrough-for-ubuntu-18.47129/

Jovan Yeo · Jun 4, 2020

I got no issue with classic "PCIE passthrough".. my VM recognize my host built-in video card "Intel UHD 630"

I just couldn't get it work when I try to use it as mediated device.

Jovan Yeo · Jun 5, 2020

my resulkted

bolzerrr · Jun 5, 2020

Would you mind to share the exact steps you took to make this working?
I keep getting this error:

[ 4271.375653] DMAR: DRHD: handling fault status reg 2
[ 4271.375664] DMAR: [DMA Write] Request device [00:02.0] PASID ffffffff fault addr 0 [fault reason 02] Present bit in context entry is clear

BobMccapherey · Jun 12, 2020

bolzerrr said:
My intension would be the ability to use the /dev/dri device but i still can`t get it working.
I currently use:

/etc/default/grub

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on video=efifb:off,vesafb=off pcie_acs_override=downstream kvm.ignore_msrs=0 i915_enable.gvt=1 irqpoll drm.debug=0 nomodeset "

/etc/modprobe.d/vfio.conf

Code:

options vfio-pci ids=8086:3185 options vfio_iommu_type1 allow_unsafe_interrupts=1 options kvm_intel nested=1 options kvm ignore_msrs=Y

But no device in
ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

doesn`t exsists.

lspci -Dnnk

Code:

0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3185] (rev 03) Subsystem: ASRock Incorporation Device [1849:2212] Kernel driver in use: vfio-pci Kernel modules: i915

It's i915.enable_gvt=1 not i915_enable.gvt=1

Also you want kvm.ignore_msrs=1 - otherwise a whole plethora of apps in your VM could crash all of the GPU-enabled VMs on your host and cause a kernel panic.

bolzerrr · Jun 12, 2020

Thanks for your advice, i changed the /etc/default/grub to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on i915.modeset=0 kvm.ignore_msrs=1 i915.enable_gvt=1 video=efifb

ff,vesafb=off"

But still the folder doesn`t exsists: ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

I see some new errors in dmesg now:

[ 52.601829] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601841] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1c9 data 0x3
[ 52.601847] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601854] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601860] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a6 data 0x11
[ 52.601867] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601873] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 52.601881] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a7 data 0x11
[ 52.601887] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 56.899797] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x639
[ 56.899811] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x611
[ 56.899818] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x619
[ 56.899826] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x641
[ 57.756166] kvm_get_msr_common: 1 callbacks suppressed

BobMccapherey · Jun 12, 2020

bolzerrr said:
Thanks for your advice, i changed the /etc/default/grub to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on i915.modeset=0 kvm.ignore_msrs=1 i915.enable_gvt=1 video=efifbff,vesafb=off"

But still the folder doesn`t exsists: ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

I see some new errors in dmesg now:

[ 52.601829] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601841] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1c9 data 0x3
[ 52.601847] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601854] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601860] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a6 data 0x11
[ 52.601867] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601873] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 52.601881] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a7 data 0x11
[ 52.601887] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 56.899797] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x639
[ 56.899811] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x611
[ 56.899818] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x619
[ 56.899826] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x641
[ 57.756166] kvm_get_msr_common: 1 callbacks suppressed

Those errors are normal. There is a kernel flag to not print those but I can't remember the option name right now.

What modules are you loading? It seems like you're missing some.

Please output the contents of the files in /etc/modules.d and /etc/modules

You need to have these modules loaded in order for gvt to work

kvmgt
vfio-iommu-type1
vfio-mdev
vfio-virqfd <-- not sure if this is necessary

Also I noticed in your options posted above that you have contradictory options set in /etc/default/grub and /etc/modprobe.d/vfio.conf (the ignore_msrs options). Where did you obtain your initial set of flags?

bolzerrr · Jun 14, 2020

Here is the output of /etc/modules
kvmgt
vfio
vfio_mdev
vfio_iommu_type1
vfio_pci
vfio_virqfd

The inital options of /etc/default/grub and /etc/modprobe.d/vfio.conf are mostly from this thread. I cleaned /etc/modprobe.d/vfio.conf up and moved into /etc/default/grub to have a single source of truth:
GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on i915.modeset=0 kvm.ignore_msrs=1 i915.enable_gvt=1 video=efifb

ff,vesafb=off vfio_iommu_type1.allow_unsafe_interrupts=1"

Only leftover in /etc/modprobe.d/vfio.conf:
options vfio-pci ids=8086:3185

I did a "update-initramfs -u -k all" and reboot. But still no "mdev_supported_types" available.

Here some logs:

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.009130] ACPI: DMAR 0x000000006D6AF0A0 0000A8 (v01 INTEL GLK-SOC 00000003 BRXT 0100000D)
[ 0.218108] DMAR: IOMMU enabled
[ 0.440233] DMAR: Host address width 39
[ 0.440235] DMAR: DRHD base: 0x000000fed64000 flags: 0x0
[ 0.440245] DMAR: dmar0: reg_base_addr fed64000 ver 1:0 cap 1c0000c40660462 ecap 9e2ff0505e
[ 0.440247] DMAR: DRHD base: 0x000000fed65000 flags: 0x1
[ 0.440256] DMAR: dmar1: reg_base_addr fed65000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.440259] DMAR: RMRR base: 0x0000006d5ef000 end: 0x0000006d60efff
[ 0.440261] DMAR: RMRR base: 0x0000006f800000 end: 0x0000007fffffff
[ 0.440264] DMAR-IR: IOAPIC id 1 under DRHD base 0xfed65000 IOMMU 1
[ 0.440266] DMAR-IR: HPET id 0 under DRHD base 0xfed65000
[ 0.440268] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.442189] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.868373] DMAR: No ATSR found
[ 1.868446] DMAR: dmar0: Using Queued invalidation
[ 1.868451] DMAR: dmar1: Using Queued invalidation
[ 1.869074] DMAR: Intel(R) Virtualization Technology for Directed I/O

ll /sys/bus/pci/devices/0000\:02\:00.0/m
max_link_speed max_link_width modalias msi_bus msi_irqs/

lspci -nnk
00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3185] (rev 03)
Subsystem: ASRock Incorporation Device [1849:2212]
Kernel driver in use: vfio-pci
Kernel modules: i915

lsmod | grep vfio
vfio_pci 49152 1
vfio_virqfd 16384 1 vfio_pci
vfio_mdev 16384 0
mdev 24576 1 vfio_mdev
vfio_iommu_type1 32768 1
vfio 32768 6 vfio_mdev,vfio_iommu_type1,vfio_pci
irqbypass 16384 18 vfio_pci,kvm

bolzerrr · Jun 14, 2020

I just figured out it if not possible at all with my board: https://github.com/intel/gvt-linux/issues/124

lalakersfan29 · Jun 15, 2020

everwisher said:
UPDATE on Jun.1, 2020
After a series of attempts, I finally worked out that the issue was associated with the kernel parameter: nomodeset (or i915.modeset=0 as equivalent for Intel UHD)

Hi, do you have a guide on setting up GVT-G. I'm setting up a new install on 6.2, I'm having trouble.

Intel gvt-g not working with 6.2

Member

Proxmox Staff Member

Member

Proxmox Staff Member

New Member

New Member

Well-Known Member

Active Member

Attachments

Well-Known Member

Active Member

Well-Known Member

Active Member

Active Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

New Member

We value your privacy