Intel gvt-g not working with 6.2

E

everwisher

Member
Jun 23, 2019
15
0
6
39
I just finished install the newly released ProxmoxVE 6.2 due to a inadvertent resize of the / which had led to my computer crumbled down. After a series of settings to enable pci-e passthrough, a few issues appeared as below:
Screenshot_2020-05-14_15-11-52.png
  1. pveproxy webUI reports "No IOMMU detected, please activate it.See Documentation for further information." when trying to add some device to guest vms. As can be seen in the screenshot,
    Code: 
    lspci -Dnnk
    shows the devices I'm gonna attach to vms are already bound to vfio-pci. And they actually can be passed through to vms and work well.
  2. Intel gvt-g doesn't work at all. The /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types folder doesn't show up after rebooting the machine and pveproxy webUI reports a weird IOMMU Group number '-1' for this device. The i915 driver has been loaded with the Intel UHD 630 graphics unit.
Here's the key criteria in the configuration files:
/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="vfio-pci.ids=1002:67ff,1002:aae0,1106:3483,8086:a170 nomodeset iommu=pt pcie_acs_override=downstream,multifunction i915_enable.gvt=1 kvm.ignore_msrs=1 irqpoll drm.debug=0 intel_iommu=on,igfx_off"

/etc/modules
kvmgt vfio vfio_mdev vfio_pci vfio_iommu_type1 vfio_virqfd

/etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67ff,1002:aae0,1106:3483,8086:a170
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm_intel nested=1
options kvm ignore_msrs=Y

Anyone please help me out of this issue?
Thanks in advance.


UPDATE on Jun.1, 2020
After a series of attempts, I finally worked out that the issue was associated with the kernel parameter: nomodeset (or i915.modeset=0 as equivalent for Intel UHD)
 
Last edited:
  • Like
Reactions: Moayad
t.lamprecht said:
This is the "issue", see the following thread: https://forum.proxmox.com/threads/i...nabled-still-error-message.67341/#post-302549
Click to expand...

Thank you for replying. I did as told from the thread to transfer "allow_unsafe_interrupts=1" from mod option to grub commandline, and it is effective now as cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts changed from "N" to "Y".

But the non-effectiveness of gvt-g still persists along with the "No IOMMU detected, please activate it.See Documentation for further information." prompt in pveproxy webUI.
 
everwisher said:
But the non-effectiveness of gvt-g still persists along with the "No IOMMU detected, please activate it.See Documentation for further information." prompt in pveproxy webUI.
Click to expand...
the prompt on the gui is simply because the first device has no iommu group (you can ignore this if you passthrough another device)

the reason why this happens is because of:

'intel_iommu=on,igfx_off' in you cmdline

i guess gvt-g does not work with that option, just set 'intel_iommu=on'
 
Hi there,

Have been trying to set this up for the first time. Am also on 6.2-4 but have not tried on previous versions.

The mdevs are not set up when only using iommu_on. This is on a Xeon E3-1245v6, C236 Chipset, Supermicro X11SSH-F board.

I ran into the same issue with IOMMU working but the mdevs not showing up.

Seems that switching them around worked -

change

intel_iommu=on,igfx_off

to

intel_iommu=igfx_off,on

However now I am finding my whole host locks up after trying to use one of the mdevs on a Win10 guest. I might be doing something else wrong though!
 
Last edited:
Slight aside - I've had a bunch of trouble with my LSI RAID card which I pass through to a FreeNAS VM after the upgrade from 5.4 to 6.2. The card seems to get reset as soon as any I/O starts happening to the drives, which sends ZFS into a grump and managed to cause some corruption. It's working fine if I don't pass through and mount the ZFS volume on the host. Have had it doing a scrub for several hours at ~8Gbps while mounted on the host and no trouble.

You might want to test your other pass-throughs thoroughly.

Will probably start another thread once I've had a change to test a bit more.
 
i somehow got it to work. But would like to know actually the background of this technology?

my question is, do I expect a display when I connected a monitor to the onboard video output? or basically, just to have a better resolution?
 

Attachments

  • Intel_GVtg_resulted_01.JPG
    Intel_GVtg_resulted_01.JPG
    207.9 KB · Views: 97
My intension would be the ability to use the /dev/dri device but i still can`t get it working.
I currently use:

/etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on video=efifb:off,vesafb=off pcie_acs_override=downstream kvm.ignore_msrs=0 i915_enable.gvt=1  irqpoll drm.debug=0 nomodeset "

/etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=8086:3185
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm_intel nested=1
options kvm ignore_msrs=Y

But no device in
ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

doesn`t exsists.

lspci -Dnnk
Code: 
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3185] (rev 03)
        Subsystem: ASRock Incorporation Device [1849:2212]
        Kernel driver in use: vfio-pci
        Kernel modules: i915
 
I got no issue with classic "PCIE passthrough".. my VM recognize my host built-in video card "Intel UHD 630"

I just couldn't get it work when I try to use it as mediated device.
 
Would you mind to share the exact steps you took to make this working?
I keep getting this error:

[ 4271.375653] DMAR: DRHD: handling fault status reg 2
[ 4271.375664] DMAR: [DMA Write] Request device [00:02.0] PASID ffffffff fault addr 0 [fault reason 02] Present bit in context entry is clear
 
Last edited:
bolzerrr said:
My intension would be the ability to use the /dev/dri device but i still can`t get it working.
I currently use:

/etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on video=efifb:off,vesafb=off pcie_acs_override=downstream kvm.ignore_msrs=0 i915_enable.gvt=1  irqpoll drm.debug=0 nomodeset "

/etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=8086:3185
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm_intel nested=1
options kvm ignore_msrs=Y

But no device in
ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

doesn`t exsists.

lspci -Dnnk
Code: 
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3185] (rev 03)
        Subsystem: ASRock Incorporation Device [1849:2212]
        Kernel driver in use: vfio-pci
        Kernel modules: i915
Click to expand...

It's i915.enable_gvt=1 not i915_enable.gvt=1

Also you want kvm.ignore_msrs=1 - otherwise a whole plethora of apps in your VM could crash all of the GPU-enabled VMs on your host and cause a kernel panic.
 
Thanks for your advice, i changed the /etc/default/grub to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on i915.modeset=0 kvm.ignore_msrs=1 i915.enable_gvt=1 video=efifb:off,vesafb=off"

But still the folder doesn`t exsists: ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

I see some new errors in dmesg now:

[ 52.601829] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601841] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1c9 data 0x3
[ 52.601847] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601854] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601860] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a6 data 0x11
[ 52.601867] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601873] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 52.601881] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a7 data 0x11
[ 52.601887] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 56.899797] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x639
[ 56.899811] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x611
[ 56.899818] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x619
[ 56.899826] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x641
[ 57.756166] kvm_get_msr_common: 1 callbacks suppressed
 
bolzerrr said:
Thanks for your advice, i changed the /etc/default/grub to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on i915.modeset=0 kvm.ignore_msrs=1 i915.enable_gvt=1 video=efifb:eek:ff,vesafb=off"

But still the folder doesn`t exsists: ls /sys/bus/pci/devices/0000\:00\:02.0/mdev_supported_types

I see some new errors in dmesg now:

[ 52.601829] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601841] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1c9 data 0x3
[ 52.601847] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1c9
[ 52.601854] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601860] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a6 data 0x11
[ 52.601867] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a6
[ 52.601873] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 52.601881] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61608 ignored wrmsr: 0x1a7 data 0x11
[ 52.601887] kvm [1817]: vcpu0, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x1a7
[ 56.899797] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x639
[ 56.899811] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x611
[ 56.899818] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x619
[ 56.899826] kvm [1817]: vcpu1, guest rIP: 0xffffffff9dc61643 ignored rdmsr: 0x641
[ 57.756166] kvm_get_msr_common: 1 callbacks suppressed
Click to expand...

Those errors are normal. There is a kernel flag to not print those but I can't remember the option name right now.

What modules are you loading? It seems like you're missing some.

Please output the contents of the files in /etc/modules.d and /etc/modules

You need to have these modules loaded in order for gvt to work
  • kvmgt
  • vfio-iommu-type1
  • vfio-mdev
  • vfio-virqfd <-- not sure if this is necessary

Also I noticed in your options posted above that you have contradictory options set in /etc/default/grub and /etc/modprobe.d/vfio.conf (the ignore_msrs options). Where did you obtain your initial set of flags?
 
Here is the output of /etc/modules
kvmgt
vfio
vfio_mdev
vfio_iommu_type1
vfio_pci
vfio_virqfd

The inital options of /etc/default/grub and /etc/modprobe.d/vfio.conf are mostly from this thread. I cleaned /etc/modprobe.d/vfio.conf up and moved into /etc/default/grub to have a single source of truth:
GRUB_CMDLINE_LINUX_DEFAULT="quiet i8042.nopnp iommu=pt intel_iommu=on i915.modeset=0 kvm.ignore_msrs=1 i915.enable_gvt=1 video=efifb:off,vesafb=off vfio_iommu_type1.allow_unsafe_interrupts=1"

Only leftover in /etc/modprobe.d/vfio.conf:
options vfio-pci ids=8086:3185

I did a "update-initramfs -u -k all" and reboot. But still no "mdev_supported_types" available.

Here some logs:

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.009130] ACPI: DMAR 0x000000006D6AF0A0 0000A8 (v01 INTEL GLK-SOC 00000003 BRXT 0100000D)
[ 0.218108] DMAR: IOMMU enabled
[ 0.440233] DMAR: Host address width 39
[ 0.440235] DMAR: DRHD base: 0x000000fed64000 flags: 0x0
[ 0.440245] DMAR: dmar0: reg_base_addr fed64000 ver 1:0 cap 1c0000c40660462 ecap 9e2ff0505e
[ 0.440247] DMAR: DRHD base: 0x000000fed65000 flags: 0x1
[ 0.440256] DMAR: dmar1: reg_base_addr fed65000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.440259] DMAR: RMRR base: 0x0000006d5ef000 end: 0x0000006d60efff
[ 0.440261] DMAR: RMRR base: 0x0000006f800000 end: 0x0000007fffffff
[ 0.440264] DMAR-IR: IOAPIC id 1 under DRHD base 0xfed65000 IOMMU 1
[ 0.440266] DMAR-IR: HPET id 0 under DRHD base 0xfed65000
[ 0.440268] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.442189] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.868373] DMAR: No ATSR found
[ 1.868446] DMAR: dmar0: Using Queued invalidation
[ 1.868451] DMAR: dmar1: Using Queued invalidation
[ 1.869074] DMAR: Intel(R) Virtualization Technology for Directed I/O

ll /sys/bus/pci/devices/0000\:02\:00.0/m
max_link_speed max_link_width modalias msi_bus msi_irqs/

lspci -nnk
00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3185] (rev 03)
Subsystem: ASRock Incorporation Device [1849:2212]
Kernel driver in use: vfio-pci
Kernel modules: i915

lsmod | grep vfio
vfio_pci 49152 1
vfio_virqfd 16384 1 vfio_pci
vfio_mdev 16384 0
mdev 24576 1 vfio_mdev
vfio_iommu_type1 32768 1
vfio 32768 6 vfio_mdev,vfio_iommu_type1,vfio_pci
irqbypass 16384 18 vfio_pci,kvm
Click to expand...
 
everwisher said:
UPDATE on Jun.1, 2020
After a series of attempts, I finally worked out that the issue was associated with the kernel parameter: nomodeset (or i915.modeset=0 as equivalent for Intel UHD)
Click to expand...

Hi, do you have a guide on setting up GVT-G. I'm setting up a new install on 6.2, I'm having trouble.
 
  • Like
Reactions: lixaotec
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom