Boot Hangs with vfio Problem trying Skylake Passthrough

[vivian]

Hi, I have tried weeks with the integrated GPU passthrough, but I got an error repeatedly on vfio-pci assignment.
I've followed
https://forum.proxmox.com/threads/gpu-passthrough-of-cpu-integrated-graphics.37897/ and
https://pve.proxmox.com/wiki/Pci_passthrough

Here is my configuration.
in BIOS, VT-d and GFX enabled

pve version

proxmox-ve: 5.1-32 (running kernel: 4.13.13-2-pve)
pve-manager: 5.1-41 (running version: 5.1-41/0b958203)
pve-kernel-4.13.13-2-pve: 4.13.13-32
libpve-http-server-perl: 2.0-8
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-19
qemu-server: 5.0-18
pve-firmware: 2.0-3
libpve-common-perl: 5.0-25
libpve-guest-common-perl: 2.0-14
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-17
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-3
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-5
pve-container: 2.0-18
pve-firewall: 3.0-5
pve-ha-manager: 2.0-4
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.1-2
lxcfs: 2.0.8-1
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on video=efifb:off"

/etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd

/etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
blacklist i915

After reboot pve,

root@pve:~#  lspci -nkvs 00:02
root@pve:~#  lspci -nkvs 00:02
00:02.0 0300: 8086:1912 (rev 06) (prog-if 00 [VGA controller])
        Subsystem: 15d9:0894
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at de000000 (64-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        I/O ports at f000 [size=64]
        [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Kernel modules: i915


root@pve:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000008B4CF4B0 0000A8 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.031394] DMAR: Host address width 39
[    0.031478] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.031568] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.031675] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.031763] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.031867] DMAR: RMRR base: 0x0000008af57000 end: 0x0000008af76fff
[    0.031955] DMAR: RMRR base: 0x0000008d000000 end: 0x0000008f7fffff
[    0.032005] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.032094] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.032180] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.032180] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.033714] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.517870] DMAR: No ATSR found
[    0.518316] DMAR: dmar0: Using Queued invalidation
[    0.518416] DMAR: dmar1: Using Queued invalidation
[    0.518721] DMAR: Setting RMRR:
[    0.518838] DMAR: Setting identity map for device 0000:00:02.0 [0x8d000000 - 0x8f7fffff]
[    0.518973] DMAR: Setting identity map for device 0000:00:14.0 [0x8af57000 - 0x8af76fff]
[    0.519090] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.519196] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.519403] DMAR: Intel(R) Virtualization Technology for Directed I/O


root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1f.3
/sys/kernel/iommu_groups/7/devices/0000:00:1f.4
/sys/kernel/iommu_groups/7/devices/0000:00:1f.2
/sys/kernel/iommu_groups/7/devices/0000:00:1f.0
/sys/kernel/iommu_groups/5/devices/0000:00:1c.4
/sys/kernel/iommu_groups/5/devices/0000:00:1c.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.3
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:1f.6
/sys/kernel/iommu_groups/6/devices/0000:00:1c.5
/sys/kernel/iommu_groups/4/devices/0000:00:17.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.2
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:03:00.0

Looks everything is fine, right?
But once I add the configuration of vfio, as following

/etc/modprobe.d/vfio.conf

options vfio-pci ids=8086:1912

and after reboot, the boot hang out at

vfio-pci 0000:00:02.0 vgaarb: change decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem

I'm really new to pve, can anyone figure out why? Any suggestion will be appriciated.

 

[Tahsin]

On /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on video=efifb:off,vesafb:off"

On /etc/modprobe.d/vfio.conf

options vfio-pci ids=8086:1912 disable_vga=1

The rest seems okay to me.

Edit: Dont forget to update grub and initramfs. Also, the screen will go blank after couple of seconds of Proxmox boot logo.

 

[vivian]

On /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on video=efifb:off,vesafb:off"

On /etc/modprobe.d/vfio.conf

options vfio-pci ids=8086:1912 disable_vga=1

The rest seems okay to me.

Edit: Dont forget to update grub and initramfs. Also, the screen will go blank after couple of seconds of Proxmox boot logo.

Thanks for you reply,Tahsin, I've tried with the new config but reboot hangs as usual:

The thing is boot always hangs out at somewhere if I configure the vfio.conf
I've seen there are case successful with skylake, but I cann't figure out WHY...

 

[Tahsin]

Thanks for you reply,Tahsin, I've tried with the new config but reboot hangs as usual:

The thing is boot always hangs out at somewhere if I configure the vfio.conf
I've seen there are case successful with skylake, but I cann't figure out WHY...

I REALLY apologize on replying so late. I didn't see the notification on the reply. Do you have any update on this or are you still having the issue?

 

[Republicus]

I don't remember the exact reason or error I received when trying passthrough on my system; though I am not passing an onboard video card. This may be worth trying..

My resolution was adding "iommu=pt" to /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

 

[vivian]

I REALLY apologize on replying so late. I didn't see the notification on the reply. Do you have any update on this or are you still having the issue?

Thank you Tahsin, I'm still having this issue, I have intended to try with install PVE in UEFI mode and create VM in UEFI mode too, but for some reason, I can't continue with the test. The thing I'm sure of is that the config in vfio.cfg file always gets the reboot hanged.