Cannot get PCIe Passthrough work

T

testcb00

Member
Jun 4, 2020
20
0
21
28
Hi everyone,
After my installation of Proxmox VE 6.2 on my Lenovo T450s, I started to work on Disk Passthrough and PCIe Passthrough.
I follow this guide and successfully passthrough my Windows 10 Bare Metal installed SSD.
https://pve.proxmox.com/wiki/Physical_disk_to_kvm

For the PCIe Passthrough, I follow these guides, but I cannot get most of the PCIe device work. Although I have Vt-d enabled in BIOS and see the IOMMU groups, the Web Console keep say the error " No IOMMU detected, please activate it. ". I found that the Intel HD Graphics 5500 was set to IOMMU Group [-1]. However, I still able to passthrough other device, but only the Realtek RTS5227 PCI Express Card Reader (SD Card Reader) work under this condition.

To check the compatibility issue, I installed Proxmox VE 5.4 and upgrade to 5.4-15. In 5.4-15, the IOMMU error does not exist, and Intel HD Graphics 5500 was set to IOMMU Group [1]. However, I still cannot get other PCIe Device work in Passthrough, only SD Card reader work.
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/

Here is my log in Proxmox 5.4-15.
root@pve:~# lsmod | grep vfio
vfio_pci 45056 1
vfio_virqfd 16384 1 vfio_pci
irqbypass 16384 3 vfio_pci,kvm
vfio_iommu_type1 24576 1
vfio 28672 5 vfio_iommu_type1,vfio_pci
root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] ACPI: DMAR 0x000000009CDD3000 0000A8 (v01 LENOVO TP-JB 00001370 PTEC 00000002)
[ 0.000000] DMAR: IOMMU enabled
[ 0.000000] DMAR: Host address width 39
[ 0.000000] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.000000] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e1ff0505e
[ 0.000000] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.000000] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[ 0.000000] DMAR: RMRR base: 0x0000009a68e000 end: 0x0000009a6a4fff
[ 0.000000] DMAR: RMRR base: 0x0000009d800000 end: 0x0000009fffffff
[ 0.000000] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.000000] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.780917] DMAR: No ATSR found
[ 0.780964] DMAR: dmar0: Using Queued invalidation
[ 0.780971] DMAR: dmar1: Using Queued invalidation
[ 0.781077] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 0.781084] DMAR: Hardware identity mapping for device 0000:00:02.0
[ 0.781086] DMAR: Hardware identity mapping for device 0000:00:03.0
[ 0.781087] DMAR: Hardware identity mapping for device 0000:00:16.0
[ 0.781088] DMAR: Hardware identity mapping for device 0000:00:19.0
[ 0.781089] DMAR: Hardware identity mapping for device 0000:00:1b.0
[ 0.781090] DMAR: Hardware identity mapping for device 0000:00:1c.0
[ 0.781092] DMAR: Hardware identity mapping for device 0000:00:1c.1
[ 0.781093] DMAR: Hardware identity mapping for device 0000:00:1c.4
[ 0.781094] DMAR: Hardware identity mapping for device 0000:00:1d.0
[ 0.781096] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 0.781097] DMAR: Hardware identity mapping for device 0000:00:1f.2
[ 0.781098] DMAR: Hardware identity mapping for device 0000:00:1f.3
[ 0.781099] DMAR: Hardware identity mapping for device 0000:00:1f.6
[ 0.781106] DMAR: Hardware identity mapping for device 0000:02:00.0
[ 0.781112] DMAR: Hardware identity mapping for device 0000:03:00.0
[ 0.781119] DMAR: Hardware identity mapping for device 0000:04:00.0
[ 0.781120] DMAR: Setting RMRR:
[ 0.781122] DMAR: Ignoring identity map for HW passthrough device 0000:00:02.0 [0x9d800000 - 0x9fffffff]
[ 0.781123] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0x9a68e000 - 0x9a6a4fff]
[ 0.781124] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.781125] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.781163] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 0.856899] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
[ 0.856955] DMAR: Setting identity map for device 0000:00:1d.0 [0x9a68e000 - 0x9a6a4fff]
root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.1
/sys/kernel/iommu_groups/5/devices/0000:00:1b.0
/sys/kernel/iommu_groups/13/devices/0000:04:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/11/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.4
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/4/devices/0000:00:19.0
/sys/kernel/iommu_groups/12/devices/0000:03:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
/sys/kernel/iommu_groups/10/devices/0000:00:1f.6
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1d.0
root@pve:~# lspci -nnk

00:00.0 Host bridge [0600]: Intel Corporation Broadwell-U Host Bridge -OPI [8086:1604] (rev 09)

Subsystem: Lenovo Broadwell-U Host Bridge -OPI [17aa:5034]

Kernel driver in use: bdw_uncore

00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 5500 [8086:1616] (rev 09)

Subsystem: Lenovo HD Graphics 5500 [17aa:5037]

Kernel driver in use: vfio-pci

Kernel modules: i915

00:03.0 Audio device [0403]: Intel Corporation Broadwell-U Audio Controller [8086:160c] (rev 09)

Subsystem: Lenovo Broadwell-U Audio Controller [17aa:5034]

Kernel driver in use: vfio-pci

Kernel modules: snd_hda_intel

00:16.0 Communication controller [0780]: Intel Corporation Wildcat Point-LP MEI Controller #1 [8086:9cba] (rev 03)

Subsystem: Lenovo Wildcat Point-LP MEI Controller [17aa:5034]

Kernel driver in use: mei_me

Kernel modules: mei_me

00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (3) I218-V [8086:15a3] (rev 03)

Subsystem: Lenovo Ethernet Connection (3) I218-V [17aa:2227]

Kernel driver in use: e1000e

Kernel modules: e1000e

00:1b.0 Audio device [0403]: Intel Corporation Wildcat Point-LP High Definition Audio Controller [8086:9ca0] (rev 03)

Subsystem: Lenovo Wildcat Point-LP High Definition Audio Controller [17aa:5036]

Kernel modules: snd_hda_intel

00:1c.0 PCI bridge [0604]: Intel Corporation Wildcat Point-LP PCI Express Root Port #6 [8086:9c9a] (rev e3)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1c.1 PCI bridge [0604]: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 [8086:9c94] (rev e3)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1c.4 PCI bridge [0604]: Intel Corporation Wildcat Point-LP PCI Express Root Port #5 [8086:9c98] (rev e3)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1d.0 USB controller [0c03]: Intel Corporation Wildcat Point-LP USB EHCI Controller [8086:9ca6] (rev 03)

Subsystem: Lenovo Wildcat Point-LP USB EHCI Controller [17aa:5034]

Kernel driver in use: ehci-pci

00:1f.0 ISA bridge [0601]: Intel Corporation Wildcat Point-LP LPC Controller [8086:9cc3] (rev 03)

Subsystem: Lenovo Wildcat Point-LP LPC Controller [17aa:5034]

Kernel driver in use: lpc_ich

Kernel modules: lpc_ich

00:1f.2 SATA controller [0106]: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] [8086:9c83] (rev 03)

Subsystem: Lenovo Wildcat Point-LP SATA Controller [AHCI Mode] [17aa:5034]

Kernel driver in use: ahci

Kernel modules: ahci

00:1f.3 SMBus [0c05]: Intel Corporation Wildcat Point-LP SMBus Controller [8086:9ca2] (rev 03)

Subsystem: Lenovo Wildcat Point-LP SMBus Controller [17aa:5034]

Kernel driver in use: i801_smbus

Kernel modules: i2c_i801

00:1f.6 Signal processing controller [1180]: Intel Corporation Wildcat Point-LP Thermal Management Controller [8086:9ca4] (rev 03)

Subsystem: Lenovo Wildcat Point-LP Thermal Management Controller [17aa:5034]

Kernel driver in use: intel_pch_thermal

Kernel modules: intel_pch_thermal

02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5227 PCI Express Card Reader [10ec:5227] (rev 01)

Subsystem: Lenovo RTS5227 PCI Express Card Reader [17aa:5034]

Kernel driver in use: vfio-pci

Kernel modules: rtsx_pci

03:00.0 Network controller [0280]: Intel Corporation Wireless 7265 [8086:095b] (rev 59)

Subsystem: Intel Corporation Dual Band Wireless-AC 7265 [8086:5210]

Kernel driver in use: vfio-pci

Kernel modules: iwlwifi

04:00.0 3D controller [0302]: NVIDIA Corporation GM108M [GeForce 940M] [10de:1347] (rev a2)

Subsystem: Lenovo GM108M [GeForce 940M] [17aa:5037]

Kernel driver in use: vfio-pci

Kernel modules: nvidiafb, nouveau
 
Also, here is the file details and the Error Code Capture when I tried to Passthrough a Intel Wi-Fi M.2. Module. I rtried to uninstall the driver and reboot the VM but no luck.

1591602698394.png

root@pve:~# cat /etc/default/grub

# If you change this file, run 'update-grub' afterwards to update

# /boot/grub/grub.cfg.

# For full documentation of the options in this file, see:

# info -f grub -n 'Simple configuration'


GRUB_DEFAULT=0

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR="Proxmox Virtual Environment"

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

GRUB_CMDLINE_LINUX=""


# Disable os-prober, it might add menu entries for each guest

GRUB_DISABLE_OS_PROBER=true


# Uncomment to enable BadRAM filtering, modify to suit your needs

# This works with Linux (no patch required) and with any kernel that obtains

# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)

#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"


# Uncomment to disable graphical terminal (grub-pc only)

#GRUB_TERMINAL=console


# The resolution used on graphical terminal

# note that you can use only modes which your graphic card supports via VBE

# you can see them in real GRUB with the command `vbeinfo'

#GRUB_GFXMODE=640x480


# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux

#GRUB_DISABLE_LINUX_UUID=true


# Disable generation of recovery mode menu entries

GRUB_DISABLE_RECOVERY="true"


# Uncomment to get a beep at grub start

#GRUB_INIT_TUNE="480 440 1"

root@pve:~# cat /etc/modules

# /etc/modules: kernel modules to load at boot time.

#

# This file contains the names of kernel modules that should be loaded

# at boot time, one per line. Lines beginning with "#" are ignored.


vfio

vfio_iommu_type1

vfio_pci

vfio_virqfd

root@pve:~# cat /etc/modprobe.d/blacklist.conf

blacklist snd_hda_intel

blacklist snd_hda_codec_hdmi

blacklist i915

blacklist rtsx_pci

blacklist iwlwifi

blacklist nvidiafb

blacklist nouveau

root@pve:~# cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=8086:1616,8086:160c,8086:095b,10de:1347,10ec:5227 disable_vga=1

root@pve:~# cat /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1

root@pve:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

root@pve:~# cat /etc/pve/qemu-server/101.conf

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

bios: ovmf

boot: c

bootdisk: sata1

cores: 4

cpu: host,hidden=1,flags=+pcid

efidisk0: local-lvm:vm-101-disk-1,size=128K

hostpci0: 03:00,pcie=1

ide0: local-lvm:vm-101-disk-0,size=512M

ide2: none,media=cdrom

machine: q35

memory: 8192

name: W10

net0: e1000=96:83:C2:91:17:28,bridge=vmbr0,firewall=1

numa: 1

ostype: win10

sata1: /dev/disk/by-id/ata-LITEON_LCH-256V2S_SD0F66157L1TH5710EPZ,size=250059096K

scsihw: virtio-scsi-pci

smbios1: uuid=b67061eb-f1cf-40c5-baa3-748036850969

sockets: 1

vga: memory=512

vmgenid: ae5dbc11-04d3-425a-a91c-43e9d3e07087
 
I have tried both Unraid and ESXi, only unraid recognise the Wi-Fi as USB device, but have malfunction, always disconnect...
Under this condition, what could i do to make any PCI passthrough device work in Proxmox VE?
 
Last edited:
Joris L. said:
do you mind sharing the output for: dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug

kernel boot params you could try: pci=noats nomodeset

Also, did you use the https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.171-1/ or https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.185-1/ ISO to install driver for the VM ? Also try out installing qemu-ga found on the ISOs.
Click to expand...

Thank you for your reply.
Here is the dmesg, I found that the PCIe address seems to have error, all ffffffff:ffffffff and no classes. Before I open the WIndows as VM, I had installed the virtio-win-gt-x64.msi in virtio-win-0.1.185-1.iso, but I am not sure the gt-x64 msi include the qemu-ga. I had also tried to install the "Virtio-win-guest-tools.exe", but meet an error.

1591678526937.png

root@pve:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=noats nofb nomodeset video=vesafb:off,efifb:off"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

root@pve:~# dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=noats nofb nomodeset video=vesafb:off,efifb:off
[ 0.000000] Intel GenuineIntel
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] ACPI: SSDT 0x000000009CDD8000 0006B0 (v02 Intel_ TpmTable 00001000 INTL 20120711)
[ 0.000000] Reserving Intel graphics memory at 0x000000009e000000-0x000000009fffffff
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=noats nofb nomodeset video=vesafb:off,efifb:off
[ 0.000000] DMAR: IOMMU enabled
[ 0.000000] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.052786] smpboot: CPU0: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz (family: 0x6, model: 0x3d, stepping: 0x4)
[ 0.052865] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
[ 0.054829] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[ 0.065180] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[ 0.078214] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.091788] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
[ 0.092016] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled
[ 0.092226] pci 0000:00:1c.4: Intel PCH root port ACS workaround enabled
[ 0.151469] pnp 00:01: [Firmware Bug]: PNP resource [mem 0xfed10000-0xfed13fff] covers only part of 0000:00:00.0 Intel MCH; extending to [mem 0xfed10000-0xfed17fff]
[ 0.785118] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 0.785169] iommu: Adding device 0000:00:00.0 to group 0
[ 0.785178] iommu: Adding device 0000:00:02.0 to group 1
[ 0.785186] iommu: Adding device 0000:00:03.0 to group 2
[ 0.785194] iommu: Adding device 0000:00:14.0 to group 3
[ 0.785209] iommu: Adding device 0000:00:16.0 to group 4
[ 0.785217] iommu: Adding device 0000:00:19.0 to group 5
[ 0.785225] iommu: Adding device 0000:00:1b.0 to group 6
[ 0.785234] iommu: Adding device 0000:00:1c.0 to group 7
[ 0.785245] iommu: Adding device 0000:00:1c.1 to group 8
[ 0.785253] iommu: Adding device 0000:00:1c.4 to group 9
[ 0.785262] iommu: Adding device 0000:00:1d.0 to group 10
[ 0.785284] iommu: Adding device 0000:00:1f.0 to group 11
[ 0.785293] iommu: Adding device 0000:00:1f.2 to group 11
[ 0.785303] iommu: Adding device 0000:00:1f.3 to group 11
[ 0.785312] iommu: Adding device 0000:00:1f.6 to group 11
[ 0.785321] iommu: Adding device 0000:02:00.0 to group 12
[ 0.785329] iommu: Adding device 0000:03:00.0 to group 13
[ 0.785337] iommu: Adding device 0000:04:00.0 to group 14
[ 0.866076] ehci-pci 0000:00:1d.0: debug port 2
[ 0.890290] intel_pstate: Intel P-state driver initializing
[ 1.537480] e1000e: Intel(R) PRO/1000 Network Driver - 3.4.1.1-NAPI
[ 1.537481] e1000e: Copyright(c) 1999 - 2018 Intel Corporation.
[ 1.711904] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[ 3.771613] vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 3.789012] vfio_pci: add [8086:1616[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.808215] vfio_pci: add [8086:160c[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.832241] vfio_pci: add [8086:095b[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.852541] vfio_pci: add [10de:1347[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.872557] vfio_pci: add [10ec:5227[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.964643] Disabling lock debugging due to kernel taint
[ 4.033642] Bluetooth: hci0: read Intel version: 370810011003110e35
[ 4.033643] Bluetooth: hci0: Intel device is already patched. patch num: 35
[ 147.679689] vfio-pci 0000:00:02.0: ready 1123ms after FLR
[ 147.780701] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
[ 149.456536] vfio-pci 0000:03:00.0: enabling device (0000 -> 0002)
[ 149.564276] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1e@0x154
[ 149.676258] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]
 
iommu=pt intel_iommu=on nobar pci=noats video=vesafb:off,efifb:off" pcie_acs_override=downstream,multifunction text nomodeset

What i personally add to vfio-pci as module option is disable_vga=1 disable_idle_d3=1 nointmask=1

wrt. the error, choose the 64 bit .msi file not the 32 bit .exe
 
Joris L. said:
iommu=pt intel_iommu=on nobar pci=noats video=vesafb:eek:ff,efifb:eek:ff" pcie_acs_override=downstream,multifunction text nomodeset

What i personally add to vfio-pci as module option is disable_vga=1 disable_idle_d3=1 nointmask=1

wrt. the error, choose the 64 bit .msi file not the 32 bit .exe
Click to expand...

Seems there is no option to install the 64bit QXL-WDDM-DOD in the virtio-win-0.1.185-1.iso
Also, applying the "nobar" and "text" in grub, the dmesg become shorter.
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=noats nobar nofb text nomodeset video=vesafb:off,efifb:off"
root@pve:~# cat /etc/pve/qemu-server/103.conf
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 0
bios: ovmf
boot: dnc
bootdisk: sata1
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: local-lvm:vm-103-disk-1,size=128K
hostpci0: 00:02,romfile=HD5500.bin,pcie=1
hostpci1: 02:00,pcie=1
hostpci2: 00:03,pcie=1
hostpci3: 03:00
ide2: none,media=cdrom
machine: q35
memory: 8192
net0: e1000=4A:30:BD:37:BB:DD,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata1: /dev/disk/by-id/ata-LITEON_LCH-256V2S_SD0F66157L1TH5710EPZ,size=250059096K
scsihw: virtio-scsi-pci
smbios1: uuid=71380034-1be6-466c-aec4-4f07fb277d01
sockets: 1
vmgenid: 91fd4ff0-b2a7-4781-89bc-d0c3ed6db4d7
root@pve:~# dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug
[ 2623.394505] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]
[ 2623.434180] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e@0x158
[ 2623.650536] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1e@0x154
 
ok, I updated the grub and reboot

root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:off,efifb:off


root@pve:~# dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:off,efifb:off
[ 0.000000] Intel GenuineIntel
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] ACPI: SSDT 0x000000009CDD8000 0006B0 (v02 Intel_ TpmTable 00001000 INTL 20120711)
[ 0.000000] Reserving Intel graphics memory at 0x000000009e000000-0x000000009fffffff
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:off,efifb:off
[ 0.000000] DMAR: IOMMU enabled
[ 0.000000] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.049024] smpboot: CPU0: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz (family: 0x6, model: 0x3d, stepping: 0x4)
[ 0.049102] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
[ 0.051063] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[ 0.065193] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[ 0.075461] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.088024] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
[ 0.088252] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled
[ 0.088465] pci 0000:00:1c.4: Intel PCH root port ACS workaround enabled
[ 0.139434] pnp 00:01: [Firmware Bug]: PNP resource [mem 0xfed10000-0xfed13fff] covers only part of 0000:00:00.0 Intel MCH; extending to [mem 0xfed10000-0xfed17fff]
[ 0.774371] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 0.774426] iommu: Adding device 0000:00:00.0 to group 0
[ 0.774437] iommu: Adding device 0000:00:02.0 to group 1
[ 0.774446] iommu: Adding device 0000:00:03.0 to group 2
[ 0.774455] iommu: Adding device 0000:00:14.0 to group 3
[ 0.774468] iommu: Adding device 0000:00:16.0 to group 4
[ 0.774478] iommu: Adding device 0000:00:19.0 to group 5
[ 0.774487] iommu: Adding device 0000:00:1b.0 to group 6
[ 0.774496] iommu: Adding device 0000:00:1c.0 to group 7
[ 0.774505] iommu: Adding device 0000:00:1c.1 to group 8
[ 0.774515] iommu: Adding device 0000:00:1c.4 to group 9
[ 0.774523] iommu: Adding device 0000:00:1d.0 to group 10
[ 0.774546] iommu: Adding device 0000:00:1f.0 to group 11
[ 0.774555] iommu: Adding device 0000:00:1f.2 to group 11
[ 0.774564] iommu: Adding device 0000:00:1f.3 to group 11
[ 0.774573] iommu: Adding device 0000:00:1f.6 to group 11
[ 0.774583] iommu: Adding device 0000:02:00.0 to group 12
[ 0.774591] iommu: Adding device 0000:03:00.0 to group 13
[ 0.774600] iommu: Adding device 0000:04:00.0 to group 14
[ 0.844507] ehci-pci 0000:00:1d.0: debug port 2
[ 0.870092] intel_pstate: Intel P-state driver initializing
[ 1.526045] e1000e: Intel(R) PRO/1000 Network Driver - 3.4.1.1-NAPI
[ 1.526046] e1000e: Copyright(c) 1999 - 2018 Intel Corporation.
[ 1.700768] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[ 3.588237] vfio_pci: unknown parameter 'nointmask' ignored
[ 3.588407] vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 3.588436] vfio_pci: add [8086:1616[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588504] vfio_pci: add [8086:160c[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588570] vfio_pci: add [8086:095b[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588657] vfio_pci: add [10de:1347[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588720] vfio_pci: add [10ec:5227[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.656713] Disabling lock debugging due to kernel taint
[ 4.835908] Bluetooth: hci0: read Intel version: 370810011003110e35
[ 4.835909] Bluetooth: hci0: Intel device is already patched. patch num: 35
[ 214.778640] vfio-pci 0000:00:02.0: ready 1123ms after FLR
[ 216.062036] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
[ 217.867109] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]
[ 234.526703] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]
[ 372.188769] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]
[ 372.228422] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e@0x158
[ 372.336751] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1e@0x154
[ 372.340681] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
[ 373.368349] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1e@0x258
[ 373.368364] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x900
 
Hmm. Looks good. My suggestion is to gradually increase the number of vfio-pci ids you configured, that are a lot of ids. Try with one GPU and one other device first. Verify if all vfio-pci ids are correct, maybe also try nointmask=1 ?

Make sure to have IOMMU set to enabled in the BIOS, not auto. Enable UEFI, not csm.

consider commenting and rebooting

option kvm ignore.msrs=1


once: vfio-pci 0000:00:02.0: BAR 2: can't reserve message are gone it should work.


Here is the full boot string i use, do note thise entries in bold are AMD specific, you can remove them. If i remember well, putting video=efifb:off,vesafb:off before all others arguments as shown here may be of importance, try it i'd say.

iommu=pt amd_iommu=forced_isolation video=efifb:off,vesafb:off text nomodeset rcu_nocbs=0-15 amdgpu.runpm=0 pci=nobar,noats,big_root_window pcie_acs_override=downstream,multifunction kvm-amd.nested=0 profile=kvm pcie_pme=nomsi pcie_aspm=force pcie_ports=native
 
Last edited:
nointmask=1 already exists. I would like to know if "vfio-pci ids" you mean is equal to the PCI device. Currently I have tried all combination of the five device I added to vfio-pci, except the Intel chipset controller.

I would like to know more about the meaning of log. Is these messages normal?

[ 0.065193] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored

[ 0.139434] pnp 00:01: [Firmware Bug]: PNP resource [mem 0xfed10000-0xfed13fff] covers only part of 0000:00:00.0 Intel MCH; extending to [mem 0xfed10000-0xfed17fff]

[ 3.588237] vfio_pci: unknown parameter 'nointmask' ignored

[ 3.588436] vfio_pci: add [8086:1616[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588504] vfio_pci: add [8086:160c[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588570] vfio_pci: add [8086:095b[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588657] vfio_pci: add [10de:1347[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.588720] vfio_pci: add [10ec:5227[ffffffff:ffffffff]] class 0x000000/00000000


[ 217.867109] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]
[ 234.526703] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]
[ 372.188769] vfio-pci 0000:00:02.0: BAR 2: can't reserve [mem 0xc0000000-0xdfffffff 64bit pref]

[ 372.228422] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e@0x158
[ 372.336751] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1e@0x154

[ 372.340681] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)

[ 373.368349] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1e@0x258
[ 373.368364] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x900



root@pve:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:1616,8086:160c,8086:095b,10de:1347,10ec:5227 disable_vga=1 disable_idle_d3=1 nointmask=1

root@pve:~# lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Broadwell-U Host Bridge -OPI [8086:1604] (rev 09)
Subsystem: Lenovo Broadwell-U Host Bridge -OPI [17aa:5034]
Kernel driver in use: bdw_uncore
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 5500 [8086:1616] (rev 09)
Subsystem: Lenovo HD Graphics 5500 [17aa:5037]
Kernel driver in use: vfio-pci
Kernel modules: i915
00:03.0 Audio device [0403]: Intel Corporation Broadwell-U Audio Controller [8086:160c] (rev 09)
Subsystem: Lenovo Broadwell-U Audio Controller [17aa:5034]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
00:14.0 USB controller [0c03]: Intel Corporation Wildcat Point-LP USB xHCI Controller [8086:9cb1] (rev 03)
Subsystem: Lenovo Wildcat Point-LP USB xHCI Controller [17aa:5034]
Kernel driver in use: xhci_hcd
00:16.0 Communication controller [0780]: Intel Corporation Wildcat Point-LP MEI Controller #1 [8086:9cba] (rev 03)
Subsystem: Lenovo Wildcat Point-LP MEI Controller [17aa:5034]
Kernel driver in use: mei_me
Kernel modules: mei_me
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (3) I218-V [8086:15a3] (rev 03)
Subsystem: Lenovo Ethernet Connection (3) I218-V [17aa:2227]
Kernel driver in use: e1000e
Kernel modules: e1000e
00:1b.0 Audio device [0403]: Intel Corporation Wildcat Point-LP High Definition Audio Controller [8086:9ca0] (rev 03)
Subsystem: Lenovo Wildcat Point-LP High Definition Audio Controller [17aa:5036]
Kernel modules: snd_hda_intel
00:1c.0 PCI bridge [0604]: Intel Corporation Wildcat Point-LP PCI Express Root Port #6 [8086:9c9a] (rev e3)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.1 PCI bridge [0604]: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 [8086:9c94] (rev e3)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.4 PCI bridge [0604]: Intel Corporation Wildcat Point-LP PCI Express Root Port #5 [8086:9c98] (rev e3)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1d.0 USB controller [0c03]: Intel Corporation Wildcat Point-LP USB EHCI Controller [8086:9ca6] (rev 03)
Subsystem: Lenovo Wildcat Point-LP USB EHCI Controller [17aa:5034]
Kernel driver in use: ehci-pci
00:1f.0 ISA bridge [0601]: Intel Corporation Wildcat Point-LP LPC Controller [8086:9cc3] (rev 03)
Subsystem: Lenovo Wildcat Point-LP LPC Controller [17aa:5034]
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] [8086:9c83] (rev 03)
Subsystem: Lenovo Wildcat Point-LP SATA Controller [AHCI Mode] [17aa:5034]
Kernel driver in use: ahci
Kernel modules: ahci
00:1f.3 SMBus [0c05]: Intel Corporation Wildcat Point-LP SMBus Controller [8086:9ca2] (rev 03)
Subsystem: Lenovo Wildcat Point-LP SMBus Controller [17aa:5034]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
00:1f.6 Signal processing controller [1180]: Intel Corporation Wildcat Point-LP Thermal Management Controller [8086:9ca4] (rev 03)
Subsystem: Lenovo Wildcat Point-LP Thermal Management Controller [17aa:5034]
Kernel driver in use: intel_pch_thermal
Kernel modules: intel_pch_thermal
02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5227 PCI Express Card Reader [10ec:5227] (rev 01)
Subsystem: Lenovo RTS5227 PCI Express Card Reader [17aa:5034]
Kernel driver in use: vfio-pci
Kernel modules: rtsx_pci
03:00.0 Network controller [0280]: Intel Corporation Wireless 7265 [8086:095b] (rev 59)
Subsystem: Intel Corporation Dual Band Wireless-AC 7265 [8086:5210]
Kernel driver in use: vfio-pci
Kernel modules: iwlwifi
04:00.0 3D controller [0302]: NVIDIA Corporation GM108M [GeForce 940M] [10de:1347] (rev a2)
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
 
iGPU
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 5500 [8086:1616] (rev 09)
GPU
04:00.0 3D controller [0302]: NVIDIA Corporation GM108M [GeForce 940M] [10de:1347] (rev a2)
Audio PCI
00:03.0 Audio device [0403]: Intel Corporation Broadwell-U Audio Controller [8086:160c] (rev 09)
 
Joris L. said:
iGPU
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 5500 [8086:1616] (rev 09)
GPU
04:00.0 3D controller [0302]: NVIDIA Corporation GM108M [GeForce 940M] [10de:1347] (rev a2)
Audio PCI
00:03.0 Audio device [0403]: Intel Corporation Broadwell-U Audio Controller [8086:160c] (rev 09)
Click to expand...

So I should edit the vfio.conf to only these device? like

options vfio-pci ids=8086:1616,8086:160c,10de:1347 disable_vga=1 disable_idle_d3=1 nointmask=1

but I met this error when added nointmask=1, does it mean that the option is not available in Proxmox 5.4-15?

[ 3.588237] vfio_pci: unknown parameter 'nointmask' ignored
 
Important to know is the VM itself must also be configured appropriately. In my experience you must reinstall the VM sometimes for the PCI passthrough to catch on, or reboot the server.

CPU: host
Chipset: q35

When you add a PCI device it is a matter of experimenting with the settings for them. I found disabling 'all functions' and enabled 'PCI-Express' worked well for the GPU i have in passthrough.

Also, upload the Virtio ISO to proxmox and you can mount it as a cdrom in windows. Much easier.

What you do with PCI passthrough entirely depends on what you want. Passthrough is interesting for some purposes, not for all. For most devices (audio, usb) there is no need, just use kvm64 or host as CPU for maximum performance.

Do not pass through what you do not need exclusively. Typically you pass through devices you do not share with the host (server)

Configuring vfio-pci ids has consequences. It is also the reason i use amd_iommu=force_isolation and also experiment with one or more of pcie_pme=nomsi pcie_aspm=force pcie_ports=native

you can check options with modinfo vfio-pci
 
Last edited:
Understand, thank you very much to explain about the PCI Passthrough.

Unfortunately I don't a spare device to install a Windows VM for testing, under this condition, I would suspend the testing, backup the entire Windows disk and install the VM with SSD Passthrough to see any difference.

Btw, seems the parameter is 'nointxmask' not 'nointmask'? Do I need to rearrange the grub config GRUB_CMDLINE_LINUX_DEFAULT?
 
Yeah, sorry, typo, my bad,indeed, notintxmask=1

Not sure what you are trying to do really. At this moment i think all the configuration you have is working as expected and should not change much.

The option i do think you can work with are

  • check: dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug -e BAR
    • when the BARx: can't reserve messages are gone you have fixed it.
  • realise there are just a few reasons why these messages change, reduce the number of changes you make per test and takes notes, rebooting when you make changes
    • configuration change in the bios
      • make sure to test if UEFI is enabled, c-states are enabled and iommu is enabled, and svm is enabled
        • in some cases setting it to enabled instead of auto works better
    • configuration change in the kernel boot parameters
      • these look fine to me, maybe test by appending one or more of pcie_pme=nomsi pcie_aspm=force pcie_ports=native
    • configuration change in the /etc/modrprobe.d option parameters
      • options kvm ignore_msrs=1
        • try booting with and without this module option enabled or commented/disabled
    • configuration change of the VM configuration
      • configure the CPU for your VM to be set host
        • kvm64 or default will considerably change the reliability of what you test, use host for all testing.

recommended reading on passthrough on Linux, never mind the distribution, Linux is Linux at this level.

https://wiki.archlinux.org/index.ph...serve_[mem]"_error_in_dmesg_after_starting_VM

These two lines may help your troubleshooting, i use a script specific to AMD CPU which works great and avoids reboot.

echo 1 > /sys/bus/pci/devices/0000\:00\:04.0/remove
echo 1 > /sys/bus/pci/rescan

and related to the above https://wiki.archlinux.org/index.ph...h_via_OVMF#UEFI_(OVMF)_Compatibility_in_VBIOS

An extensive article is here

https://wiki.installgentoo.com/index.php/PCI_passthrough


Know it took me over a week to grow understanding and come to a fix, and diagnosed isolation of a non-fixable issue for which i found the reset script for amdgpu, persist and try things out.
 
Thank you very much, Joris.

Here is the new dmesg which I added the notintxmask=1 in /etc/modprobe.d/vfio.conf
Look like IOMMU not work after added this option
I found I just change the intel_iommu=on to intel_iommu=force_isolation without understanding

root@pve:~# dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=force_isolation iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff
[ 0.000000] Intel GenuineIntel
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] ACPI: SSDT 0x000000009CDD8000 0006B0 (v02 Intel_ TpmTable 00001000 INTL 20120711)
[ 0.000000] Reserving Intel graphics memory at 0x000000009e000000-0x000000009fffffff
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=force_isolation iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff
[ 0.000000] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.048940] smpboot: CPU0: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz (family: 0x6, model: 0x3d, stepping: 0x4)
[ 0.049018] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
[ 0.050979] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[ 0.061263] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[ 0.075557] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.139519] pnp 00:01: [Firmware Bug]: PNP resource [mem 0xfed10000-0xfed13fff] covers only part of 0000:00:00.0 Intel MCH; extending to [mem 0xfed10000-0xfed17fff]
[ 0.847798] ehci-pci 0000:00:1d.0: debug port 2
[ 0.870950] intel_pstate: Intel P-state driver initializing
[ 1.515241] e1000e: Intel(R) PRO/1000 Network Driver - 3.4.1.1-NAPI
[ 1.515242] e1000e: Copyright(c) 1999 - 2018 Intel Corporation.
[ 1.688799] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[ 3.536906] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536913] vfio_pci: add [8086:1616[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536920] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536926] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536930] vfio_pci: add [8086:160c[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536936] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536941] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536949] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.536951] vfio_pci: add [8086:095b[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536957] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536962] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536970] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.536976] vfio-pci: probe of 0000:04:00.0 failed with error -22
[ 3.536978] vfio_pci: add [10de:1347[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536984] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536993] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.537001] vfio-pci: probe of 0000:02:00.0 failed with error -22
[ 3.537007] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.537013] vfio-pci: probe of 0000:04:00.0 failed with error -22
[ 3.537015] vfio_pci: add [10ec:5227[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.599892] Disabling lock debugging due to kernel taint
[ 3.814587] Bluetooth: hci0: read Intel version: 370810011003110e35
[ 3.814589] Bluetooth: hci0: Intel device is already patched. patch num: 35


I would like to have a crosscheck with your successful sample before I start to test the combination of multiple parameter.

The GRUB_CMDLINE_LINUX_DEFAULT in your /etc/default/grub should be posted in #11:

iommu=pt amd_iommu=forced_isolation video=efifb:eek:ff,vesafb:eek:ff text nomodeset rcu_nocbs=0-15 amdgpu.runpm=0 pci=nobar,noats,big_root_window pcie_acs_override=downstream,multifunction kvm-amd.nested=0 profile=kvm pcie_pme=nomsi pcie_aspm=force pcie_ports=native

Currently my GRUB config is:

quiet intel_iommu=on iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff"

Seems that I can remove the quiet config? Also, what is the difference to add eek inside the video driver efifb and vesafb? I cannot find anything about the eek option.

May I have your vm config and dmesg so that I can know which part I should clear and which part I should achieve?
 
Last edited:
testcb00 said:
Thank you very much, Joris.

Here is the new dmesg which I added the notintxmask=1 in /etc/modprobe.d/vfio.conf
Look like IOMMU not work after added this option

root@pve:~# dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=force_isolation iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff
[ 0.000000] Intel GenuineIntel
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] ACPI: SSDT 0x000000009CDD8000 0006B0 (v02 Intel_ TpmTable 00001000 INTL 20120711)
[ 0.000000] Reserving Intel graphics memory at 0x000000009e000000-0x000000009fffffff
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=force_isolation iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff
[ 0.000000] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.048940] smpboot: CPU0: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz (family: 0x6, model: 0x3d, stepping: 0x4)
[ 0.049018] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
[ 0.050979] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[ 0.061263] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[ 0.075557] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.139519] pnp 00:01: [Firmware Bug]: PNP resource [mem 0xfed10000-0xfed13fff] covers only part of 0000:00:00.0 Intel MCH; extending to [mem 0xfed10000-0xfed17fff]
[ 0.847798] ehci-pci 0000:00:1d.0: debug port 2
[ 0.870950] intel_pstate: Intel P-state driver initializing
[ 1.515241] e1000e: Intel(R) PRO/1000 Network Driver - 3.4.1.1-NAPI
[ 1.515242] e1000e: Copyright(c) 1999 - 2018 Intel Corporation.
[ 1.688799] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[ 3.536906] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536913] vfio_pci: add [8086:1616[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536920] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536926] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536930] vfio_pci: add [8086:160c[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536936] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536941] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536949] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.536951] vfio_pci: add [8086:095b[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536957] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536962] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536970] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.536976] vfio-pci: probe of 0000:04:00.0 failed with error -22
[ 3.536978] vfio_pci: add [10de:1347[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536984] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536993] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.537001] vfio-pci: probe of 0000:02:00.0 failed with error -22
[ 3.537007] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.537013] vfio-pci: probe of 0000:04:00.0 failed with error -22
[ 3.537015] vfio_pci: add [10ec:5227[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.599892] Disabling lock debugging due to kernel taint
[ 3.814587] Bluetooth: hci0: read Intel version: 370810011003110e35
[ 3.814589] Bluetooth: hci0: Intel device is already patched. patch num: 35


I would like to have a crosscheck with your successful sample before I start to test the combination of multiple parameter.

The GRUB_CMDLINE_LINUX_DEFAULT in your /etc/default/grub should be posted in #11:

iommu=pt amd_iommu=forced_isolation video=efifb:eek:ff,vesafb:eek:ff text nomodeset rcu_nocbs=0-15 amdgpu.runpm=0 pci=nobar,noats,big_root_window pcie_acs_override=downstream,multifunction kvm-amd.nested=0 profile=kvm pcie_pme=nomsi pcie_aspm=force pcie_ports=native

Currently my GRUB config is:

quiet intel_iommu=force_isolation iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff"

Seems that I can remove the quiet config? Also, what is the difference to add eek inside the video driver efifb and vesafb? I cannot find anything about the eek option.

May I have your vm config and dmesg so that I can know which part I should clear and which part I should achieve?
Click to expand...

check here https://www.kernel.org/doc/html/v5.4/admin-guide/kernel-parameters.html
i have see eek, i have not looked into what it mean, may work for your setup

Use this line, you cannot simply copy stuff :)

iommu=pt intel_iommu=on video=vesafb:eek:ff,efifb:eek:ff pci=nobar,noats pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction nofb text nomodeset

i have no vm config, just q35 and host cpu

when you add the PCI device you have to select the IDS matching port number and figure out what setting works for the particular PCI device.
 
testcb00 said:
Thank you very much, Joris.

Here is the new dmesg which I added the notintxmask=1 in /etc/modprobe.d/vfio.conf
Look like IOMMU not work after added this option
I found I just change the intel_iommu=on to intel_iommu=force_isolation without understanding

root@pve:~# dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=force_isolation iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff
[ 0.000000] Intel GenuineIntel
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] ACPI: SSDT 0x000000009CDD8000 0006B0 (v02 Intel_ TpmTable 00001000 INTL 20120711)
[ 0.000000] Reserving Intel graphics memory at 0x000000009e000000-0x000000009fffffff
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.18-29-pve root=/dev/mapper/pve-root ro quiet intel_iommu=force_isolation iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff
[ 0.000000] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.048940] smpboot: CPU0: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz (family: 0x6, model: 0x3d, stepping: 0x4)
[ 0.049018] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
[ 0.050979] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[ 0.061263] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[ 0.075557] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.139519] pnp 00:01: [Firmware Bug]: PNP resource [mem 0xfed10000-0xfed13fff] covers only part of 0000:00:00.0 Intel MCH; extending to [mem 0xfed10000-0xfed17fff]
[ 0.847798] ehci-pci 0000:00:1d.0: debug port 2
[ 0.870950] intel_pstate: Intel P-state driver initializing
[ 1.515241] e1000e: Intel(R) PRO/1000 Network Driver - 3.4.1.1-NAPI
[ 1.515242] e1000e: Copyright(c) 1999 - 2018 Intel Corporation.
[ 1.688799] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[ 3.536906] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536913] vfio_pci: add [8086:1616[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536920] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536926] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536930] vfio_pci: add [8086:160c[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536936] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536941] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536949] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.536951] vfio_pci: add [8086:095b[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536957] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536962] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.536970] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.536976] vfio-pci: probe of 0000:04:00.0 failed with error -22
[ 3.536978] vfio_pci: add [10de:1347[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.536984] vfio-pci: probe of 0000:00:02.0 failed with error -22
[ 3.536993] vfio-pci: probe of 0000:00:03.0 failed with error -22
[ 3.537001] vfio-pci: probe of 0000:02:00.0 failed with error -22
[ 3.537007] vfio-pci: probe of 0000:03:00.0 failed with error -22
[ 3.537013] vfio-pci: probe of 0000:04:00.0 failed with error -22
[ 3.537015] vfio_pci: add [10ec:5227[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.599892] Disabling lock debugging due to kernel taint
[ 3.814587] Bluetooth: hci0: read Intel version: 370810011003110e35
[ 3.814589] Bluetooth: hci0: Intel device is already patched. patch num: 35


I would like to have a crosscheck with your successful sample before I start to test the combination of multiple parameter.

The GRUB_CMDLINE_LINUX_DEFAULT in your /etc/default/grub should be posted in #11:

iommu=pt amd_iommu=forced_isolation video=efifb:eek:ff,vesafb:eek:ff text nomodeset rcu_nocbs=0-15 amdgpu.runpm=0 pci=nobar,noats,big_root_window pcie_acs_override=downstream,multifunction kvm-amd.nested=0 profile=kvm pcie_pme=nomsi pcie_aspm=force pcie_ports=native

Currently my GRUB config is:

quiet intel_iommu=on iommu=pt pcie_pme=nomsi pcie_aspm=force pcie_ports=native pcie_acs_override=downstream,multifunction pci=nobar,noats nofb text nomodeset video=vesafb:eek:ff,efifb:eek:ff"

Seems that I can remove the quiet config? Also, what is the difference to add eek inside the video driver efifb and vesafb? I cannot find anything about the eek option.

May I have your vm config and dmesg so that I can know which part I should clear and which part I should achieve?
Click to expand...
The output here looks really good.

Check again with

dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug -e BAR
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom