Nvidia GRID K1 passthrough - Error 43 strikes again!

I

ivdok

New Member
Mar 12, 2019
12
0
1
This is a tricky one - I've tried everything Google suggested, and then some.
Relevant PVE stack, H/W and guest configurations are below:
Code: 
:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000007DDAD200 00051C (v01 HP ProLiant 00000001 \xffffffd2? 0000162E)
[ 0.000000] DMAR: IOMMU enabled
[ 0.000000] DMAR: Host address width 46
[ 0.000000] DMAR: DRHD base: 0x000000b5ffe000 flags: 0x0
[ 0.000000] DMAR: dmar0: reg_base_addr b5ffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[ 0.000000] DMAR: DRHD base: 0x000000b0ffe000 flags: 0x1
[ 0.000000] DMAR: dmar1: reg_base_addr b0ffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[ 0.000000] DMAR: RMRR base: 0x0000007dffd000 end: 0x0000007dffffff
[ 0.000000] DMAR: RMRR base: 0x0000007dff6000 end: 0x0000007dffcfff
[ 0.000000] DMAR: RMRR base: 0x0000007df83000 end: 0x0000007df84fff
[ 0.000000] DMAR: RMRR base: 0x0000007df7f000 end: 0x0000007df82fff
[ 0.000000] DMAR: RMRR base: 0x0000007df6f000 end: 0x0000007df7efff
[ 0.000000] DMAR: RMRR base: 0x0000007df6e000 end: 0x0000007df6efff
[ 0.000000] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
[ 0.000000] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
[ 0.000000] DMAR: RMRR base: 0x0000007ddde000 end: 0x0000007dddefff
[ 0.000000] DMAR: ATSR flags: 0x0
[ 0.000000] DMAR-IR: IOAPIC id 10 under DRHD base 0xb5ffe000 IOMMU 0
[ 0.000000] DMAR-IR: IOAPIC id 8 under DRHD base 0xb0ffe000 IOMMU 1
[ 0.000000] DMAR-IR: IOAPIC id 0 under DRHD base 0xb0ffe000 IOMMU 1
[ 0.000000] DMAR-IR: HPET id 0 under DRHD base 0xb0ffe000
[ 0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 1.200771] DMAR: dmar0: Using Queued invalidation
[ 1.200781] DMAR: dmar1: Using Queued invalidation
[ 1.201487] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 1.201493] DMAR: Hardware identity mapping for device 0000:00:01.0
[ 1.201498] DMAR: Hardware identity mapping for device 0000:00:01.1
[ 1.201504] DMAR: Hardware identity mapping for device 0000:00:02.0
[ 1.201509] DMAR: Hardware identity mapping for device 0000:00:02.1
[ 1.201515] DMAR: Hardware identity mapping for device 0000:00:02.2
[ 1.201520] DMAR: Hardware identity mapping for device 0000:00:02.3
[ 1.201525] DMAR: Hardware identity mapping for device 0000:00:03.0
[ 1.201531] DMAR: Hardware identity mapping for device 0000:00:03.1
[ 1.201536] DMAR: Hardware identity mapping for device 0000:00:03.2
[ 1.201543] DMAR: Hardware identity mapping for device 0000:00:03.3
[ 1.201545] DMAR: Hardware identity mapping for device 0000:00:04.0
[ 1.201548] DMAR: Hardware identity mapping for device 0000:00:04.1
[ 1.201550] DMAR: Hardware identity mapping for device 0000:00:04.2
[ 1.201553] DMAR: Hardware identity mapping for device 0000:00:04.3
[ 1.201555] DMAR: Hardware identity mapping for device 0000:00:04.4
[ 1.201557] DMAR: Hardware identity mapping for device 0000:00:04.5
[ 1.201560] DMAR: Hardware identity mapping for device 0000:00:04.6
[ 1.201562] DMAR: Hardware identity mapping for device 0000:00:04.7
[ 1.201565] DMAR: Hardware identity mapping for device 0000:00:05.0
[ 1.201567] DMAR: Hardware identity mapping for device 0000:00:05.2
[ 1.201570] DMAR: Hardware identity mapping for device 0000:00:05.4
[ 1.201574] DMAR: Hardware identity mapping for device 0000:00:11.0
[ 1.201576] DMAR: Hardware identity mapping for device 0000:00:1a.0
[ 1.201579] DMAR: Hardware identity mapping for device 0000:00:1c.0
[ 1.201582] DMAR: Hardware identity mapping for device 0000:00:1c.7
[ 1.201584] DMAR: Hardware identity mapping for device 0000:00:1d.0
[ 1.201586] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 1.201596] DMAR: Hardware identity mapping for device 0000:01:00.1
[ 1.201599] DMAR: Hardware identity mapping for device 0000:01:00.4
[ 1.201608] DMAR: Hardware identity mapping for device 0000:1f:08.0
[ 1.201610] DMAR: Hardware identity mapping for device 0000:1f:09.0
[ 1.201612] DMAR: Hardware identity mapping for device 0000:1f:0a.0
[ 1.201613] DMAR: Hardware identity mapping for device 0000:1f:0a.1
[ 1.201615] DMAR: Hardware identity mapping for device 0000:1f:0a.2
[ 1.201617] DMAR: Hardware identity mapping for device 0000:1f:0a.3
[ 1.201618] DMAR: Hardware identity mapping for device 0000:1f:0b.0
[ 1.201620] DMAR: Hardware identity mapping for device 0000:1f:0b.3
[ 1.201622] DMAR: Hardware identity mapping for device 0000:1f:0c.0
[ 1.201623] DMAR: Hardware identity mapping for device 0000:1f:0c.1
[ 1.201625] DMAR: Hardware identity mapping for device 0000:1f:0c.2
[ 1.201627] DMAR: Hardware identity mapping for device 0000:1f:0c.3
[ 1.201629] DMAR: Hardware identity mapping for device 0000:1f:0c.4
[ 1.201630] DMAR: Hardware identity mapping for device 0000:1f:0d.0
[ 1.201632] DMAR: Hardware identity mapping for device 0000:1f:0d.1
[ 1.201634] DMAR: Hardware identity mapping for device 0000:1f:0d.2
[ 1.201636] DMAR: Hardware identity mapping for device 0000:1f:0d.3
[ 1.201637] DMAR: Hardware identity mapping for device 0000:1f:0d.4
[ 1.201639] DMAR: Hardware identity mapping for device 0000:1f:0e.0
[ 1.201641] DMAR: Hardware identity mapping for device 0000:1f:0e.1
[ 1.201643] DMAR: Hardware identity mapping for device 0000:1f:0f.0
[ 1.201645] DMAR: Hardware identity mapping for device 0000:1f:0f.1
[ 1.201647] DMAR: Hardware identity mapping for device 0000:1f:0f.2
[ 1.201649] DMAR: Hardware identity mapping for device 0000:1f:0f.3
[ 1.201651] DMAR: Hardware identity mapping for device 0000:1f:0f.4
[ 1.201654] DMAR: Hardware identity mapping for device 0000:1f:0f.5
[ 1.201656] DMAR: Hardware identity mapping for device 0000:1f:10.0
[ 1.201658] DMAR: Hardware identity mapping for device 0000:1f:10.1
[ 1.201660] DMAR: Hardware identity mapping for device 0000:1f:10.2
[ 1.201662] DMAR: Hardware identity mapping for device 0000:1f:10.3
[ 1.201664] DMAR: Hardware identity mapping for device 0000:1f:10.4
[ 1.201666] DMAR: Hardware identity mapping for device 0000:1f:10.5
[ 1.201668] DMAR: Hardware identity mapping for device 0000:1f:10.6
[ 1.201670] DMAR: Hardware identity mapping for device 0000:1f:10.7
[ 1.201672] DMAR: Hardware identity mapping for device 0000:1f:13.0
[ 1.201673] DMAR: Hardware identity mapping for device 0000:1f:13.1
[ 1.201675] DMAR: Hardware identity mapping for device 0000:1f:13.4
[ 1.201677] DMAR: Hardware identity mapping for device 0000:1f:13.5
[ 1.201679] DMAR: Hardware identity mapping for device 0000:1f:16.0
[ 1.201680] DMAR: Hardware identity mapping for device 0000:1f:16.1
[ 1.201682] DMAR: Hardware identity mapping for device 0000:1f:16.2
[ 1.201696] DMAR: Hardware identity mapping for device 0000:20:00.0
[ 1.201702] DMAR: Hardware identity mapping for device 0000:20:01.0
[ 1.201708] DMAR: Hardware identity mapping for device 0000:20:01.1
[ 1.201714] DMAR: Hardware identity mapping for device 0000:20:02.0
[ 1.201720] DMAR: Hardware identity mapping for device 0000:20:02.1
[ 1.201726] DMAR: Hardware identity mapping for device 0000:20:02.2
[ 1.201732] DMAR: Hardware identity mapping for device 0000:20:02.3
[ 1.201739] DMAR: Hardware identity mapping for device 0000:20:03.0
[ 1.201745] DMAR: Hardware identity mapping for device 0000:20:03.1
[ 1.201751] DMAR: Hardware identity mapping for device 0000:20:03.2
[ 1.201757] DMAR: Hardware identity mapping for device 0000:20:03.3
[ 1.201760] DMAR: Hardware identity mapping for device 0000:20:04.0
[ 1.201762] DMAR: Hardware identity mapping for device 0000:20:04.1
[ 1.201765] DMAR: Hardware identity mapping for device 0000:20:04.2
[ 1.201768] DMAR: Hardware identity mapping for device 0000:20:04.3
[ 1.201770] DMAR: Hardware identity mapping for device 0000:20:04.4
[ 1.201772] DMAR: Hardware identity mapping for device 0000:20:04.5
[ 1.201775] DMAR: Hardware identity mapping for device 0000:20:04.6
[ 1.201778] DMAR: Hardware identity mapping for device 0000:20:04.7
[ 1.201787] DMAR: Hardware identity mapping for device 0000:20:05.0
[ 1.201790] DMAR: Hardware identity mapping for device 0000:20:05.2
[ 1.201793] DMAR: Hardware identity mapping for device 0000:20:05.4
[ 1.201804] DMAR: Hardware identity mapping for device 0000:24:00.0
[ 1.201816] DMAR: Hardware identity mapping for device 0000:25:08.0
[ 1.201822] DMAR: Hardware identity mapping for device 0000:25:09.0
[ 1.201827] DMAR: Hardware identity mapping for device 0000:25:10.0
[ 1.201833] DMAR: Hardware identity mapping for device 0000:25:11.0
[ 1.201845] DMAR: Hardware identity mapping for device 0000:26:00.0
[ 1.201858] DMAR: Hardware identity mapping for device 0000:27:00.0
[ 1.201870] DMAR: Hardware identity mapping for device 0000:28:00.0
[ 1.201881] DMAR: Hardware identity mapping for device 0000:29:00.0
[ 1.201889] DMAR: Hardware identity mapping for device 0000:3f:08.0
[ 1.201891] DMAR: Hardware identity mapping for device 0000:3f:09.0
[ 1.201893] DMAR: Hardware identity mapping for device 0000:3f:0a.0
[ 1.201895] DMAR: Hardware identity mapping for device 0000:3f:0a.1
[ 1.201897] DMAR: Hardware identity mapping for device 0000:3f:0a.2
[ 1.201899] DMAR: Hardware identity mapping for device 0000:3f:0a.3
[ 1.201901] DMAR: Hardware identity mapping for device 0000:3f:0b.0
[ 1.201903] DMAR: Hardware identity mapping for device 0000:3f:0b.3
[ 1.201905] DMAR: Hardware identity mapping for device 0000:3f:0c.0
[ 1.201907] DMAR: Hardware identity mapping for device 0000:3f:0c.1
[ 1.201910] DMAR: Hardware identity mapping for device 0000:3f:0c.2
[ 1.201912] DMAR: Hardware identity mapping for device 0000:3f:0c.3
[ 1.201914] DMAR: Hardware identity mapping for device 0000:3f:0c.4
[ 1.201916] DMAR: Hardware identity mapping for device 0000:3f:0d.0
[ 1.201918] DMAR: Hardware identity mapping for device 0000:3f:0d.1
[ 1.201920] DMAR: Hardware identity mapping for device 0000:3f:0d.2
[ 1.201922] DMAR: Hardware identity mapping for device 0000:3f:0d.3
[ 1.201924] DMAR: Hardware identity mapping for device 0000:3f:0d.4
[ 1.201926] DMAR: Hardware identity mapping for device 0000:3f:0e.0
[ 1.201929] DMAR: Hardware identity mapping for device 0000:3f:0e.1
[ 1.201931] DMAR: Hardware identity mapping for device 0000:3f:0f.0
[ 1.201933] DMAR: Hardware identity mapping for device 0000:3f:0f.1
[ 1.201936] DMAR: Hardware identity mapping for device 0000:3f:0f.2
[ 1.201938] DMAR: Hardware identity mapping for device 0000:3f:0f.3
[ 1.201940] DMAR: Hardware identity mapping for device 0000:3f:0f.4
[ 1.201942] DMAR: Hardware identity mapping for device 0000:3f:0f.5
[ 1.201944] DMAR: Hardware identity mapping for device 0000:3f:10.0
[ 1.201947] DMAR: Hardware identity mapping for device 0000:3f:10.1
[ 1.201949] DMAR: Hardware identity mapping for device 0000:3f:10.2
[ 1.201952] DMAR: Hardware identity mapping for device 0000:3f:10.3
[ 1.201954] DMAR: Hardware identity mapping for device 0000:3f:10.4
[ 1.201956] DMAR: Hardware identity mapping for device 0000:3f:10.5
[ 1.201959] DMAR: Hardware identity mapping for device 0000:3f:10.6
[ 1.201961] DMAR: Hardware identity mapping for device 0000:3f:10.7
[ 1.201963] DMAR: Hardware identity mapping for device 0000:3f:13.0
[ 1.201965] DMAR: Hardware identity mapping for device 0000:3f:13.1
[ 1.201967] DMAR: Hardware identity mapping for device 0000:3f:13.4
[ 1.201969] DMAR: Hardware identity mapping for device 0000:3f:13.5
[ 1.201971] DMAR: Hardware identity mapping for device 0000:3f:16.0
[ 1.201973] DMAR: Hardware identity mapping for device 0000:3f:16.1
[ 1.201975] DMAR: Hardware identity mapping for device 0000:3f:16.2
[ 1.201977] DMAR: Setting RMRR:
[ 1.203004] DMAR: Setting identity map for device 0000:01:00.0 [0x7ddde000 - 0x7dddefff]
[ 1.204189] DMAR: Setting identity map for device 0000:01:00.2 [0x7ddde000 - 0x7dddefff]
[ 1.204208] DMAR: Ignoring identity map for HW passthrough device 0000:01:00.4 [0x7ddde000 - 0x7dddefff]
[ 1.205130] DMAR: Setting identity map for device 0000:00:1f.2 [0xe8000 - 0xe8fff]
[ 1.205149] DMAR: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
[ 1.205163] DMAR: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
[ 1.206184] DMAR: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff]
[ 1.207271] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
[ 1.208382] DMAR: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
[ 1.209331] DMAR: Setting identity map for device 0000:03:00.2 [0xe8000 - 0xe8fff]
[ 1.210221] DMAR: Setting identity map for device 0000:03:00.3 [0xe8000 - 0xe8fff]
[ 1.211304] DMAR: Setting identity map for device 0000:0a:00.0 [0xe8000 - 0xe8fff]
[ 1.211322] DMAR: Setting identity map for device 0000:00:1f.2 [0xf4000 - 0xf4fff]
[ 1.211325] DMAR: Setting identity map for device 0000:01:00.0 [0xf4000 - 0xf4fff]
[ 1.211327] DMAR: Setting identity map for device 0000:01:00.2 [0xf4000 - 0xf4fff]
[ 1.211329] DMAR: Setting identity map for device 0000:02:00.0 [0xf4000 - 0xf4fff]
[ 1.211331] DMAR: Setting identity map for device 0000:03:00.0 [0xf4000 - 0xf4fff]
[ 1.211332] DMAR: Setting identity map for device 0000:03:00.1 [0xf4000 - 0xf4fff]
[ 1.211334] DMAR: Setting identity map for device 0000:03:00.2 [0xf4000 - 0xf4fff]
[ 1.211336] DMAR: Setting identity map for device 0000:03:00.3 [0xf4000 - 0xf4fff]
[ 1.211338] DMAR: Setting identity map for device 0000:0a:00.0 [0xf4000 - 0xf4fff]
[ 1.211340] DMAR: Setting identity map for device 0000:00:1f.2 [0x7df6e000 - 0x7df6efff]
[ 1.211353] DMAR: Setting identity map for device 0000:01:00.0 [0x7df6e000 - 0x7df6efff]
[ 1.211360] DMAR: Setting identity map for device 0000:01:00.2 [0x7df6e000 - 0x7df6efff]
[ 1.211367] DMAR: Setting identity map for device 0000:02:00.0 [0x7df6e000 - 0x7df6efff]
[ 1.211380] DMAR: Setting identity map for device 0000:03:00.0 [0x7df6e000 - 0x7df6efff]
[ 1.211393] DMAR: Setting identity map for device 0000:03:00.1 [0x7df6e000 - 0x7df6efff]
[ 1.211406] DMAR: Setting identity map for device 0000:03:00.2 [0x7df6e000 - 0x7df6efff]
[ 1.211419] DMAR: Setting identity map for device 0000:03:00.3 [0x7df6e000 - 0x7df6efff]
[ 1.211431] DMAR: Setting identity map for device 0000:0a:00.0 [0x7df6e000 - 0x7df6efff]
[ 1.211449] DMAR: Setting identity map for device 0000:00:1f.2 [0x7df6f000 - 0x7df7efff]
[ 1.211452] DMAR: Setting identity map for device 0000:01:00.0 [0x7df6f000 - 0x7df7efff]
[ 1.211454] DMAR: Setting identity map for device 0000:01:00.2 [0x7df6f000 - 0x7df7efff]
[ 1.211457] DMAR: Setting identity map for device 0000:02:00.0 [0x7df6f000 - 0x7df7efff]
[ 1.211459] DMAR: Setting identity map for device 0000:03:00.0 [0x7df6f000 - 0x7df7efff]
[ 1.211461] DMAR: Setting identity map for device 0000:03:00.1 [0x7df6f000 - 0x7df7efff]
[ 1.211463] DMAR: Setting identity map for device 0000:03:00.2 [0x7df6f000 - 0x7df7efff]
[ 1.211465] DMAR: Setting identity map for device 0000:03:00.3 [0x7df6f000 - 0x7df7efff]
[ 1.211466] DMAR: Setting identity map for device 0000:0a:00.0 [0x7df6f000 - 0x7df7efff]
[ 1.211468] DMAR: Setting identity map for device 0000:00:1f.2 [0x7df7f000 - 0x7df82fff]
[ 1.211470] DMAR: Setting identity map for device 0000:01:00.0 [0x7df7f000 - 0x7df82fff]
[ 1.211472] DMAR: Setting identity map for device 0000:01:00.2 [0x7df7f000 - 0x7df82fff]
[ 1.211474] DMAR: Setting identity map for device 0000:02:00.0 [0x7df7f000 - 0x7df82fff]
[ 1.211476] DMAR: Setting identity map for device 0000:03:00.0 [0x7df7f000 - 0x7df82fff]
[ 1.211477] DMAR: Setting identity map for device 0000:03:00.1 [0x7df7f000 - 0x7df82fff]
[ 1.211479] DMAR: Setting identity map for device 0000:03:00.2 [0x7df7f000 - 0x7df82fff]
[ 1.211481] DMAR: Setting identity map for device 0000:03:00.3 [0x7df7f000 - 0x7df82fff]
[ 1.211483] DMAR: Setting identity map for device 0000:0a:00.0 [0x7df7f000 - 0x7df82fff]
[ 1.211484] DMAR: Setting identity map for device 0000:00:1f.2 [0x7df83000 - 0x7df84fff]
[ 1.211486] DMAR: Setting identity map for device 0000:01:00.0 [0x7df83000 - 0x7df84fff]
[ 1.211488] DMAR: Setting identity map for device 0000:01:00.2 [0x7df83000 - 0x7df84fff]
[ 1.211489] DMAR: Setting identity map for device 0000:02:00.0 [0x7df83000 - 0x7df84fff]
[ 1.211491] DMAR: Setting identity map for device 0000:03:00.0 [0x7df83000 - 0x7df84fff]
[ 1.211493] DMAR: Setting identity map for device 0000:03:00.1 [0x7df83000 - 0x7df84fff]
[ 1.211494] DMAR: Setting identity map for device 0000:03:00.2 [0x7df83000 - 0x7df84fff]
[ 1.211496] DMAR: Setting identity map for device 0000:03:00.3 [0x7df83000 - 0x7df84fff]
[ 1.211498] DMAR: Setting identity map for device 0000:0a:00.0 [0x7df83000 - 0x7df84fff]
[ 1.211500] DMAR: Setting identity map for device 0000:01:00.0 [0x7dff6000 - 0x7dffcfff]
[ 1.211502] DMAR: Setting identity map for device 0000:01:00.2 [0x7dff6000 - 0x7dffcfff]
[ 1.211503] DMAR: Ignoring identity map for HW passthrough device 0000:01:00.4 [0x7dff6000 - 0x7dffcfff]
[ 1.211504] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0x7dffd000 - 0x7dffffff]
[ 1.211505] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0x7dffd000 - 0x7dffffff]
[ 1.211509] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 1.211510] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 1.211514] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 1.466224] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
[ 1.467415] DMAR: Setting identity map for device 0000:00:1a.0 [0x7dffd000 - 0x7dffffff]
[ 1.485154] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
[ 1.486330] DMAR: Setting identity map for device 0000:00:1d.0 [0x7dffd000 - 0x7dffffff]
[ 1.505132] DMAR: 32bit 0000:01:00.4 uses non-identity mapping
[ 1.506323] DMAR: Setting identity map for device 0000:01:00.4 [0x7ddde000 - 0x7dddefff]
[ 1.506361] DMAR: Setting identity map for device 0000:01:00.4 [0x7dff6000 - 0x7dffcfff]
~# cat /etc/modprobe.d/*
blacklist radeon
blacklist nouveau
blacklist nvidia
# This file contains a list of modules which are not supported by Proxmox VE

# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
options vfio-pci ids=10de:0ff2 disable_vga=1
options kvm ignore_msrs=1
~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
~# ./rom-parser /usr/share/kvm/NVIDIA.GridK1.4096.121001.rom
Valid ROM signature found @400h, PCIR offset 190h
PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 0ff2, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found @f600h, PCIR offset 1ch
PCIR: type 3 (EFI), vendor: 10de, device: 0ff2, class: 030000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid, Subsystem: Boot, Machine: X64
Last image
qm showcmd 120
/usr/bin/kvm -id 120 -name ws2016 -chardev 'socket,id=qmp,path=/var/run/qemu-server/120.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/120.pid -daemonize -smbios 'type=1,uuid=679d0291-0cf6-4cae-8368-93eaf688ce14' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/zvol/rpool/data/vm-120-disk-1' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,enforce,kvm=off' -m 8192 -object 'memory-backend-ram,id=ram-node0,size=8192M' -numa 'node,nodeid=0,cpus=0-3,memdev=ram-node0' -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=fuckoffhuang,kvm=off' -machine 'pc,max-ram-below-4g=1G' -device 'vmgenid,guid=fa523034-6f5e-4cda-bf37-3a910501eae0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'vfio-pci,host=26:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -chardev 'socket,path=/var/run/qemu-server/120.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -spice 'tls-port=61003,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on' -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:9e308118dd63' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/zvol/rpool/data/vm-120-disk-0,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap120i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=96:F9:E8:CD:01:A4,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'
qm> info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 8086:29c0
id ""
Bus 0, device 26, function 0:
USB controller: PCI device 8086:2937
IRQ 16.
BAR4: I/O at 0xe100 [0xe11f].
id "uhci-4"
Bus 0, device 26, function 1:
USB controller: PCI device 8086:2938
IRQ 17.
BAR4: I/O at 0xe0e0 [0xe0ff].
id "uhci-5"
Bus 0, device 26, function 2:
USB controller: PCI device 8086:2939
IRQ 18.
BAR4: I/O at 0xe0c0 [0xe0df].
id "uhci-6"
Bus 0, device 26, function 7:
USB controller: PCI device 8086:293c
IRQ 19.
BAR0: 32 bit memory at 0x92006000 [0x92006fff].
id "ehci-2"
Bus 0, device 27, function 0:
Audio controller: PCI device 8086:293e
IRQ 16.
BAR0: 32 bit memory at 0x92000000 [0x92003fff].
id "audio0"
Bus 0, device 28, function 0:
PCI bridge: PCI device 8086:3420
IRQ 0.
BUS 0.
secondary bus 1.
subordinate bus 1.
IO range [0xd000, 0xdfff]
memory range [0x90000000, 0x90ffffff]
prefetchable memory range [0x800000000, 0x809ffffff]
id "ich9-pcie-port-1"
Bus 1, device 0, function 0:
VGA controller: PCI device 10de:0ff2
IRQ 0.
BAR0: 32 bit memory at 0x90000000 [0x90ffffff].
BAR1: 64 bit prefetchable memory at 0x800000000 [0x807ffffff].
BAR3: 64 bit prefetchable memory at 0x808000000 [0x809ffffff].
BAR5: I/O at 0xd000 [0xd07f].
id "hostpci0"
Bus 0, device 28, function 1:
PCI bridge: PCI device 8086:3420
IRQ 0.
BUS 0.
secondary bus 2.
subordinate bus 2.
IO range [0xc000, 0xcfff]
memory range [0x91e00000, 0x91ffffff]
prefetchable memory range [0xfff00000, 0x000fffff]
id "ich9-pcie-port-2"
Bus 0, device 28, function 2:
PCI bridge: PCI device 8086:3420
IRQ 0.
BUS 0.
secondary bus 3.
subordinate bus 3.
IO range [0xb000, 0xbfff]
memory range [0x91c00000, 0x91dfffff]
prefetchable memory range [0xfff00000, 0x000fffff]
id "ich9-pcie-port-3"
Bus 0, device 28, function 3:
PCI bridge: PCI device 8086:3420
IRQ 0.
BUS 0.
secondary bus 4.
subordinate bus 4.
IO range [0xa000, 0xafff]
memory range [0x91a00000, 0x91bfffff]
prefetchable memory range [0xfff00000, 0x000fffff]
id "ich9-pcie-port-4"
Bus 0, device 29, function 0:
USB controller: PCI device 8086:2934
IRQ 16.
BAR4: I/O at 0xe0a0 [0xe0bf].
id "uhci-1"
Bus 0, device 29, function 1:
USB controller: PCI device 8086:2935
IRQ 17.
BAR4: I/O at 0xe080 [0xe09f].
id "uhci-2"
Bus 0, device 29, function 2:
USB controller: PCI device 8086:2936
IRQ 18.
BAR4: I/O at 0xe060 [0xe07f].
id "uhci-3"
Bus 0, device 29, function 7:
USB controller: PCI device 8086:293a
IRQ 19.
BAR0: 32 bit memory at 0x92005000 [0x92005fff].
id "ehci"
Bus 0, device 30, function 0:
PCI bridge: PCI device 8086:244e
BUS 0.
secondary bus 5.
subordinate bus 9.
IO range [0x6000, 0x9fff]
memory range [0x91000000, 0x918fffff]
prefetchable memory range [0x80a000000, 0x80a0fffff]
id "pcidmi"
Bus 5, device 1, function 0:
PCI bridge: PCI device 1b36:0001
IRQ 10.
BUS 5.
secondary bus 6.
subordinate bus 6.
IO range [0x9000, 0x9fff]
memory range [0x91600000, 0x917fffff]
prefetchable memory range [0x80a000000, 0x80a0fffff]
BAR0: 64 bit memory at 0x91800000 [0x918000ff].
id "pci.0"
Bus 6, device 8, function 0:
Class 1920: PCI device 1af4:1003
IRQ 0.
BAR0: I/O at 0x9080 [0x90bf].
BAR1: 32 bit memory at 0x91603000 [0x91603fff].
BAR4: 64 bit prefetchable memory at 0x80a00c000 [0x80a00ffff].
id "qga0"
Bus 6, device 9, function 0:
Class 1920: PCI device 1af4:1003
IRQ 0.
BAR0: I/O at 0x9040 [0x907f].
BAR1: 32 bit memory at 0x91602000 [0x91602fff].
BAR4: 64 bit prefetchable memory at 0x80a008000 [0x80a00bfff].
id "spice"
Bus 6, device 10, function 0:
SCSI controller: PCI device 1af4:1001
IRQ 0.
BAR0: I/O at 0x9000 [0x903f].
BAR1: 32 bit memory at 0x91601000 [0x91601fff].
BAR4: 64 bit prefetchable memory at 0x80a004000 [0x80a007fff].
id "virtio0"
Bus 6, device 18, function 0:
Ethernet controller: PCI device 1af4:1000
IRQ 0.
BAR0: I/O at 0x90c0 [0x90df].
BAR1: 32 bit memory at 0x91600000 [0x91600fff].
BAR4: 64 bit prefetchable memory at 0x80a000000 [0x80a003fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id "net0"
Bus 5, device 2, function 0:
PCI bridge: PCI device 1b36:0001
IRQ 11.
BUS 5.
secondary bus 7.
subordinate bus 7.
IO range [0x8000, 0x8fff]
memory range [0x91400000, 0x915fffff]
prefetchable memory range [0xfff00000, 0x000fffff]
BAR0: 64 bit memory at 0x91801000 [0x918010ff].
id "pci.1"
Bus 5, device 3, function 0:
PCI bridge: PCI device 1b36:0001
IRQ 11.
BUS 5.
secondary bus 8.
subordinate bus 8.
IO range [0x7000, 0x7fff]
memory range [0x91200000, 0x913fffff]
prefetchable memory range [0xfff00000, 0x000fffff]
BAR0: 64 bit memory at 0x91802000 [0x918020ff].
id "pci.2"
Bus 5, device 4, function 0:
PCI bridge: PCI device 1b36:0001
IRQ 10.
BUS 5.
secondary bus 9.
subordinate bus 9.
IO range [0x6000, 0x6fff]
memory range [0x91000000, 0x911fffff]
prefetchable memory range [0xfff00000, 0x000fffff]
BAR0: 64 bit memory at 0x91803000 [0x918030ff].
id "pci.3"
Bus 0, device 31, function 0:
ISA bridge: PCI device 8086:2918
id ""
Bus 0, device 31, function 2:
SATA controller: PCI device 8086:2922
IRQ 0.
BAR4: I/O at 0xffffffffffffffff [0x001e].
BAR5: 32 bit memory at 0x92004000 [0x92004fff].
id ""
Bus 0, device 31, function 3:
SMBus: PCI device 8086:2930
IRQ 10.
BAR4: I/O at 0xe000 [0xe03f].
id ""
 
Sorry, I wanted to post this thing first, but XenForo complains about post size:

Code: 
==== general system info ====

# hostname
prod-3

# pveversion --verbose
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Stepping:              4
CPU MHz:               2107.674
CPU max MHz:           2500.0000
CPU min MHz:           1200.0000
BogoMIPS:              4987.91
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-9,20-29
NUMA node1 CPU(s):     10-19,30-39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts

==== info about virtual guests ====

# qm list
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID      
      ==== data expunged ====  
       120 ws2016               running    8192              20.00 8161    

# cat /etc/pve/qemu-server/120.conf
agent: 1
balloon: 0
bios: ovmf
bootdisk: virtio0
cores: 4
efidisk0: local-zfs:vm-120-disk-1,size=128K
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=fuckoffhuang,kvm=off' -machine pc,max-ram-below-4g=1G
ide2: none,media=cdrom
memory: 8192
name: ws2016
net0: virtio=96:F9:E8:CD:01:A4,bridge=vmbr2,tag=700
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=679d0291-0cf6-4cae-8368-93eaf688ce14
sockets: 1
vga: qxl
virtio0: local-zfs:vm-120-disk-0,cache=writeback,size=20G
vmgenid: fa523034-6f5e-4cda-bf37-3a910501eae0
machine: q35
hostpci0: 26:00,pcie=1,x-vga=on,rom=NVIDIA.GridK1.4096.121001.rom

==== info about bios ====

# dmidecode -t bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
   Vendor: HP
   Version: P70
   Release Date: 07/01/2015
   Address: 0xF0000
   Runtime Size: 64 kB
   ROM Size: 8192 kB
   Characteristics:
       PCI is supported
       PNP is supported
       BIOS is upgradeable
       BIOS shadowing is allowed
       ESCD support is available
       Boot from CD is supported
       Selectable boot is supported
       EDD is supported
       5.25"/360 kB floppy services are supported (int 13h)
       5.25"/1.2 MB floppy services are supported (int 13h)
       3.5"/720 kB floppy services are supported (int 13h)
       Print screen service is supported (int 5h)
       8042 keyboard services are supported (int 9h)
       Serial services are supported (int 14h)
       Printer services are supported (int 17h)
       CGA/mono video services are supported (int 10h)
       ACPI is supported
       USB legacy is supported
       BIOS boot specification is supported
       Function key-initiated network boot is supported
       Targeted content distribution is supported
   Firmware Revision: 2.55

==== info about pci ====

# lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 [8086:0e00] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 [103c:18a8]
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a [8086:0e02] (rev 04)
   Kernel driver in use: pcieport
   Kernel modules: shpchp
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b [8086:0e03] (rev 04)
   Kernel driver in use: pcieport
   Kernel modules: shpchp
    ====similar devices next====
00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc [8086:0e28] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc [103c:18a8]
00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS [8086:0e2a] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS [103c:18a8]
00:05.4 PIC [0800]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC [8086:0e2c] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC [103c:18a8]
00:11.0 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port [8086:1d3e] (rev 05)
   Kernel driver in use: pcieport
   Kernel modules: shpchp
00:1a.0 USB controller [0c03]: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 [8086:1d2d] (rev 05)
   Subsystem: Hewlett-Packard Company C600/X79 series chipset USB2 Enhanced Host Controller [103c:18a9]
   Kernel driver in use: ehci-pci
00:1c.0 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 [8086:1d10] (rev b5)
   Kernel driver in use: pcieport
   Kernel modules: shpchp
00:1c.7 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 [8086:1d1e] (rev b5)
   Kernel driver in use: pcieport
   Kernel modules: shpchp
00:1d.0 USB controller [0c03]: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 [8086:1d26] (rev 05)
   Subsystem: Hewlett-Packard Company C600/X79 series chipset USB2 Enhanced Host Controller [103c:18a9]
   Kernel driver in use: ehci-pci
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a5)
00:1f.0 ISA bridge [0601]: Intel Corporation C600/X79 series chipset LPC Controller [8086:1d41] (rev 05)
   Kernel driver in use: lpc_ich
   Kernel modules: lpc_ich
00:1f.2 IDE interface [0101]: Intel Corporation C600/X79 series chipset 4-Port SATA IDE Controller [8086:1d00] (rev 05)
   Subsystem: Hewlett-Packard Company C600/X79 series chipset 4-Port SATA IDE Controller [103c:18a9]
   Kernel driver in use: ata_piix
   Kernel modules: pata_acpi
01:00.0 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support [103c:3306] (rev 05)
   Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
   Kernel modules: hpwdt
01:00.1 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200EH [102b:0533]
   Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
   Kernel driver in use: mgag200
   Kernel modules: mgag200
01:00.2 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging [103c:3307] (rev 05)
   Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
   Kernel driver in use: hpilo
   Kernel modules: hpilo
01:00.4 USB controller [0c03]: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller [103c:3300] (rev 02)
   Subsystem: Hewlett-Packard Company iLO4 [103c:3381]
   Kernel driver in use: uhci_hcd
02:00.0 RAID bus controller [0104]: Hewlett-Packard Company Smart Array Gen8 Controllers [103c:323b] (rev 01)
   Subsystem: Hewlett-Packard Company P420i [103c:3354]
   Kernel driver in use: hpsa
   Kernel modules: hpsa
03:00.0 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
   Subsystem: Hewlett-Packard Company OneConnect 10Gb NIC (be3) [103c:3376]
   Kernel driver in use: be2net
   Kernel modules: be2net
03:00.1 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 01)
   Subsystem: Hewlett-Packard Company OneConnect 10Gb NIC (be3) [103c:3376]
   Kernel driver in use: be2net
   Kernel modules: be2net
03:00.2 Mass storage controller [0180]: Emulex Corporation OneConnect 10Gb iSCSI Initiator (be3) [19a2:0712] (rev 01)
   Subsystem: Hewlett-Packard Company OneConnect 10Gb iSCSI Initiator (be3) [103c:3376]
   Kernel driver in use: be2iscsi
   Kernel modules: be2iscsi
03:00.3 Mass storage controller [0180]: Emulex Corporation OneConnect 10Gb iSCSI Initiator (be3) [19a2:0712] (rev 01)
   Subsystem: Hewlett-Packard Company OneConnect 10Gb iSCSI Initiator (be3) [103c:3376]
   Kernel driver in use: be2iscsi
   Kernel modules: be2iscsi
0a:00.0 Fibre Channel [0c04]: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter [10df:f100] (rev 03)
   Subsystem: Hewlett-Packard Company Saturn-X: LightPulse Fibre Channel Host Adapter [103c:3281]
   Kernel driver in use: lpfc
   Kernel modules: lpfc
1f:08.0 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [8086:0e80] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [103c:18a8]
    ====more of the similar devices====
20:00.0 PCI bridge [0604]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port in DMI2 Mode [8086:0e01] (rev 04)
   Kernel driver in use: pcieport
   Kernel modules: shpchp
    ====more root hubs, PCIe switches and DMI channels====
26:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
   Subsystem: NVIDIA Corporation GK107GL [GRID K1] [10de:1012]
   Kernel driver in use: vfio-pci
   Kernel modules: nvidiafb, nouveau
27:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
   Subsystem: NVIDIA Corporation GK107GL [GRID K1] [10de:1012]
   Kernel driver in use: vfio-pci
   Kernel modules: nvidiafb, nouveau
28:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
   Subsystem: NVIDIA Corporation GK107GL [GRID K1] [10de:1012]
   Kernel driver in use: vfio-pci
   Kernel modules: nvidiafb, nouveau
29:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
   Subsystem: NVIDIA Corporation GK107GL [GRID K1] [10de:1012]
   Kernel driver in use: vfio-pci
   Kernel modules: nvidiafb, nouveau
3f:08.0 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [8086:0e80] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [103c:18a8]
    ====even more similar devices====
3f:0e.1 Performance counters [1101]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0e30] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [103c:18a8]
   Kernel driver in use: ivbep_uncore
3f:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers [8086:0ea8] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers [103c:18a8]
3f:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers [8086:0e71] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers [103c:18a8]
3f:0f.2 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [808... (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [103c:18a8]
3f:0f.3 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [808... (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [103c:18a8]
3f:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [808... (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [103c:18a8]
3f:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [808... (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [103c:18a8]
3f:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 [8086:0eb0] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 [103c:18a8]
   Kernel driver in use: ivbep_uncore
3f:10.1 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 [8086:0eb1] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 [103c:18a8]
   Kernel driver in use: ivbep_uncore
3f:10.2 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 [8086:0eb2] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 [103c:18a8]
3f:10.3 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 [8086:0eb3] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 [103c:18a8]
3f:10.4 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 [8086:0eb4] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 [103c:18a8]
   Kernel driver in use: ivbep_uncore
3f:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 [8086:0eb5] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 [103c:18a8]
   Kernel driver in use: ivbep_uncore
3f:10.6 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 [8086:0eb6] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 [103c:18a8]
3f:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 [8086:0eb7] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 [103c:18a8]
3f:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e1d] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [103c:18a8]
3f:13.1 Performance counters [1101]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e34] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [103c:18a8]
   Kernel driver in use: ivbep_uncore
3f:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers [8086:0e81] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers [103c:18a8]
3f:13.5 Performance counters [1101]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring [8086:0e36] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring [103c:18a8]
   Kernel driver in use: ivbep_uncore
3f:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder [8086:0ec8] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder [103c:18a8]
3f:16.1 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0ec9] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [103c:18a8]
3f:16.2 System peripheral [0880]: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0eca] (rev 04)
   Subsystem: Hewlett-Packard Company Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [103c:18a8]
I'm using latest quadro drivers, but I can't specify the version, as forum flags my edit for spam (sigh...)
 
Last edited:
Just tried
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=null,-hypervisor'
Click to expand...
Still fails.
 
agent: 1
balloon: 0
bios: ovmf
bootdisk: virtio0
cores: 4
efidisk0: local-zfs:vm-120-disk-1,size=128K
cpu: cputype=host,hidden=1
ide2: none,media=cdrom
memory: 8192
name: ws2016
net0: virtio=96:F9:E8:CD:01:A4,bridge=vmbr2,tag=700
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=679d0291-0cf6-4cae-8368-93eaf688ce14
sockets: 1
vga: qxl
virtio0: local-zfs:vm-120-disk-0,cache=writeback,size=20G
vmgenid: fa523034-6f5e-4cda-bf37-3a910501eae0
machine: q35
hostpci0: 26:00,pcie=1,x-vga=on,romfile=NVIDIA.GridK1.4096.121001.rom
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,-hypervisor,hv_vendor_id=magic,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off'
qm showcmd 120
/usr/bin/kvm -id 120 -name ws2016 -chardev 'socket,id=qmp,path=/var/run/qemu-server/120.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/120.pid -daemonize -smbios 'type=1,uuid=679d0291-0cf6-4cae-8368-93eaf688ce14' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/zvol/rpool/data/vm-120-disk-1' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' -m 8192 -object 'memory-backend-ram,id=ram-node0,size=8192M' -numa 'node,nodeid=0,cpus=0-3,memdev=ram-node0' -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,-hypervisor,hv_vendor_id=magic,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' -device 'vmgenid,guid=fa523034-6f5e-4cda-bf37-3a910501eae0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'vfio-pci,host=26:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,romfile=/usr/share/kvm/NVIDIA.GridK1.4096.121001.rom' -chardev 'socket,path=/var/run/qemu-server/120.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -spice 'tls-port=61002,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on' -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:9e308118dd63' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/zvol/rpool/data/vm-120-disk-0,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap120i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=96:F9:E8:CD:01:A4,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'
That's it, I'm spent. Any further suggestions are more than welcome.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom