Quadro P2000 Passthrough Code 43

A

Andy Huang

Renowned Member
Jan 1, 2013
32
1
73
I've followed the tutorial on the wiki and I'm still having nothing but Code 43.

I've tried OVMF BIOS with PCI passthrough, as well as with PCIe passthrough, neither worked.
I've also tried SeaBIOS with PCI passthrough, as well as with PCIe passthrough, neither worked.

For each of the above, I've reinstalled the OS each time, so there's no residual from the previous attempt. Based on what I've read, my card is not UEFI (OVMF) compatible, so I'm currently on SeaBIOS setup. Here are my relevant configs, can someone please help me look over to see if I'm missing anything obvious?

Server: ASUS RS926-E7/RS8
CPU: E5-4640
GPU: nVidia Quadro P2000

Code: 
# grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off"

Code: 
# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# Generated by sensors-detect on Thu Jul  4 07:18:59 2019
# Chip drivers
coretemp
nct6775

Code: 
# dmesg | grep ecap
[    0.000000] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fa
[    0.000000] DMAR: dmar1: reg_base_addr e7ffe000 ver 1:0 cap d2078c106f0462 ecap f020fa
[    0.000000] DMAR: dmar2: reg_base_addr bbffe000 ver 1:0 cap d2078c106f0462 ecap f020fa
[    0.000000] DMAR: dmar3: reg_base_addr 8fffc000 ver 1:0 cap d2078c106f0462 ecap f020fa
[  137.354879] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900
[ 8935.406342] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900
[26434.034348] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900
[27325.951843] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900
[27622.911072] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900
[28105.791191] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900
[31274.528335] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900
[32803.835145] vfio_ecap_init: 0000:43:00.0 hiding ecap 0x19@0x900

Code: 
# find /sys/kernel/iommu_groups/ -type l
...
/sys/kernel/iommu_groups/41/devices/0000:43:00.0
/sys/kernel/iommu_groups/41/devices/0000:43:00.1
...
Very big list, I've omitted all of it except for the P2000.

Code: 
# lspci | grep NVIDIA
43:00.0 VGA compatible controller: NVIDIA Corporation GP106GL (rev a1)
43:00.1 Audio device: NVIDIA Corporation Device 10f1 (rev a1)

# lspci -n -s 43:00
43:00.0 0300: 10de:1c30 (rev a1)
43:00.1 0403: 10de:10f1 (rev a1)

Code: 
# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1c30,10de:10f1

Code: 
# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

Code: 
# cat /etc/pve/qemu-server/102.conf
bios: seabios
balloon: 0
bootdisk: virtio0
cores: 8
cpu: host,hidden=1
hostpci0: 43:00,x-vga=on,pcie=on,rombar=on,romfile=vbios-p2000.bin
ide0: local:iso/virtio-win-0.1.141.iso,media=cdrom,size=309208K
ide2: local:iso/en_windows_server_2016_updated_feb_2018_x64_dvd_11636692.iso,media=cdrom
machine: q35
memory: 32768
name: WindowsServer2016
net0: e1000=A2:9D:D8:16:B0:01,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=586a6412-f4e7-4e98-834d-7d6622fd5792
sockets: 4
startup: order=1
virtio0: local:102/vm-102-disk-1.qcow2,size=128G

Code: 
# qm showcmd 102
/usr/bin/kvm -id 102 \
    -name WindowsServer2016 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' \
    -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
    -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize \
    -smbios 'type=1,uuid=586a6412-f4e7-4e98-834d-7d6622fd5792' \
    -smp '32,sockets=4,cores=8,maxcpus=32' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
    -vga none -nographic -no-hpet \
    -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' \
    -m 32768 -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
    -device 'vfio-pci,host=43:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on,romfile=/usr/share/kvm/vbios-p2000.bin' \
    -device 'vfio-pci,host=43:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:e217b313912d' \
    -drive 'file=/var/lib/vz/template/iso/virtio-win-0.1.141.iso,if=none,id=drive-ide0,media=cdrom,aio=threads' \
    -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' \
    -drive 'file=/var/lib/vz/template/iso/en_windows_server_2016_updated_feb_2018_x64_dvd_11636692.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' \
    -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201' \
    -drive 'file=/var/lib/vz/images/102/vm-102-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on' \
    -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' \
    -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' \
    -device 'e1000,mac=A2:9D:D8:16:B0:01,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' \
    -rtc 'driftfix=slew,base=localtime' \
    -machine 'type=q35' \
    -global 'kvm-pit.lost_tick_policy=discard'

Code: 
# echo 1 > /sys/bus/pci/devices/0000:43:00.0/rom
# cat /sys/bus/pci/devices/0000:43:00.0/rom > /tmp/image.rom
# echo 0 > /sys/bus/pci/devices/0000:43:00.0/rom
#
# ./rom-parser /tmp/image.rom
Valid ROM signature found @0h, PCIR offset 1a0h
    PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1c30, class: 030000
    PCIR: revision 0, vendor revision: 1
Valid ROM signature found @e800h, PCIR offset 194h
    PCIR: type 0 (x86 PC-AT), vendor: 8086, device: 1d6b, class: 010400
    PCIR: revision 3, vendor revision: 308
    Last image

Code: 
# cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
options kvm report_ignored_msrs=0

On Windows side, it is running Windows Server 2016 Data Centre (MSDNAA license), fully activated, fresh install. Attempted to use the most recent driver `R430 U4 (431.02)` as well as the older `R390 U10 (392.53)` driver. Neither seems to make a difference; it comes up as "Windows has stopped this device because it has reported problems (code 43)".

Phew... that was long. Any help would be greatly appreciated!
 
Forgot to include this:

Code: 
# info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:29c0
      id ""
  Bus  0, device  26, function 0:
    USB controller: PCI device 8086:2937
      IRQ 16.
      BAR4: I/O at 0x6040 [0x605f].
      id "uhci-4"
  Bus  0, device  26, function 1:
    USB controller: PCI device 8086:2938
      IRQ 17.
      BAR4: I/O at 0x6060 [0x607f].
      id "uhci-5"
  Bus  0, device  26, function 2:
    USB controller: PCI device 8086:2939
      IRQ 18.
      BAR4: I/O at 0x6080 [0x609f].
      id "uhci-6"
  Bus  0, device  26, function 7:
    USB controller: PCI device 8086:293c
      IRQ 19.
      BAR0: 32 bit memory at 0xfe004000 [0xfe004fff].
      id "ehci-2"
  Bus  0, device  27, function 0:
    Audio controller: PCI device 8086:293e
      IRQ 16.
      BAR0: 32 bit memory at 0xfe000000 [0xfe003fff].
      id "audio0"
  Bus  0, device  28, function 0:
    PCI bridge: PCI device 8086:3420
      IRQ 0.
      BUS 0.
      secondary bus 1.
      subordinate bus 1.
      IO range [0x5000, 0x5fff]
      memory range [0xfb000000, 0xfcffffff]
      prefetchable memory range [0xd0000000, 0xefffffff]
      id "ich9-pcie-port-1"
  Bus  1, device   0, function 0:
    VGA controller: PCI device 10de:1c30
      IRQ 0.
      BAR0: 32 bit memory at 0xfb000000 [0xfbffffff].
      BAR1: 64 bit prefetchable memory at 0xd0000000 [0xdfffffff].
      BAR3: 64 bit prefetchable memory at 0xe0000000 [0xe1ffffff].
      BAR5: I/O at 0x5000 [0x507f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "hostpci0.0"
  Bus  1, device   0, function 1:
    Audio controller: PCI device 10de:10f1
      IRQ 17.
      BAR0: 32 bit memory at 0xfc020000 [0xfc023fff].
      id "hostpci0.1"
  Bus  0, device  28, function 1:
    PCI bridge: PCI device 8086:3420
      IRQ 0.
      BUS 0.
      secondary bus 2.
      subordinate bus 2.
      IO range [0xf000, 0x0fff]
      memory range [0xfde00000, 0xfdffffff]
      prefetchable memory range [0xf0c00000, 0xf0dfffff]
      id "ich9-pcie-port-2"
  Bus  0, device  28, function 2:
    PCI bridge: PCI device 8086:3420
      IRQ 0.
      BUS 0.
      secondary bus 3.
      subordinate bus 3.
      IO range [0xf000, 0x0fff]
      memory range [0xfdc00000, 0xfddfffff]
      prefetchable memory range [0xf0a00000, 0xf0bfffff]
      id "ich9-pcie-port-3"
  Bus  0, device  28, function 3:
    PCI bridge: PCI device 8086:3420
      IRQ 0.
      BUS 0.
      secondary bus 4.
      subordinate bus 4.
      IO range [0xf000, 0x0fff]
      memory range [0xfda00000, 0xfdbfffff]
      prefetchable memory range [0xf0800000, 0xf09fffff]
      id "ich9-pcie-port-4"
  Bus  0, device  29, function 0:
    USB controller: PCI device 8086:2934
      IRQ 16.
      BAR4: I/O at 0x60a0 [0x60bf].
      id "uhci-1"
  Bus  0, device  29, function 1:
    USB controller: PCI device 8086:2935
      IRQ 17.
      BAR4: I/O at 0x60c0 [0x60df].
      id "uhci-2"
  Bus  0, device  29, function 2:
    USB controller: PCI device 8086:2936
      IRQ 18.
      BAR4: I/O at 0x60e0 [0x60ff].
      id "uhci-3"
  Bus  0, device  29, function 7:
    USB controller: PCI device 8086:293a
      IRQ 19.
      BAR0: 32 bit memory at 0xfe005000 [0xfe005fff].
      id "ehci"
  Bus  0, device  30, function 0:
    PCI bridge: PCI device 8086:244e
      BUS 0.
      secondary bus 5.
      subordinate bus 9.
      IO range [0x1000, 0x4fff]
      memory range [0xfd000000, 0xfd9fffff]
      prefetchable memory range [0xf0000000, 0xf07fffff]
      id "pcidmi"
  Bus  5, device   1, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10.
      BUS 5.
      secondary bus 6.
      subordinate bus 6.
      IO range [0x4000, 0x4fff]
      memory range [0xfd600000, 0xfd7fffff]
      prefetchable memory range [0xf0600000, 0xf07fffff]
      BAR0: 64 bit memory at 0xfd800000 [0xfd8000ff].
      id "pci.0"
  Bus  6, device  10, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0x4000 [0x403f].
      BAR1: 32 bit memory at 0xfd660000 [0xfd660fff].
      BAR4: 64 bit prefetchable memory at 0xf0600000 [0xf0603fff].
      id "virtio0"
  Bus  6, device  18, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 23.
      BAR0: 32 bit memory at 0xfd640000 [0xfd65ffff].
      BAR1: I/O at 0x4040 [0x407f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net0"
  Bus  5, device   2, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 11.
      BUS 5.
      secondary bus 7.
      subordinate bus 7.
      IO range [0x3000, 0x3fff]
      memory range [0xfd400000, 0xfd5fffff]
      prefetchable memory range [0xf0400000, 0xf05fffff]
      BAR0: 64 bit memory at 0xfd801000 [0xfd8010ff].
      id "pci.1"
  Bus  5, device   3, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 11.
      BUS 5.
      secondary bus 8.
      subordinate bus 8.
      IO range [0x2000, 0x2fff]
      memory range [0xfd200000, 0xfd3fffff]
      prefetchable memory range [0xf0200000, 0xf03fffff]
      BAR0: 64 bit memory at 0xfd802000 [0xfd8020ff].
      id "pci.2"
  Bus  5, device   4, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10.
      BUS 5.
      secondary bus 9.
      subordinate bus 9.
      IO range [0x1000, 0x1fff]
      memory range [0xfd000000, 0xfd1fffff]
      prefetchable memory range [0xf0000000, 0xf01fffff]
      BAR0: 64 bit memory at 0xfd803000 [0xfd8030ff].
      id "pci.3"
  Bus  0, device  31, function 0:
    ISA bridge: PCI device 8086:2918
      id ""
  Bus  0, device  31, function 2:
    SATA controller: PCI device 8086:2922
      IRQ 0.
      BAR4: I/O at 0xffffffffffffffff [0x001e].
      BAR5: 32 bit memory at 0xfe006000 [0xfe006fff].
      id ""
  Bus  0, device  31, function 3:
    SMBus: PCI device 8086:2930
      IRQ 10.
      BAR4: I/O at 0x0700 [0x073f].
      id ""

The PCI device is appearing under `Bus 1, device 0, function 0` and
`Bus 1, device 0, function 1`.
 
from a first glance it seems you did everything right, sadly we do not have the hardware here to reproduce this, but you could try to use a different os (linux maybe) to see if
it a general problem or a windows/driver specific problem
 
  • Like
Reactions: Andy Huang
dcsapak said:
from a first glance it seems you did everything right, sadly we do not have the hardware here to reproduce this, but you could try to use a different os (linux maybe) to see if
it a general problem or a windows/driver specific problem
Click to expand...
The plot thickens... I tried to attach it to a newly installed OpenMediaVault 4 VM, and it wouldn't start. I tried to run the command from shwocmd manually and I get this:
Code: 
kvm: -device vfio-pci,host=43:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,x-vga=on,multifunction=on: vfio error: 0000:43:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga

I've seen in this comment that the device is accessible after reboot, so I gave that a shot right as well, but it came back the same. I'm guessing this must mean there's something else on the host configuration that I'm missing... but what could it be?

Edit: Aha, I added something in /etc/modprobe.d/vfio.conf to disable vga because it was recommended somewhere after I posted the original message. Removing that to test again now. Will report back after reboot.

Edit2: After removing that, rebooting, I was able to attach the GPU to my OpenMediaVault (Debian Stretch) and get Plex to use the GPU to do hardware transcoding:
Screen Shot 2019-07-10 at 5.19.10 AM.png

However, this disable_vga thing was added after the original post, so I'm back to square one trying to figure out why Windows doesn't like it.

Since the original post, I've also installed Windows 10 only to get to the same error code... so there's definitely something quirky on Windows side. Thanks for the debugging!
 
Last edited:
You ever get the P2000 to work? Trying to find something similar for Windows Server was going to go with a T1000 but seems it has issues, then a A6000 but maybe the P2000 if it works.
 
peril said:
did you try all of the permutations of MBAR and PCIe ? My problems have usually been some combination of primary gpu, pcie, or mbar settings, haven't had a card that didn't work so far.
Click to expand...
Good to know. In the process of looking for a GPU to pair with this: SuperMicro 6028U-TR4T But wanted to make sure whatever I buy would actually work with Windows 2019 Server. :)
 
Andy Huang said:
The plot thickens... I tried to attach it to a newly installed OpenMediaVault 4 VM, and it wouldn't start. I tried to run the command from shwocmd manually and I get this:
Code: 
kvm: -device vfio-pci,host=43:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,x-vga=on,multifunction=on: vfio error: 0000:43:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga

I've seen in this comment that the device is accessible after reboot, so I gave that a shot right as well, but it came back the same. I'm guessing this must mean there's something else on the host configuration that I'm missing... but what could it be?

Edit: Aha, I added something in /etc/modprobe.d/vfio.conf to disable vga because it was recommended somewhere after I posted the original message. Removing that to test again now. Will report back after reboot.

Edit2: After removing that, rebooting, I was able to attach the GPU to my OpenMediaVault (Debian Stretch) and get Plex to use the GPU to do hardware transcoding:
View attachment 10850

However, this disable_vga thing was added after the original post, so I'm back to square one trying to figure out why Windows doesn't like it.

Since the original post, I've also installed Windows 10 only to get to the same error code... so there's definitely something quirky on Windows side. Thanks for the debugging!
Click to expand...

Hi!

P2000 card is the only VGA in your system?

It should be 2ea VGA card: primary VGA for Proxmox , secondary for the VMs ( P2000 ).
Meaning primary VGA is for BIOS/Booting the Proxmox host (this card will initialize by the host),
secondary card will initialize by the VMs only.

Some VGA card can't handle the above, when initialized by the Proxmox host first (BIOS/Boot), then passed through to the VMs the same VGA card.

ROM file is not necessary needed, did you tried start the VM without ROM file?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back