Code 43 GPU Passthrough with Nvidia: I've tried all guides and still can't get success.

Discussion in 'Proxmox VE: Installation and configuration' started by Payden K Pringle, Dec 14, 2017.

  1. Payden K Pringle

    Payden K Pringle New Member

    Joined:
    Nov 2, 2017
    Messages:
    7
    Likes Received:
    0
    To summarize:

    1. Trying to passthrough the GPU so that I can have a Linux host to manage the computer while the users only see Windows.

    2. Hardware is the following:
    - AMD Athlon X4 845 CPU
    - ASRock FM2A88M PRO3+ Motherboard
    - 8GB of DDR3 Kingston RAM
    - 120GB SSD
    - Nvidia GT 710 GPU​

    3. Config file for the VM on ProxMox is the following:

    Code:
            bios: ovmf
            bootdisk: virtio0
            cores: 4
            cpu: host
            efidisk0: local-lvm:vm-100-disk-2,size=128K
            hostpci0: 02:00,pcie=1,x-vga=on,romfile=GK208_BIOS_FILE.bin
            machine: q35
            memory: 7168
            name: Test-Windows
            net0: virtio=96:24:CF:B4:AA:A2,bridge=vmbr0
            numa: 0
            ostype: win10
            scsihw: virtio-scsi-pci
            smbios1: uuid=f675c872-c390-4668-9c48-423f5b4ff239
            sockets: 1
            usb0: host=6-1.2 # Mouse & Keyboard
            usb1: host=2-4 # Other
            usb2: host=3-1.2.3.4 # Physical
            usb3: host=1-1.2.3.4 # USB Ports
            virtio0: local-lvm:vm-100-disk-1,cache=writeback,size=90G

    4. The BIOS bin file is parsed correctly and UEFI capable. The rom parsing instructions comes from the ProxMox Wiki.

    Code:
            root@pve-001:~/rom-parser# ./rom-parser /usr/share/kvm/GK208_BIOS_FILE.bin
                Valid ROM signature found @600h, PCIR offset 190h
                    PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 128b, class: 030000
                    PCIR: revision 0, vendor revision: 1
                Valid ROM signature found @fc00h, PCIR offset 1ch
                    PCIR: type 3 (EFI), vendor: 10de, device: 128b, class: 030000
                    PCIR: revision 3, vendor revision: 0
                        EFI: Signature Valid, Subsystem: Boot, Machine: X64
                Last image

    The Motherboard has the issue where it shadows the BIOS of the GPU in the first PCI-e slot. As far as I'm aware, providing a rom file is required in this situation for GPU passthrough.

    5. I've followed the following guides trying a mix and match of settings:

    https://forum.proxmox.com/threads/gpu-passthrough-tutorial-reference.34303/

    https://forum.proxmox.com/threads/nvidia-gpu-passthrough-assistance.27237/

    https://pve.proxmox.com/wiki/Pci_passthrough

    https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

    https://www.reddit.com/r/VFIO/comments/5sh41p/any_other_reasons_for_nvidia_driver_code_43/

    I realize this isn't Fedora 26 or RyZen, but useful info regardless:

    https://level1techs.com/article/ryzen-gpu-passthrough-setup-guide-fedora-26-windows-gaming-linux

    I could've easily missed a step in one of them given all the information I'm trying to combine here.

    6. I have the latest Nvidia Drivers installed (version 388).

    7. IOMMU is working:

    Code:
    root@pve-001:~# dmesg | grep -e IOMMU
    [    0.615425] AMD-Vi: IOMMU performance counters supported
    [    0.617069] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
    [    0.618547] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
    
    root@pve-001:~# find /sys/kernel/iommu_groups/ -type l
    /sys/kernel/iommu_groups/7/devices/0000:00:13.2
    /sys/kernel/iommu_groups/7/devices/0000:00:13.0
    /sys/kernel/iommu_groups/5/devices/0000:00:11.0
    /sys/kernel/iommu_groups/3/devices/0000:00:09.0
    /sys/kernel/iommu_groups/11/devices/0000:00:15.2
    /sys/kernel/iommu_groups/11/devices/0000:00:15.0
    /sys/kernel/iommu_groups/11/devices/0000:05:00.0
    /sys/kernel/iommu_groups/1/devices/0000:00:03.0
    /sys/kernel/iommu_groups/1/devices/0000:02:00.1
    /sys/kernel/iommu_groups/1/devices/0000:00:03.1
    /sys/kernel/iommu_groups/1/devices/0000:02:00.0
    /sys/kernel/iommu_groups/8/devices/0000:00:14.2
    /sys/kernel/iommu_groups/8/devices/0000:00:14.0
    /sys/kernel/iommu_groups/8/devices/0000:00:14.3
    /sys/kernel/iommu_groups/6/devices/0000:00:12.2
    /sys/kernel/iommu_groups/6/devices/0000:00:12.0
    /sys/kernel/iommu_groups/4/devices/0000:00:10.0
    /sys/kernel/iommu_groups/12/devices/0000:00:18.4
    /sys/kernel/iommu_groups/12/devices/0000:00:18.2
    /sys/kernel/iommu_groups/12/devices/0000:00:18.0
    /sys/kernel/iommu_groups/12/devices/0000:00:18.5
    /sys/kernel/iommu_groups/12/devices/0000:00:18.3
    /sys/kernel/iommu_groups/12/devices/0000:00:18.1
    /sys/kernel/iommu_groups/2/devices/0000:00:08.0
    /sys/kernel/iommu_groups/10/devices/0000:00:14.5
    /sys/kernel/iommu_groups/0/devices/0000:00:02.2
    /sys/kernel/iommu_groups/0/devices/0000:00:02.0
    /sys/kernel/iommu_groups/0/devices/0000:01:00.0
    /sys/kernel/iommu_groups/9/devices/0000:00:14.4

    8. The vfio-pci driver is being used correctly:

    Code:
    root@pve-001:~# lspci -k
    02:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
    Subsystem: eVga.com. Corp. GK208 [GeForce GT 710B]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau

    Other relevant information:

    Code:
    root@pve-001:~# pveversion -v
        proxmox-ve: 5.1-25 (running kernel: 4.13.4-1-pve)
        pve-manager: 5.1-35 (running version: 5.1-35/722cc488)
        pve-kernel-4.13.4-1-pve: 4.13.4-25
        libpve-http-server-perl: 2.0-6
        lvm2: 2.02.168-pve6
        corosync: 2.4.2-pve3
        libqb0: 1.0.1-1
        pve-cluster: 5.0-15
        qemu-server: 5.0-17
        pve-firmware: 2.0-3
        libpve-common-perl: 5.0-20
        libpve-guest-common-perl: 2.0-13
        libpve-access-control: 5.0-7
        libpve-storage-perl: 5.0-16
        pve-libspice-server1: 0.12.8-3
        vncterm: 1.5-2
        pve-docs: 5.1-12
        pve-qemu-kvm: 2.9.1-2
        pve-container: 2.0-17
        pve-firewall: 3.0-3
        pve-ha-manager: 2.0-3
        ksm-control-daemon: 1.2-2
        glusterfs-client: 3.8.8-1
        lxc-pve: 2.1.0-2
        lxcfs: 2.0.7-pve4
        criu: 2.11.1-1~bpo90
        novnc-pve: 0.6-4
        smartmontools: 6.5+svn4324-1
        zfsutils-linux: 0.7.2-pve1~bpo90
    But after all that, I still get Error Code 43. Note that the Microsoft Basic Display Adapter is not listed under Display Adapters.

    Is it because I'm using VirtIO? All the guides seem to be using SCSI without VirtIO. I'd like as much performance as possible, but if it's just not happening with VirtIO, I can use SCSI.

    I am getting this error when I run the VM:

    Code:
    kvm: -device vfio-pci,host=02:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,romfile=/usr/share/kvm/GK208_BIOS_FILE.bin: Failed to mmap 0000:02:00.0 BAR 3. Performance may be slow
    I have disabled the Framebuffer in the kernel settings:
    Code:
    GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=vesafb:off,efifb:off"
    But I am still seeing BIOS messages, up to a point, on the display.
     
    #1 Payden K Pringle, Dec 14, 2017
    Last edited: Dec 14, 2017
  2. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,690
    Likes Received:
    338
    do you have a dedicated gpu for the host?
    also yes, try with scsi instead of virtio, but i cannot say if this works (i guess rather not)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Payden K Pringle

    Payden K Pringle New Member

    Joined:
    Nov 2, 2017
    Messages:
    7
    Likes Received:
    0
    I do not. I thought it didn't require one. None of the guides I've found specify that the host requires a GPU when passing one through to a guest. Given that ProxMox offers a WebGUI for managing nodes, I had expected that I wouldn't need a GPU for the host.

    I will try out SCSI itself and see if that changes things.

    My research regarding Code 43 Error is that this is Nvidia's drivers noticing that it is being virtualized. I imagine using virtio *could* be what they notice?

    I've also read that using x-vga=on adds the option hv_vendor_id=proxmox to the cpu settings and that using proxmox as the hv_vendor_id doesn't always work, so I've tried manually entering the command in the Shell rather than just clicking Start on the VM and changing the hv_vendor_id from proxmox to Nvidia43FIX (or anything else really). Unfortunately that didn't solve the problem.
     
  4. Payden K Pringle

    Payden K Pringle New Member

    Joined:
    Nov 2, 2017
    Messages:
    7
    Likes Received:
    0
    Changing my config file to this (just removing pcie=1 and changing x-vga=on to x-vga=1):

    Code:
    args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=Nvidia43FIX,kvm=off'
    bios: ovmf
    bootdisk: virtio0
    cores: 4
    cpu: host
    efidisk0: local-lvm:vm-100-disk-2,size=128K
    hostpci0: 02:00,x-vga=1,romfile=GK208_BIOS_FILE.bin
    hotplug: 0
    memory: 7168
    name: Test-Windows
    machine: q35
    net0: virtio=96:24:CF:B4:AA:A2,bridge=vmbr0
    numa: 0
    onboot: 1
    ostype: win10
    scsihw: virtio-scsi-pci
    smbios1: uuid=f675c872-c390-4668-9c48-423f5b4ff239
    sockets: 1
    usb0: host=6-1.2
    usb1: host=2-4
    usb2: host=3-1.2.3.4
    usb3: host=1-1.2.3.4
    virtio0: local-lvm:vm-100-disk-1,cache=writeback,size=90G
    

    Changes the error code to the following:

    Your computer's system firmware does not include enough information to properly configure and use this device. To use this device, contact your computer manufacturer to obtain a firmware or BIOS update. (Code 35)
    I also allowed unsafe IOMMU Interupts in GRUB as it says in the Wiki:
    https://pve.proxmox.com/wiki/Pci_passthrough#IOMMU_interrupt_remapping

    I still have the "Failed to mmap" error however:
    Code:
    kvm: -device vfio-pci,host=02:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,multifunction=on,romfile=/usr/share/kvm/GK208_BIOS_FILE.bin: Failed to mmap 0000:02:00.0 BAR 3. Performance may be slow
    

    Doing `cat /proc/iomem` gives me this:

    Code:
        f0000000-fed3ffff : PCI Bus 0000:00
          f0000000-f9ffffff : PCI Bus 0000:02
            f0000000-f7ffffff : 0000:02:00.0
              f0000000-f7ffffff : vfio-pci
            f8000000-f9ffffff : 0000:02:00.0
              f9000000-f92fffff : efifb
          fd000000-fe0fffff : PCI Bus 0000:02
            fd000000-fdffffff : 0000:02:00.0
              fd000000-fdffffff : vfio-pci
            fe080000-fe083fff : 0000:02:00.1
              fe080000-fe083fff : vfio-pci
    
    Where device 02:00.0 is my GPU and 02:00.1 is the audio for it.

    And after uninstalling and reinstalling the drivers, I'm back to Code 43. sigh
     
    #4 Payden K Pringle, Dec 18, 2017
    Last edited: Dec 18, 2017
  5. lharalambus

    lharalambus New Member

    Joined:
    Dec 25, 2017
    Messages:
    1
    Likes Received:
    0
    I used to have a PCIe passthrough setup on my arch desktop. IIRC, code 43 means hypervisor detection. Have you tried specifying the kvm=off option for your cpu?
     
  6. Gerus87

    Gerus87 New Member

    Joined:
    Feb 8, 2018
    Messages:
    3
    Likes Received:
    0
    Hi all! I get the same error - code 43 then I try to pass through my GTX650Ti to Win10 guest.
    Could you help me to debug this issue?

    My hardware:
    P8Z77V-LX
    i5-3470
    32 Gb RAM
    Gigabyte GTX650Ti (GV-N65TOC-1GI)

    GRUB_DEFAULT=0
    GRUB_TIMEOUT=5
    GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
    GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=vesafb:eek:ff,efifb:eek:ff"
    GRUB_CMDLINE_LINUX=""
    GRUB_DISABLE_OS_PROBER=true
    GRUB_DISABLE_RECOVERY="true"

    [ 0.000000] ACPI: DMAR 0x00000000DD9C1B18 000080 (v01 INTEL SNB 00000001 INTL 00000001)
    [ 0.000000] DMAR: IOMMU enabled
    [ 0.030892] DMAR: Host address width 36
    [ 0.030893] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
    [ 0.030899] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f0105a
    [ 0.030900] DMAR: RMRR base: 0x000000dd934000 end: 0x000000dd945fff
    [ 0.030902] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0
    [ 0.030903] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
    [ 0.030904] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
    [ 0.031055] DMAR-IR: Enabled IRQ remapping in x2apic mode
    [ 0.489852] DMAR: No ATSR found
    [ 0.489918] DMAR: dmar0: Using Queued invalidation
    [ 0.489926] DMAR: Setting RMRR:
    [ 0.489939] DMAR: Setting identity map for device 0000:00:14.0 [0xdd934000 - 0xdd945fff]
    [ 0.489952] DMAR: Setting identity map for device 0000:00:1a.0 [0xdd934000 - 0xdd945fff]
    [ 0.489964] DMAR: Setting identity map for device 0000:00:1d.0 [0xdd934000 - 0xdd945fff]
    [ 0.489971] DMAR: Prepare 0-16MiB unity mapping for LPC
    [ 0.489977] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
    [ 0.490048] DMAR: Intel(R) Virtualization Technology for Directed I/O

    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd

    02:00.0 0300: 10de:11c6 (rev a1)
    02:00.1 0403: 10de:0e0b (rev a1)

    options vfio-pci ids=10de:11c6,10de:0e0b

    hostpci0: 02:00.0,pcie=1,x-vga=on,romfile=Gigabyte.GTX650Ti.1024.130717.rom
    bios: ovmf
    bootdisk: virtio0
    cores: 2
    cpu: host
    efidisk0: local:114/vm-114-disk-2.qcow2,size=128K
    ide2: none,media=cdrom
    machine: q35
    memory: 4096
    name: w10x64
    net0: virtio=C2:C0:49:2E:09:1D,bridge=vmbr0
    numa: 0
    ostype: win10
    scsihw: virtio-scsi-pci
    smbios1: uuid=f1777767-6bb6-4f64-a350-cba0104f8317
    sockets: 1
    virtio0: local:114/vm-114-disk-1.qcow2,size=50G

    lspci -v shows "Kernel driver in use: vfio-pci"

    I get an error 43 before installing the driver. When I install the driver VM does not boot.

    Thank you!
     
  7. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    3,690
    Likes Received:
    338
    try with:
    Code:
    hostpci0: 02:00,pcie=1,x-vga=on,romfile=Gigabyte.GTX650Ti.1024.130717.rom
    
    instead (so that the audio part also gets passed through)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Gerus87

    Gerus87 New Member

    Joined:
    Feb 8, 2018
    Messages:
    3
    Likes Received:
    0
    Thank you to fast reply!
    I tried, but it not help.
    Снимок экрана от 2018-02-09 15-19-32.png

    proxmox-ve: 4.4-105 (running kernel: 4.4.98-5-pve)
    pve-manager: 4.4-22 (running version: 4.4-22/2728f613)
    pve-kernel-4.4.35-1-pve: 4.4.35-77
    pve-kernel-4.4.98-5-pve: 4.4.98-105
    lvm2: 2.02.116-pve3
    corosync-pve: 2.4.2-2~pve4+1
    libqb0: 1.0.1-1
    pve-cluster: 4.0-54
    qemu-server: 4.0-115
    pve-firmware: 1.1-11
    libpve-common-perl: 4.0-96
    libpve-access-control: 4.0-23
    libpve-storage-perl: 4.0-76
    pve-libspice-server1: 0.12.8-2
    vncterm: 1.3-2
    pve-docs: 4.4-4
    pve-qemu-kvm: 2.9.1-6~pve4
    pve-container: 1.0-104
    pve-firewall: 2.0-33
    pve-ha-manager: 1.0-41
    ksm-control-daemon: 1.2-1
    glusterfs-client: 3.5.2-2+deb8u3
    lxc-pve: 2.0.7-4
    lxcfs: 2.0.6-pve1
    criu: 1.6.0-1
    novnc-pve: 0.5-9
    smartmontools: 6.5+svn4324-1~pve80

    Linux pve 4.4.98-5-pve #1 SMP PVE 4.4.98-105 (Thu, 18 Jan 2018 10:20:04 +0100) x86_64 GNU/Linux
     
  9. Gerus87

    Gerus87 New Member

    Joined:
    Feb 8, 2018
    Messages:
    3
    Likes Received:
    0
    Also I tried to patch nvidia drivers, I installed Window 8...nothing helps
     
  10. doctaweeks

    doctaweeks New Member

    Joined:
    Mar 23, 2018
    Messages:
    1
    Likes Received:
    0
    It used to be sufficient to specify kvm=off to hide KVM from the NVIDIA driver. (This masked the CPUID bits the NVIDIA driver was specifically checking.) However, drivers after 387.34 (390.x, maybe others) are making another check and this mask isn't enough anymore. You can try an earlier driver if possible.
     
  11. Aist

    Aist New Member

    Joined:
    Mar 21, 2018
    Messages:
    1
    Likes Received:
    0
    try github.com/sk1080/nvidia-kvm-patcher
    I have the driver 391.35 for 03-04-2018 it's the last one
     
  12. vinnyvinny

    vinnyvinny New Member

    Joined:
    Dec 11, 2017
    Messages:
    8
    Likes Received:
    0
    Hello, I had the same problem, after 16 hours, multiplicity how to (e.g. nvidia-kvm-patcher, msi mode). The problem was resolved when i try passthrough GeForce 960 instead GeForce 1050. Nothing change only video card, that all.
    The latest nvidia driver worked!
    960 pass trugh.png

    gpu_conf.png
     
  13. Eugene Karmazin

    Eugene Karmazin New Member

    Joined:
    May 21, 2018
    Messages:
    1
    Likes Received:
    0
    @vinnyvinny, can you provide steps how you did that, please. I'm still facing the same issue.
     
  14. vinnyvinny

    vinnyvinny New Member

    Joined:
    Dec 11, 2017
    Messages:
    8
    Likes Received:
    0

    Attached Files:

  15. vinnyvinny

    vinnyvinny New Member

    Joined:
    Dec 11, 2017
    Messages:
    8
    Likes Received:
    0
    Code:
    cat /etc/modprobe.d/vfio.conf
    options vfio-pci ids=10de:1c03,10de:10f1
    
    vm conf
    hostpci0: 04:00.0,x-vga=on,pcie=1
    hostpci1: 04:00.1,pcie=1
    machine: q35
    cpu: host
    balloon: 0
    bios: ovmf
    bootdisk: scsi0
    
    cat /etc/default/grub
    GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifb:off pcie_acs_override=downstream"
    
    my conf
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice