pcie passthrough nvidia error 43-> HOw to get rid of ->single display (solved)

[boxieee]

nano /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init"
GRUB_CMDLINE_LINUX=""

(tried this too GRUB_CMDLINE_LINUX_DEFAULT="textonly quiet intel_iommu=on video=vesafbff video=efifbff video=simplefbff")
and
(tried this too GRUB_CMDLINE_LINUX_DEFAULT="textonly quiet intel_iommu=on video=efifbff")

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1287,10de:0e0f disable_vga=1

/etc/modules
# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1

bios: ovmf
boot:
cores: 2
cpu: host,hidden=1,flags=+pcid
efidisk0: Vm_Files:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1,romfile=GK208.rom,x-vga=1
ide0: Vm_Files:vm-100-disk-1,size=130G
machine: pc-q35-7.1
memory: 4096
meta: creation-qemu=7.1.0,ctime=1676384472
name: win10-1
net0: e1000=4E:7E:42:74:92:07,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=96372ad3-a23a-499d-852e-6c3626095676
sockets: 1
tpmstate0: Vm_Files:vm-100-disk-2,size=4M,version=v2.0
vmgenid: 929baed7-89e2-4248-ab9e-d227cea5392c
vmstatestorage: Vm_Files

before star of vm win10
dmesg | grep -i vfio
[ 4.617534] VFIO - User Level meta-driver version: 0.3
[ 4.629385] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=nonewns=io+mem
[ 4.649214] vfio_pci: add [10de:1287[ffffffff:ffffffff]] class 0x000000/00000000
[ 4.669169] vfio_pci: add [10de:0e0f[ffffffff:ffffffff]] class 0x000000/00000000


after start of vm win10
dmesg | grep -i vfio
[ 4.617534] VFIO - User Level meta-driver version: 0.3
[ 4.629385] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=nonewns=io+mem
[ 4.649214] vfio_pci: add [10de:1287[ffffffff:ffffffff]] class 0x000000/00000000
[ 4.669169] vfio_pci: add [10de:0e0f[ffffffff:ffffffff]] class 0x000000/00000000
[ 175.862562] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 175.863842] vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
[ 175.881707] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)


dmesg | grep -e DMAR -e IOMMU
[ 0.008823] ACPI: DMAR 0x00000000B70E76C0 000070 (v01 INTEL SKL 00000001 INTL 00000001)
[ 0.008842] ACPI: Reserving DMAR table memory at [mem 0xb70e76c0-0xb70e772f]
[ 0.030274] DMAR: IOMMU enabled
[ 0.085020] DMAR: Host address width 39
[ 0.085021] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.085025] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.085027] DMAR: RMRR base: 0x000000c61a4000 end: 0x000000c61c3fff
[ 0.085029] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0
[ 0.085030] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[ 0.085031] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.086377] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.293641] DMAR: No ATSR found
[ 0.293642] DMAR: No SATC found
[ 0.293645] DMAR: dmar0: Using Queued invalidation
[ 0.294399] DMAR: Intel(R) Virtualization Technology for Directed I/O

lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)
00:1c.7 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #8 (rev f1)
00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation B150 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.3 Audio device: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 730] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

Error code 43. any ideas??

 

[leesteken]

bios: ovmf
boot:
cores: 2
cpu: host,hidden=1,flags=+pcid
efidisk0: Vm_Files:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1,romfile=GK208.rom,x-vga=1
ide0: Vm_Files:vm-100-disk-1,size=130G
machine: pc-q35-7.1

Try pc-q35-6.2 instead. I think Windows drivers don't like the new virtual motherboard version's PCIe-layout or something. Windows might need a reboot or two after such a change.

memory: 4096
meta: creation-qemu=7.1.0,ctime=1676384472
name: win10-1
net0: e1000=4E:7E:42:74:92:07,bridge=vmbr0,firewall=1
numa: 0
ostype: win11

Use win10 instead, if you are using Windows 10.

scsihw: virtio-scsi-single
smbios1: uuid=96372ad3-a23a-499d-852e-6c3626095676
sockets: 1
tpmstate0: Vm_Files:vm-100-disk-2,size=4M,version=v2.0
vmgenid: 929baed7-89e2-4248-ab9e-d227cea5392c
vmstatestorage: Vm_Files

before star of vm win10

 

[boxieee]

Try pc-q35-6.2 instead. I think Windows drivers = done



Win10 selected

Removed nvidia installed again

2 reboots nothing

Reboot whole PC still same

 

[leesteken]

Try pc-q35-6.2 instead. I think Windows drivers = done



Win10 selected

Removed nvidia installed again

2 reboots nothing

Reboot whole PC still same

Sometimes args: -machine 'max-ram-below-4g=1G,kernel_irqchip=on helps for people with NVidia GPUs. Please also search the forum for NVidia work-arounds.

 

[boxieee]

nope does not work, cereated a new vm

balloon: 0
bios: ovmf
boot: order=ide0;ide2;net0
cores: 4
cpu: host
efidisk0: Vm_Files:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1,x-vga=1
ide0: Vm_Files:vm-102-disk-1,size=100G
ide2: local:iso/Windows1022h2.iso,media=cdrom,size=4671872K
machine: pc-q35-6.2
memory: 4096
meta: creation-qemu=7.1.0,ctime=1676546121
name: win10-2
net0: e1000=D6:E5:BFE:2F:23,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=0acc14fc-1f21-4ed2-85e6-f0e088db1c14
sockets: 1
vmgenid: 8a502a0e-fee1-475f-96cf-2e01c0e1fe94

on first install all okay on reboot error 43

back before i am trying proxmox i used hyper v with win10 on this system and i had to punch in like 10 lines in powershell and thats it reboot vm and gpu works. it is so difficult with proxmox??

 

[boxieee]

Finally Worked
Try 1 - go to the bios and set igpu turn on , then primary display is cpu display . rebooted and without any change got display of vm on the monitor and no more error code 43

now let me try this with without connecting any monitors

 

[leesteken]

Finally Worked
Try 1 - go to the bios and set igpu turn on , then primary display is cpu display . rebooted and without any change got display of vm on the monitor and no more error code 43

now let me try this with without connecting any monitors

If that's the solution then initcall_blacklist=sysfb_init is not applied correctly. What is the output of cat /proc/cmdline? Note that Proxmox has two bootloaders.

 

[boxieee]

Try 2
standalone restart of computer and 730 nivida works in windows 10 , which means proxmox has error in switching off proxmox display for proxmox cli and giving it to the vm which is turned on.

For my setup primary gpu must be selected in pcie nvidia for error 43 to go away and igpu on, primary display cpu(intel)onboard

 

[boxieee]

root@prox:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.85-1-pve root=/dev/mapper/pve-root ro textonly quiet intel_iommu=on video=vesafbff video=efifbff video=simplefbff iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init

 

[boxieee]

tried below with igpu off But no success. the display from nvidia goes off and error code 43

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nomodeset pcie_acs_override=downstream,multifunction nofb textonly initcall_blacklist=sysfb_init"3


# block NVIDIA driver in pve-blacklist
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

# IOMMU interrupt remapping
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
# ignored RDMSR warning (Nvidia GPU ONLY)
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf

 

[boxieee]

tried below with igpu off But no success. the display from nvidia goes off and error code 43

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nomodeset pcie_acs_override=downstream,multifunction nofb textonly initcall_blacklist=sysfb_init"3


# block NVIDIA driver in pve-blacklist
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

# IOMMU interrupt remapping
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
# ignored RDMSR warning (Nvidia GPU ONLY)
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf

 

[boxieee]

even after setting pci-stub why is proxmox still grabbing nvidia vga???

[ 0.269356] pci 0000:01:00.0: vgaarb: setting as boot VGA device
[ 0.269356] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=i o+mem,locks=none
[ 0.269356] pci 0000:01:00.0: vgaarb: bridge control possible
[ 0.269356] vgaarb: loaded

 

[boxieee]

OK got it working. took the vgarom file . opened it in the editor found 55AA in hex and on right where nvidia text starts above that deleted all saved and used that rom file and it works.

 

[lmiksche]

Can you please describe the process with the ROM? I all ower sudden happen to have the same error with kind of same symptoms on my DELL R730, there is surely no igpu stetting anywhere and on the old proxmox install it worked no problem. Now It worked on 1st boot then refused to work. It came op once when I switched between the q35 versions, then code 43 after reboot again.

 

[boxieee]

process for nvidia rom i did was to get external display