PCI-E option not enabled when trying Nvidia PCI-E Passthrough.

rodillo · Feb 26, 2023

Good afternoon everyone.

I'm trying to do a PCIE passthrough on an Nvidia GT 630 graphics card. I'm pretty stuck because I've tried a lot of options by following forums and youtube videos. None work for me.

The fact is that I tryied the same operation with a 3060 TI that they lent me for the test, obtaining the same result...

I indicate the specs of my server and the hardware used:
Board: Huananzhi x99 f8d supports vt-x and vt-d
CPU: x2 Xeon 2696v3 this supports vt-x and vt-d
GPU: Nvidia GT 630
Proxmox Version: 7.3-3

First of all, I can do PCIE passthrough since with other PCIE devices I have not had any problems. (like PCIE USB HUB)
Secondly, Windows recognizes both graphics cards (GT630 and RTX3060) but when initializing them it generates an error43 indicating that it is not possible to initialize the device. The funny thing is that Windows update recognizes the device and installs the drivers correctly.

I have seen that there are people who can enable a function that does not appear in my case (img-01.png). For me, this option appears disabled to me (img-02.png)

IOMMU is operational:

Code:

root@console:~# dmesg | grep -e IOMMU -e iommu -e vfio -e Intel -e bug -e Bug -e BAR
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt video=vesafb:off video=efifb:off
[    0.000000]   Intel GenuineIntel
[    0.218103] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt video=vesafb:off video=efifb:off
[    0.218189] DMAR: IOMMU enabled
[    0.524019] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
[    0.524021] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 2
[    0.524022] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 2
[    0.609928] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz (family: 0x6, model: 0x3f, stepping: 0x2)
[    0.610190] Performance Events: PEBS fmt2+, Haswell events, 16-deep LBR, full-width counters, Intel PMU driver.
[    0.781007] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[    0.781007] MMIO Stale Data CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/processor_mmio_stale_data.html for more details.
[    0.960228] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    1.033479] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.144370] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
[    1.144717] pci 0000:00:1c.3: Intel PCH root port ACS workaround enabled
[    1.145064] pci 0000:00:1c.7: Intel PCH root port ACS workaround enabled
[    1.147469] pci 0000:03:00.0: BAR 3: assigned to efifb
[    1.163495] iommu: Default domain type: Passthrough (set via kernel command line)
[    1.256479] DMAR: [Firmware Bug]: RMRR entry for device 02:00.0 is broken - applying workaround
[    1.256490] DMAR: IOMMU feature sc_support inconsistent
[    1.256491] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    1.256493] DMAR: IOMMU feature sc_support inconsistent
[    1.256494] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    1.256763] pci 0000:ff:08.0: Adding to iommu group 0
[    1.256798] pci 0000:ff:08.2: Adding to iommu group 0
[    1.256831] pci 0000:ff:08.3: Adding to iommu group 0
[    1.256950] pci 0000:ff:09.0: Adding to iommu group 1
[    1.256984] pci 0000:ff:09.2: Adding to iommu group 1
[    1.257019] pci 0000:ff:09.3: Adding to iommu group 1
[    1.257138] pci 0000:ff:0b.0: Adding to iommu group 2
[    1.257172] pci 0000:ff:0b.1: Adding to iommu group 2
[    1.257207] pci 0000:ff:0b.2: Adding to iommu group 2
[    1.257467] pci 0000:ff:0c.0: Adding to iommu group 3
[    1.257501] pci 0000:ff:0c.1: Adding to iommu group 3
[    1.257540] pci 0000:ff:0c.2: Adding to iommu group 3
[    1.257574] pci 0000:ff:0c.3: Adding to iommu group 3
[    1.257609] pci 0000:ff:0c.4: Adding to iommu group 3
[    1.257646] pci 0000:ff:0c.5: Adding to iommu group 3
[    1.257680] pci 0000:ff:0c.6: Adding to iommu group 3
[    1.257714] pci 0000:ff:0c.7: Adding to iommu group 3
[    1.257973] pci 0000:ff:0d.0: Adding to iommu group 4
[    1.258011] pci 0000:ff:0d.1: Adding to iommu group 4
[    1.258046] pci 0000:ff:0d.2: Adding to iommu group 4
[    1.258082] pci 0000:ff:0d.3: Adding to iommu group 4
[    1.258117] pci 0000:ff:0d.4: Adding to iommu group 4
[    1.258152] pci 0000:ff:0d.5: Adding to iommu group 4
[    1.258187] pci 0000:ff:0d.6: Adding to iommu group 4
[    1.258223] pci 0000:ff:0d.7: Adding to iommu group 4
[    1.258314] pci 0000:ff:0e.0: Adding to iommu group 5
[    1.258351] pci 0000:ff:0e.1: Adding to iommu group 5
[    1.258581] pci 0000:ff:0f.0: Adding to iommu group 6
[    1.258618] pci 0000:ff:0f.1: Adding to iommu group 6
[    1.258655] pci 0000:ff:0f.2: Adding to iommu group 6
[    1.258691] pci 0000:ff:0f.3: Adding to iommu group 6
[    1.258728] pci 0000:ff:0f.4: Adding to iommu group 6
[    1.258768] pci 0000:ff:0f.5: Adding to iommu group 6
[    1.258804] pci 0000:ff:0f.6: Adding to iommu group 6
[    1.258978] pci 0000:ff:10.0: Adding to iommu group 7
[    1.259017] pci 0000:ff:10.1: Adding to iommu group 7
[    1.259054] pci 0000:ff:10.5: Adding to iommu group 7
[    1.259092] pci 0000:ff:10.6: Adding to iommu group 7
[    1.259129] pci 0000:ff:10.7: Adding to iommu group 7
[    1.259276] pci 0000:ff:12.0: Adding to iommu group 8
[    1.259315] pci 0000:ff:12.1: Adding to iommu group 8
[    1.259353] pci 0000:ff:12.4: Adding to iommu group 8
[    1.259391] pci 0000:ff:12.5: Adding to iommu group 8
[    1.259593] pci 0000:ff:13.0: Adding to iommu group 9
[    1.259632] pci 0000:ff:13.1: Adding to iommu group 9
[    1.259673] pci 0000:ff:13.2: Adding to iommu group 9
[    1.259711] pci 0000:ff:13.3: Adding to iommu group 9
[    1.259750] pci 0000:ff:13.6: Adding to iommu group 9
[    1.259789] pci 0000:ff:13.7: Adding to iommu group 9
[    1.260062] pci 0000:ff:14.0: Adding to iommu group 10
[    1.260106] pci 0000:ff:14.1: Adding to iommu group 10
[    1.260145] pci 0000:ff:14.2: Adding to iommu group 10
[    1.260186] pci 0000:ff:14.3: Adding to iommu group 10
[    1.260225] pci 0000:ff:14.4: Adding to iommu group 10
[    1.260264] pci 0000:ff:14.5: Adding to iommu group 10
[    1.260303] pci 0000:ff:14.6: Adding to iommu group 10
[    1.260343] pci 0000:ff:14.7: Adding to iommu group 10
[    1.260544] pci 0000:ff:16.0: Adding to iommu group 11
[    1.260585] pci 0000:ff:16.1: Adding to iommu group 11
[    1.260628] pci 0000:ff:16.2: Adding to iommu group 11
[    1.260669] pci 0000:ff:16.3: Adding to iommu group 11
[    1.260709] pci 0000:ff:16.6: Adding to iommu group 11
[    1.260750] pci 0000:ff:16.7: Adding to iommu group 11
[    1.261007] pci 0000:ff:17.0: Adding to iommu group 12
[    1.261050] pci 0000:ff:17.1: Adding to iommu group 12
[    1.261092] pci 0000:ff:17.2: Adding to iommu group 12
[    1.261135] pci 0000:ff:17.3: Adding to iommu group 12
[    1.261177] pci 0000:ff:17.4: Adding to iommu group 12
[    1.261219] pci 0000:ff:17.5: Adding to iommu group 12
[    1.261261] pci 0000:ff:17.6: Adding to iommu group 12
[    1.261303] pci 0000:ff:17.7: Adding to iommu group 12
[    1.261477] pci 0000:ff:1e.0: Adding to iommu group 13
[    1.261520] pci 0000:ff:1e.1: Adding to iommu group 13
[    1.261565] pci 0000:ff:1e.2: Adding to iommu group 13
[    1.261608] pci 0000:ff:1e.3: Adding to iommu group 13
[    1.261652] pci 0000:ff:1e.4: Adding to iommu group 13
[    1.261742] pci 0000:ff:1f.0: Adding to iommu group 14
[    1.261787] pci 0000:ff:1f.2: Adding to iommu group 14
[    1.261910] pci 0000:7f:08.0: Adding to iommu group 15
[    1.261956] pci 0000:7f:08.2: Adding to iommu group 15
[    1.262000] pci 0000:7f:08.3: Adding to iommu group 15
[    1.262119] pci 0000:7f:09.0: Adding to iommu group 16
[    1.262164] pci 0000:7f:09.2: Adding to iommu group 16
[    1.262210] pci 0000:7f:09.3: Adding to iommu group 16
[    1.262328] pci 0000:7f:0b.0: Adding to iommu group 17
[    1.262376] pci 0000:7f:0b.1: Adding to iommu group 17
[    1.262421] pci 0000:7f:0b.2: Adding to iommu group 17
[    1.262681] pci 0000:7f:0c.0: Adding to iommu group 18
[    1.262727] pci 0000:7f:0c.1: Adding to iommu group 18
[    1.262773] pci 0000:7f:0c.2: Adding to iommu group 18
[    1.262819] pci 0000:7f:0c.3: Adding to iommu group 18
[    1.262864] pci 0000:7f:0c.4: Adding to iommu group 18
[    1.262913] pci 0000:7f:0c.5: Adding to iommu group 18
[    1.262959] pci 0000:7f:0c.6: Adding to iommu group 18
[    1.263005] pci 0000:7f:0c.7: Adding to iommu group 18
[    1.263264] pci 0000:7f:0d.0: Adding to iommu group 19
[    1.263311] pci 0000:7f:0d.1: Adding to iommu group 19
[    1.263358] pci 0000:7f:0d.2: Adding to iommu group 19
[    1.263404] pci 0000:7f:0d.3: Adding to iommu group 19
[    1.263455] pci 0000:7f:0d.4: Adding to iommu group 19
[    1.263502] pci 0000:7f:0d.5: Adding to iommu group 19
[    1.263549] pci 0000:7f:0d.6: Adding to iommu group 19
[    1.263596] pci 0000:7f:0d.7: Adding to iommu group 19
[    1.263686] pci 0000:7f:0e.0: Adding to iommu group 20
[    1.263734] pci 0000:7f:0e.1: Adding to iommu group 20
[    1.263967] pci 0000:7f:0f.0: Adding to iommu group 21
[    1.264035] pci 0000:7f:0f.1: Adding to iommu group 21
[    1.264084] pci 0000:7f:0f.2: Adding to iommu group 21
[    1.264132] pci 0000:7f:0f.3: Adding to iommu group 21
[    1.264181] pci 0000:7f:0f.4: Adding to iommu group 21
[    1.264229] pci 0000:7f:0f.5: Adding to iommu group 21
[    1.264277] pci 0000:7f:0f.6: Adding to iommu group 21
[    1.264454] pci 0000:7f:10.0: Adding to iommu group 22
[    1.264504] pci 0000:7f:10.1: Adding to iommu group 22
[    1.264553] pci 0000:7f:10.5: Adding to iommu group 22
[    1.264602] pci 0000:7f:10.6: Adding to iommu group 22
[    1.264651] pci 0000:7f:10.7: Adding to iommu group 22
[    1.264798] pci 0000:7f:12.0: Adding to iommu group 23
[    1.264848] pci 0000:7f:12.1: Adding to iommu group 23
[    1.264899] pci 0000:7f:12.4: Adding to iommu group 23
[    1.264950] pci 0000:7f:12.5: Adding to iommu group 23
[    1.265153] pci 0000:7f:13.0: Adding to iommu group 24
[    1.265203] pci 0000:7f:13.1: Adding to iommu group 24
[    1.265254] pci 0000:7f:13.2: Adding to iommu group 24
[    1.265304] pci 0000:7f:13.3: Adding to iommu group 24
[    1.265356] pci 0000:7f:13.6: Adding to iommu group 24
[    1.265408] pci 0000:7f:13.7: Adding to iommu group 24
[    1.265666] pci 0000:7f:14.0: Adding to iommu group 25
[    1.265718] pci 0000:7f:14.1: Adding to iommu group 25
[    1.265769] pci 0000:7f:14.2: Adding to iommu group 25
[    1.265820] pci 0000:7f:14.3: Adding to iommu group 25
[    1.265872] pci 0000:7f:14.4: Adding to iommu group 25
[    1.265924] pci 0000:7f:14.5: Adding to iommu group 25
[    1.265977] pci 0000:7f:14.6: Adding to iommu group 25
[    1.266030] pci 0000:7f:14.7: Adding to iommu group 25
[    1.266232] pci 0000:7f:16.0: Adding to iommu group 26
[    1.266285] pci 0000:7f:16.1: Adding to iommu group 26
[    1.266338] pci 0000:7f:16.2: Adding to iommu group 26
[    1.266390] pci 0000:7f:16.3: Adding to iommu group 26
[    1.266443] pci 0000:7f:16.6: Adding to iommu group 26
[    1.266498] pci 0000:7f:16.7: Adding to iommu group 26
[    1.266757] pci 0000:7f:17.0: Adding to iommu group 27
[    1.266811] pci 0000:7f:17.1: Adding to iommu group 27
[    1.266865] pci 0000:7f:17.2: Adding to iommu group 27
[    1.266919] pci 0000:7f:17.3: Adding to iommu group 27
[    1.266973] pci 0000:7f:17.4: Adding to iommu group 27
[    1.267027] pci 0000:7f:17.5: Adding to iommu group 27
[    1.267083] pci 0000:7f:17.6: Adding to iommu group 27
[    1.267137] pci 0000:7f:17.7: Adding to iommu group 27
[    1.267312] pci 0000:7f:1e.0: Adding to iommu group 28
[    1.267367] pci 0000:7f:1e.1: Adding to iommu group 28
[    1.267423] pci 0000:7f:1e.2: Adding to iommu group 28
[    1.267479] pci 0000:7f:1e.3: Adding to iommu group 28
[    1.267533] pci 0000:7f:1e.4: Adding to iommu group 28
[    1.267625] pci 0000:7f:1f.0: Adding to iommu group 29
[    1.267681] pci 0000:7f:1f.2: Adding to iommu group 29
[    1.267715] pci 0000:00:00.0: Adding to iommu group 30
[    1.267749] pci 0000:00:01.0: Adding to iommu group 31
[    1.267788] pci 0000:00:02.0: Adding to iommu group 32
[    1.267823] pci 0000:00:03.0: Adding to iommu group 33
[    1.267857] pci 0000:00:05.0: Adding to iommu group 34
[    1.267891] pci 0000:00:05.1: Adding to iommu group 35
[    1.267927] pci 0000:00:05.2: Adding to iommu group 36
[    1.267961] pci 0000:00:05.4: Adding to iommu group 37
[    1.268006] pci 0000:00:11.0: Adding to iommu group 38
[    1.268069] pci 0000:00:11.4: Adding to iommu group 39
[    1.268105] pci 0000:00:14.0: Adding to iommu group 40
[    1.268194] pci 0000:00:16.0: Adding to iommu group 41
[    1.268252] pci 0000:00:16.1: Adding to iommu group 41
[    1.268286] pci 0000:00:1a.0: Adding to iommu group 42
[    1.268320] pci 0000:00:1b.0: Adding to iommu group 43
[    1.268356] pci 0000:00:1c.0: Adding to iommu group 44
[    1.268390] pci 0000:00:1c.3: Adding to iommu group 45
[    1.268425] pci 0000:00:1c.7: Adding to iommu group 46
[    1.268459] pci 0000:00:1d.0: Adding to iommu group 47
[    1.268578] pci 0000:00:1f.0: Adding to iommu group 48
[    1.268638] pci 0000:00:1f.2: Adding to iommu group 48
[    1.268696] pci 0000:00:1f.3: Adding to iommu group 48
[    1.268733] pci 0000:02:00.0: Adding to iommu group 49
[    1.268826] pci 0000:03:00.0: Adding to iommu group 50
[    1.268889] pci 0000:03:00.1: Adding to iommu group 50
[    1.268924] pci 0000:05:00.0: Adding to iommu group 51
[    1.268958] pci 0000:06:00.0: Adding to iommu group 52
[    1.268992] pci 0000:80:02.0: Adding to iommu group 53
[    1.269028] pci 0000:80:05.0: Adding to iommu group 54
[    1.269065] pci 0000:80:05.1: Adding to iommu group 55
[    1.269099] pci 0000:80:05.2: Adding to iommu group 56
[    1.269134] pci 0000:80:05.4: Adding to iommu group 57
[    1.269172] pci 0000:81:00.0: Adding to iommu group 58
[    1.269375] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.378734] intel_pstate: Intel P-state driver initializing
[    2.665941] ehci-pci 0000:00:1a.0: debug port 2
[    2.691054] ehci-pci 0000:00:1d.0: debug port 2
[    4.958387] systemd[1]: Mounting Kernel Debug File System...
[    4.973498] systemd[1]: Mounted Kernel Debug File System.
[    5.021209] Disabling lock debugging due to kernel taint
[    5.416055] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input16
[    5.416222] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input17
[    5.416284] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input18
[    5.416352] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card0/input19
[    5.416507] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card0/input20
[    5.416773] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card0/input21
[    5.416875] input: HDA Intel PCH Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card0/input22
[    5.417030] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input23
[   19.530656] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.

I have inserted the dmesg command and the following appears:

Code:

[ 0.526138] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.526139] x2apic: IRQ remapping doesn't support X2APIC mode

As I indicate again, I have followed the steps of multiple tutorials, including the official documentation, but for some strange reason, the results that appear to me are completely different...

Thank you all, greetings!

leesteken · Feb 26, 2023

The important information is unreadable in your screenshots. The i440fx virtual machine type does not support PCI Express, only the q35 virtual machine type.

rodillo · Feb 27, 2023

leesteken said:
The important information is unreadable in your screenshots. The i440fx virtual machine type does not support PCI Express, only the q35 virtual machine type.

Good night leesteken i set up my vm like this, error 43 continues appearing:

As you can see the machine is in q35... ¿Any ideas? Thanks a lot for your response!!

PD: I followed step by step this tutorial: https://www.youtube.com/watch?v=ecFtSFCJqSg&t=174s&ab_channel=Harvey’sVirtualEnvironment

MadCowDzz · Jun 1, 2023

Hi rodillo, were you able to resolve this? I'm getting Error 43 in my system and I'm looking for some suggestions.

levtol · Jun 14, 2023

MadCowDzz said:
Hi rodillo, were you able to resolve this? I'm getting Error 43 in my system and I'm looking for some suggestions.

rodillo said:
Good night leesteken i set up my vm like this, error 43 continues appearing:

View attachment 47298
View attachment 47300

As you can see the machine is in q35... ¿Any ideas? Thanks a lot for your response!!

PD: I followed step by step this tutorial: https://www.youtube.com/watch?v=ecFtSFCJqSg&t=174s&ab_channel=Harvey’sVirtualEnvironment

Hello everyone, I hope my answer helps anybody.

My board: Huananzhi x99 f8d
CPU: x2 Xeon 2650v4
GPU: Nvidia GT 650 (https://www.techpowerup.com/vgabios/129106/asus-gtx650-1024-120815)

I have followed this guide entirely https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/ , but also failed with error 43. I've tried EVERY THING anyone mentioned at this post or many others.

I've read tons of forum pages and came to conclusion that the problem must be either in chinize BIOS or GPU itself (too old, maybe).
Point about GPU came to my mind because of no "UEFI" check set up in GPU-Z (pic for example):

So, I went to Huananzhi BIOS, found "CSM" menu and tried to turn off CSM, but it said that I need to to try switch Video to UEFI first. I did it and got no picture at my monitor no more. I had to clear my BIOS settings after that.

So, this was my 3rd night I was trying to solve the problem. I got tired of all of this and just took my brother's 1650Ti.

And... it worked! I switched on Video to UEFI, rebooted and saw BIOS loaded. After that, I've disabled CSM at all, turned on X2APIC, set "Active video" to "Onboard device", and "Above 4G Decoding" to Enabled.
I really do not know which one of this helped, but seams like just only "switched on Video to UEFI" can help.

GPU: Nvidia GT 630

Found just now: it seems like not every GT630 supports UEFI load: https://www.techpowerup.com/vgabios...el=GT+630&interface=&memType=&memSize=&since=

This ones do (below), but many others do not

https://www.techpowerup.com/vgabios/149981/gigabyte-gt630-2048-130823
https://www.techpowerup.com/vgabios/149461/asus-gt630-2048-130724
https://www.techpowerup.com/vgabios/145607/gigabyte-gt630-2048-130107

My GT 650 does not support UEFI load, so I think this is the core reason of error 43

levtol · Jun 14, 2023

Search

Search

PCI-E option not enabled when trying Nvidia PCI-E Passthrough.

rodillo

New Member

Attachments

leesteken

Distinguished Member

rodillo

New Member

MadCowDzz

New Member

levtol

New Member

levtol

New Member

Attachments