Nvidia 5000 series passthrough , failing miserably.

adolfotregosa · Apr 24, 2025

I'm trying..... My 4060ti 8GB works perfectly but I wanted to have 16GB of vram so I got the 5060ti 16GB.
I have never had an nvidia that I could not passthough from 10XX, 20xx, 30xx and 40xx but I cannot with this 5060ti. With original vbios, modded vbios. uefi...csm... The monitor turns on but I get no image. SSHing into the VM the nvidia driver is loaded but without output. nvidia-smi confirms no monitor detected. I've yet to see the ovmf bios screen !
Tried proxmox kernel 6.8, 611 and 6.14, 4G decoding and rebar on/off in bios, no dice.

Surely I'm not alone... am I ?

My hookscript still brings back proxmox cli when I stop the VM. I even try enabling the iGPU and leaving the 5060TI alone, no dice. Monitor "turns on" but I get no image.

Any ideas??

adolfotregosa · Apr 25, 2025

Well, progress but it's a compromise. I'm using linux guest btw.
I'm not able to see tianocore bios screen so you have to prepare the vm using some virtual gpu first and install the nvidia driver.

You have to untick rombar when adding the gpu!!! As soon the nvidia driver loads inside the VM, image will show up.
I'm able to recover proxmox terminal after vm shutdown using hookscript to unbind from vfio-pci and rebind nvidia. (And also to unload the nvidia driver in proxmox before the vm starts).

The most important option is to load nvidia_drm with modeset=1 on both proxmox and the linux vm!

so put in some conf file in e.g. /etc/modprobe.d/nvidia.conf on both proxmox and your linux guest:

options nvidia_drm modeset=1

and update your initramfs, e.g. update-initramfs -u -k all

Unfortunately this is not 100% reliable. Sometimes the machine freezes when powering off the VM. Some reset problem OR the nvidia driver is still very problematic for the 5000 series making all sort of problems.

adolfotregosa · Wednesday at 13:27

For the 5060 series there's this update directly from nvidia. I now finally have VM bios screen. Rombar now has to be enabled.

number201724 · Wednesday at 17:50

adolfotregosa said:
For the 5060 series there's this update directly from nvidia. I now finally have VM bios screen. Rombar now has to be enabled.

Can you passthrough? I use vfio passthrough and when I install the nvidia driver (windows 11) it gives me a black screen.I'm using RTX 5060. I installed the firmware update to no avail.

I don't know if it is a graphics card design or firmware issue of the 50 series.
When I switched back to a 40 series graphics card all the problems disappeared.
It will get stuck in the Windows BOOT stage, and nvlddmkm will always stop at GetBusData, an infinite loop.

my linux kernel is one I modified to support aperf/mperf and arch-lbr, pmu, etc. I use qemu directly

Code:

PROCESS ffffcb0fa46fe140
    SessionId: none  Cid: 03c0    Peb: 2cd97ab000  ParentCid: 0348
    DirBase: 75fd05000  ObjectTable: ffff9d0f87568c40  HandleCount:  73.
    Image: csrss.exe
    VadRoot ffffcb0fa4766560 Vads 43 Clone 0 Private 220. Modified 4. Locked 0.
    DeviceMap ffff9d0f8726cc40
    Token                             ffff9d0f8767b880
    ElapsedTime                       00:01:23.400
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         43680
    QuotaPoolUsage[NonPagedPool]      6208
    Working Set Sizes (now,min,max)  (1133, 50, 345) (4532KB, 200KB, 1380KB)
    PeakWorkingSetSize                1084
    VirtualSize                       2101316 Mb
    PeakVirtualSize                   2101316 Mb
    PageFaultCount                    1166
    MemoryPriority                    BACKGROUND
    BasePriority                      13
    CommitCharge                      298

        THREAD ffffcb0fa47080c0  Cid 03c0.03c4  Teb: 0000002cd97ac000 Win32Thread: ffffcb0fa4760060 RUNNING on processor 2
        Not impersonating
        DeviceMap                 ffff9d0f8726cc40
        Owning Process            ffffcb0fa46fe140       Image:         csrss.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      607            Ticks: 3624 (0:00:00:56.625)
        Context Switch Count      257            IdealProcessor: 2            
        UserTime                  00:00:00.000
        KernelTime                00:00:57.062
        Win32 Start Address 0x00007ff677ed1010
        Stack Init ffffdd82c6445c70 Current ffffdd82c64441a0
        Base ffffdd82c6446000 Limit ffffdd82c6440000 Call 0000000000000000
        Priority 16  BasePriority 13  Priority Floor 16  IoPriority 2  PagePriority 5
        Child-SP          RetAddr               : Args to Child                                                           : Call Site
        ffffdd82`c64444e0 fffff803`387d224f     : 90ebea4d`74e91674 823be73c`d2a52225 00730073`0061006c 00000000`00000000 : nt!KzLowerIrql+0x22
        ffffdd82`c6444510 fffff803`3c362640     : 00000000`0000000f 00000000`00000000 ffffffff`ffffffd1 fffff803`00000002 : pci!PciBusInterface_GetBusData+0x38f
        ffffdd82`c64445e0 fffff803`3c5c79bc     : 00000000`00000000 00000000`0008a000 ffffcb0f`a4815000 00000000`00000002 : nvlddmkm+0xf2640
        ffffdd82`c6444650 fffff803`3c9c9167     : 00000000`00000fff 00000000`0008a000 00000000`00000000 00000000`00000000 : nvlddmkm+0x3579bc
        ffffdd82`c64446a0 fffff803`3c9c63f7     : fffff803`3d66dbc0 ffffcb0f`a4815000 00000000`00000000 ffffcb0f`a4815000 : nvlddmkm+0x759167
        ffffdd82`c6444710 fffff803`3c37da01     : 00000000`00000000 ffffcb0f`a4815000 ffffdd82`c64447a9 00000000`00000000 : nvlddmkm+0x7563f7
        ffffdd82`c6444740 fffff803`3c67dad4     : ffffcb0f`a4815000 ffffcb0f`a4815000 ffffcb0f`a4815000 00000000`00000001 : nvlddmkm+0x10da01
        ffffdd82`c6444810 fffff803`3c3b17d8     : ffffcb0f`a4815000 ffffcb0f`a4815000 ffffcb0f`a4815000 00000000`00000000 : nvlddmkm+0x40dad4
        ffffdd82`c6444870 fffff803`3c357f9c     : ffffcb0f`a4815000 00000000`00000000 00000000`00000000 00000000`00000000 : nvlddmkm+0x1417d8
        ffffdd82`c64448b0 fffff803`3c357bc3     : ffffdd82`c6445628 00000070`00000000 00000000`00000000 00000000`00000000 : nvlddmkm+0xe7f9c
        ffffdd82`c6444940 fffff803`3db9e890     : 01000000`00100000 ffffcb0f`a3db4000 ffffdd82`c6444b30 00000000`00000001 : nvlddmkm+0xe7bc3
        ffffdd82`c6444a30 fffff803`3db18084     : ffffcb0f`a3db4000 ffffcb0f`a3db4000 ffffcb0f`a3db4000 ffffdd82`c6444e50 : nvlddmkm!nvDumpConfig+0x84ab20
        ffffdd82`c6444d50 fffff803`3dbe651d     : 00000000`00000000 ffffdd82`c6445628 ffffcb0f`00000003 00000000`00000000 : nvlddmkm!nvDumpConfig+0x7c4314
        ffffdd82`c6445330 fffff803`3dbf66cb     : fffff803`3dbe643e ffffcb0f`a3db4000 ffffcb0f`a3b838c0 00000000`00000240 : nvlddmkm!nvDumpConfig+0x8927ad
        ffffdd82`c64453f0 fffff803`3d99459e     : 00000000`00000000 ffffcb0f`a3b838c0 ffffdd82`c64454b0 fffff803`37be2df2 : nvlddmkm!nvDumpConfig+0x8a295b
        ffffdd82`c6445430 fffff803`37dbd7dc     : ffffcb0f`a3bd2848 ffffdd82`c6445480 00000000`00000000 00000000`00000000 : nvlddmkm!nvDumpConfig+0x64082e
        ffffdd82`c6445470 fffff803`37bdaf78     : ffffcb0f`a42ac010 ffffdd82`c64455d0 ffffcb0f`a3bd1180 00000000`00000000 : dxgkrnl!DpiDxgkDdiStartDevice+0x80
        ffffdd82`c64454d0 fffff803`37bdcecd     : 00000000`00000000 00000000`00000000 ffffcb0f`a3bd1180 00000000`00000000 : dxgkrnl!DpiFdoStartAdapter+0xac0
        ffffdd82`c64456d0 fffff803`37bdcb50     : ffffcb0f`a3bd1180 ffffdd82`c6445820 00000000`00000000 ffffcb0f`a3b838c0 : dxgkrnl!DpiFdoStartNonLdaAdapter+0x85
        ffffdd82`c6445720 fffff803`37bdc473     : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000000 : dxgkrnl!DpiFdoStartAdapterThreadImpl+0x604
        ffffdd82`c64458a0 fffff803`37d83277     : fffff803`37afdc00 00000000`00000000 00000000`00000000 00000000`00000000 : dxgkrnl!DpiFdoStartAdapterThread+0x33
        ffffdd82`c64458d0 fffff803`37d82e8a     : ffffffff`fffffffd 00000000`00000000 00000000`00000000 fffff803`37988076 : dxgkrnl!DpiSessionCreateCallback+0x6b
        ffffdd82`c6445910 fffff803`3798442a     : 00000000`00000000 00000000`000000ec 00000000`00000000 00000000`000000ec : dxgkrnl!DxgkNotifySessionStateChange+0x5a
        ffffdd82`c6445960 fffff803`378b4047     : 00000000`00000000 00000000`00000000 00000000`00000000 fffff803`378b54bd : watchdog!SMgrNotifySessionChange+0x4a
        ffffdd82`c6445990 fffff803`3bd6d883     : 00000217`23106db4 00000000`000000f0 fffff803`3bfbcdb8 fffff803`3bfbcdc0 : win32k!SysEntrySMgrNotifySessionChange+0x17
        ffffdd82`c64459c0 fffff803`3bd6d815     : 00000000`00000040 00000000`00000000 00000000`00000000 fffff803`3c0093e2 : win32kbase!SysEntrySMgrNotifySessionChange+0x33
        ffffdd82`c64459f0 fffff803`3bd6da85     : 00000000`00000000 00000000`00000001 ffffbf54`232ca04c 00000000`00000000 : win32kbase!DrvNotifySessionStateChange+0x45
        ffffdd82`c6445a20 fffff803`3be9661c     : 00000000`00000001 00000000`00000001 00000000`00000000 00000000`00000000 : win32kbase!InitializeGreCSRSS+0xd
        ffffdd82`c6445a50 fffff803`378b4ce4     : 00000000`000000f0 ffffdd82`c6445b60 00000000`001f0003 00000000`000000ec : win32kbase!Win32kBaseUserInitialize+0x21c
        ffffdd82`c6445ab0 fffff803`a60b8655     : ffffcb0f`a47080c0 00000000`00000030 ffffcb0f`a47080c0 00000000`001f0003 : win32k!NtUserInitialize+0x34
        ffffdd82`c6445ae0 00007ffd`6610a5b4     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffffdd82`c6445ae0)
        0000002c`d983f438 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffd`6610a5b4

GetBusData can never get any data, it always stops here, and no output is displayed.

Code:

10: kd> g
Breakpoint 0 hit
pci!PciBusInterface_GetBusData:
fffff800`47081ec0 48895c2408      mov     qword ptr [rsp+8],rbx
3: kd> r
rax=fffff80047081ec0 rbx=000000000000000f rcx=ffff808f5b5ea4b0
rdx=0000000000000000 rsi=0000000000ff0002 rdi=ffff808f5ebf3dd0
rip=fffff80047081ec0 rsp=ffffa4058c0ec5d8 rbp=000000000000ff00
 r8=ffffa4058c0ec610  r9=0000000000000fff r10=0000000000000000
r11=ffffa4058c0ec620 r12=ffff808f5eafbf00 r13=fffff8004d3ddbc0
r14=0000000000000000 r15=0000000000000fff
iopl=0         nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00040246
pci!PciBusInterface_GetBusData:
fffff800`47081ec0 48895c2408      mov     qword ptr [rsp+8],rbx ss:0018:ffffa405`8c0ec5e0=000000000000000f
3: kd> dq rsp+20h
ffffa405`8c0ec5f8  fffff800`00000003 00000000`00000004
ffffa405`8c0ec608  00000000`00040246 ffffa405`8c0ec620
ffffa405`8c0ec618  ffff4a4e`afdf0780 00000000`00000000
ffffa405`8c0ec628  00000000`00000000 00000000`00000000
ffffa405`8c0ec638  ffffa405`8c0ec6dc 00000000`00000fff
ffffa405`8c0ec648  fffff800`4c3379bc 00000000`00000000
ffffa405`8c0ec658  00000000`0008a000 ffff808f`5e12e000
ffffa405`8c0ec668  00000000`00000002 ffffffff`ffffffd1
3: kd> k
 # Child-SP          RetAddr               Call Site
00 ffffa405`8c0ec5d8 fffff800`4c0d2640     pci!PciBusInterface_GetBusData
01 ffffa405`8c0ec5e0 fffff800`4c3379bc     nvlddmkm+0xf2640
02 ffffa405`8c0ec650 fffff800`4c739167     nvlddmkm+0x3579bc
03 ffffa405`8c0ec6a0 fffff800`4c7363f7     nvlddmkm+0x759167
04 ffffa405`8c0ec710 fffff800`4c0eda01     nvlddmkm+0x7563f7
05 ffffa405`8c0ec740 fffff800`4c3edad4     nvlddmkm+0x10da01
06 ffffa405`8c0ec810 fffff800`4c1217d8     nvlddmkm+0x40dad4
07 ffffa405`8c0ec870 fffff800`4c0c7f9c     nvlddmkm+0x1417d8
08 ffffa405`8c0ec8b0 fffff800`4c0c7bc3     nvlddmkm+0xe7f9c
09 ffffa405`8c0ec940 fffff800`4d90e890     nvlddmkm+0xe7bc3
0a ffffa405`8c0eca30 fffff800`4d888084     nvlddmkm!nvDumpConfig+0x84ab20
0b ffffa405`8c0ecd50 fffff800`4d95651d     nvlddmkm!nvDumpConfig+0x7c4314
0c ffffa405`8c0ed330 fffff800`4d9666cb     nvlddmkm!nvDumpConfig+0x8927ad
0d ffffa405`8c0ed3f0 fffff800`4d70459e     nvlddmkm!nvDumpConfig+0x8a295b
0e ffffa405`8c0ed430 fffff800`4665d7dc     nvlddmkm!nvDumpConfig+0x64082e
0f ffffa405`8c0ed470 fffff800`4647af78     dxgkrnl!DpiDxgkDdiStartDevice+0x80
10 ffffa405`8c0ed4d0 fffff800`4647cecd     dxgkrnl!DpiFdoStartAdapter+0xac0
11 ffffa405`8c0ed6d0 fffff800`4647cb50     dxgkrnl!DpiFdoStartNonLdaAdapter+0x85
12 ffffa405`8c0ed720 fffff800`4647c473     dxgkrnl!DpiFdoStartAdapterThreadImpl+0x604
13 ffffa405`8c0ed8a0 fffff800`46623277     dxgkrnl!DpiFdoStartAdapterThread+0x33
14 ffffa405`8c0ed8d0 fffff800`46622e8a     dxgkrnl!DpiSessionCreateCallback+0x6b
15 ffffa405`8c0ed910 fffff800`4622442a     dxgkrnl!DxgkNotifySessionStateChange+0x5a
16 ffffa405`8c0ed960 fffff800`46154047     watchdog!SMgrNotifySessionChange+0x4a
17 ffffa405`8c0ed990 fffff800`4b15d883     win32k!SysEntrySMgrNotifySessionChange+0x17
18 ffffa405`8c0ed9c0 fffff800`4b15d815     win32kbase!SysEntrySMgrNotifySessionChange+0x33
19 ffffa405`8c0ed9f0 fffff800`4b15da85     win32kbase!DrvNotifySessionStateChange+0x45
1a ffffa405`8c0eda20 fffff800`4b28661c     win32kbase!InitializeGreCSRSS+0xd
1b ffffa405`8c0eda50 fffff800`46154ce4     win32kbase!Win32kBaseUserInitialize+0x21c
1c ffffa405`8c0edab0 fffff800`b48b8655     win32k!NtUserInitialize+0x34
1d ffffa405`8c0edae0 00007ffb`613ca5b4     nt!KiSystemServiceCopyEnd+0x25
1e 0000003b`c65bf808 00007ffb`61284a6a     0x00007ffb`613ca5b4
1f 0000003b`c65bf810 000001dd`cfaa08cc     0x00007ffb`61284a6a
20 0000003b`c65bf818 00000000`00000030     0x000001dd`cfaa08cc
21 0000003b`c65bf820 0000003b`c65bf879     0x30
22 0000003b`c65bf828 00000000`001f0003     0x0000003b`c65bf879
23 0000003b`c65bf830 00000000`00000000     0x1f0003
3: kd> r
rax=0000000000000000 rbx=000000000000000f rcx=0000000000000000
rdx=ffff808f590f0960 rsi=0000000000ff0002 rdi=ffff808f5ebf3dd0
rip=fffff8004708226b rsp=ffffa4058c0ec5d8 rbp=000000000000ff00
 r8=fffff800b460b150  r9=0000000000000000 r10=fffff800b4625150
r11=0000000000000072 r12=ffff808f5eafbf00 r13=fffff8004d3ddbc0
r14=0000000000000000 r15=0000000000000fff
iopl=0         nv up ei ng nz na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00040286
pci!PciBusInterface_GetBusData+0x3ab:
fffff800`4708226b c3              ret

adolfotregosa · Wednesday at 17:57

number201724 said:
Can you passthrough? I use vfio passthrough and when I install the nvidia driver (windows 11) it gives me a black screen.I'm using RTX 5060. I installed the firmware update to no avail.

Yes I can. I don't use windows though only linux, sorry I can't help with windows.

adolfotregosa · Wednesday at 18:00

number201724 said:

Can you passthrough? I use vfio passthrough and when I install the nvidia driver (windows 11) it gives me a black screen.I'm using RTX 5060. I installed the firmware update to no avail.

I don't know if it is a graphics card design or firmware issue of the 50 series.
When I switched back to a 40 series graphics card all the problems disappeared.
It will get stuck in the Windows BOOT stage, and nvlddmkm will always stop at GetBusData, an infinite loop.

my linux kernel is one I modified to support aperf/mperf and arch-lbr, pmu, etc. I use qemu directly

Code:

PROCESS ffffcb0fa46fe140
    SessionId: none  Cid: 03c0    Peb: 2cd97ab000  ParentCid: 0348
    DirBase: 75fd05000  ObjectTable: ffff9d0f87568c40  HandleCount:  73.
    Image: csrss.exe
    VadRoot ffffcb0fa4766560 Vads 43 Clone 0 Private 220. Modified 4. Locked 0.
    DeviceMap ffff9d0f8726cc40
    Token                             ffff9d0f8767b880
    ElapsedTime                       00:01:23.400
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         43680
    QuotaPoolUsage[NonPagedPool]      6208
    Working Set Sizes (now,min,max)  (1133, 50, 345) (4532KB, 200KB, 1380KB)
    PeakWorkingSetSize                1084
    VirtualSize                       2101316 Mb
    PeakVirtualSize                   2101316 Mb
    PageFaultCount                    1166
    MemoryPriority                    BACKGROUND
    BasePriority                      13
    CommitCharge                      298

        THREAD ffffcb0fa47080c0  Cid 03c0.03c4  Teb: 0000002cd97ac000 Win32Thread: ffffcb0fa4760060 RUNNING on processor 2
        Not impersonating
        DeviceMap                 ffff9d0f8726cc40
        Owning Process            ffffcb0fa46fe140       Image:         csrss.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      607            Ticks: 3624 (0:00:00:56.625)
        Context Switch Count      257            IdealProcessor: 2           
        UserTime                  00:00:00.000
        KernelTime                00:00:57.062
        Win32 Start Address 0x00007ff677ed1010
        Stack Init ffffdd82c6445c70 Current ffffdd82c64441a0
        Base ffffdd82c6446000 Limit ffffdd82c6440000 Call 0000000000000000
        Priority 16  BasePriority 13  Priority Floor 16  IoPriority 2  PagePriority 5
        Child-SP          RetAddr               : Args to Child                                                           : Call Site
        ffffdd82`c64444e0 fffff803`387d224f     : 90ebea4d`74e91674 823be73c`d2a52225 00730073`0061006c 00000000`00000000 : nt!KzLowerIrql+0x22
        ffffdd82`c6444510 fffff803`3c362640     : 00000000`0000000f 00000000`00000000 ffffffff`ffffffd1 fffff803`00000002 : pci!PciBusInterface_GetBusData+0x38f
        ffffdd82`c64445e0 fffff803`3c5c79bc     : 00000000`00000000 00000000`0008a000 ffffcb0f`a4815000 00000000`00000002 : nvlddmkm+0xf2640
        ffffdd82`c6444650 fffff803`3c9c9167     : 00000000`00000fff 00000000`0008a000 00000000`00000000 00000000`00000000 : nvlddmkm+0x3579bc
        ffffdd82`c64446a0 fffff803`3c9c63f7     : fffff803`3d66dbc0 ffffcb0f`a4815000 00000000`00000000 ffffcb0f`a4815000 : nvlddmkm+0x759167
        ffffdd82`c6444710 fffff803`3c37da01     : 00000000`00000000 ffffcb0f`a4815000 ffffdd82`c64447a9 00000000`00000000 : nvlddmkm+0x7563f7
        ffffdd82`c6444740 fffff803`3c67dad4     : ffffcb0f`a4815000 ffffcb0f`a4815000 ffffcb0f`a4815000 00000000`00000001 : nvlddmkm+0x10da01
        ffffdd82`c6444810 fffff803`3c3b17d8     : ffffcb0f`a4815000 ffffcb0f`a4815000 ffffcb0f`a4815000 00000000`00000000 : nvlddmkm+0x40dad4
        ffffdd82`c6444870 fffff803`3c357f9c     : ffffcb0f`a4815000 00000000`00000000 00000000`00000000 00000000`00000000 : nvlddmkm+0x1417d8
        ffffdd82`c64448b0 fffff803`3c357bc3     : ffffdd82`c6445628 00000070`00000000 00000000`00000000 00000000`00000000 : nvlddmkm+0xe7f9c
        ffffdd82`c6444940 fffff803`3db9e890     : 01000000`00100000 ffffcb0f`a3db4000 ffffdd82`c6444b30 00000000`00000001 : nvlddmkm+0xe7bc3
        ffffdd82`c6444a30 fffff803`3db18084     : ffffcb0f`a3db4000 ffffcb0f`a3db4000 ffffcb0f`a3db4000 ffffdd82`c6444e50 : nvlddmkm!nvDumpConfig+0x84ab20
        ffffdd82`c6444d50 fffff803`3dbe651d     : 00000000`00000000 ffffdd82`c6445628 ffffcb0f`00000003 00000000`00000000 : nvlddmkm!nvDumpConfig+0x7c4314
        ffffdd82`c6445330 fffff803`3dbf66cb     : fffff803`3dbe643e ffffcb0f`a3db4000 ffffcb0f`a3b838c0 00000000`00000240 : nvlddmkm!nvDumpConfig+0x8927ad
        ffffdd82`c64453f0 fffff803`3d99459e     : 00000000`00000000 ffffcb0f`a3b838c0 ffffdd82`c64454b0 fffff803`37be2df2 : nvlddmkm!nvDumpConfig+0x8a295b
        ffffdd82`c6445430 fffff803`37dbd7dc     : ffffcb0f`a3bd2848 ffffdd82`c6445480 00000000`00000000 00000000`00000000 : nvlddmkm!nvDumpConfig+0x64082e
        ffffdd82`c6445470 fffff803`37bdaf78     : ffffcb0f`a42ac010 ffffdd82`c64455d0 ffffcb0f`a3bd1180 00000000`00000000 : dxgkrnl!DpiDxgkDdiStartDevice+0x80
        ffffdd82`c64454d0 fffff803`37bdcecd     : 00000000`00000000 00000000`00000000 ffffcb0f`a3bd1180 00000000`00000000 : dxgkrnl!DpiFdoStartAdapter+0xac0
        ffffdd82`c64456d0 fffff803`37bdcb50     : ffffcb0f`a3bd1180 ffffdd82`c6445820 00000000`00000000 ffffcb0f`a3b838c0 : dxgkrnl!DpiFdoStartNonLdaAdapter+0x85
        ffffdd82`c6445720 fffff803`37bdc473     : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000000 : dxgkrnl!DpiFdoStartAdapterThreadImpl+0x604
        ffffdd82`c64458a0 fffff803`37d83277     : fffff803`37afdc00 00000000`00000000 00000000`00000000 00000000`00000000 : dxgkrnl!DpiFdoStartAdapterThread+0x33
        ffffdd82`c64458d0 fffff803`37d82e8a     : ffffffff`fffffffd 00000000`00000000 00000000`00000000 fffff803`37988076 : dxgkrnl!DpiSessionCreateCallback+0x6b
        ffffdd82`c6445910 fffff803`3798442a     : 00000000`00000000 00000000`000000ec 00000000`00000000 00000000`000000ec : dxgkrnl!DxgkNotifySessionStateChange+0x5a
        ffffdd82`c6445960 fffff803`378b4047     : 00000000`00000000 00000000`00000000 00000000`00000000 fffff803`378b54bd : watchdog!SMgrNotifySessionChange+0x4a
        ffffdd82`c6445990 fffff803`3bd6d883     : 00000217`23106db4 00000000`000000f0 fffff803`3bfbcdb8 fffff803`3bfbcdc0 : win32k!SysEntrySMgrNotifySessionChange+0x17
        ffffdd82`c64459c0 fffff803`3bd6d815     : 00000000`00000040 00000000`00000000 00000000`00000000 fffff803`3c0093e2 : win32kbase!SysEntrySMgrNotifySessionChange+0x33
        ffffdd82`c64459f0 fffff803`3bd6da85     : 00000000`00000000 00000000`00000001 ffffbf54`232ca04c 00000000`00000000 : win32kbase!DrvNotifySessionStateChange+0x45
        ffffdd82`c6445a20 fffff803`3be9661c     : 00000000`00000001 00000000`00000001 00000000`00000000 00000000`00000000 : win32kbase!InitializeGreCSRSS+0xd
        ffffdd82`c6445a50 fffff803`378b4ce4     : 00000000`000000f0 ffffdd82`c6445b60 00000000`001f0003 00000000`000000ec : win32kbase!Win32kBaseUserInitialize+0x21c
        ffffdd82`c6445ab0 fffff803`a60b8655     : ffffcb0f`a47080c0 00000000`00000030 ffffcb0f`a47080c0 00000000`001f0003 : win32k!NtUserInitialize+0x34
        ffffdd82`c6445ae0 00007ffd`6610a5b4     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffffdd82`c6445ae0)
        0000002c`d983f438 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffd`6610a5b4

GetBusData can never get any data, it always stops here, and no output is displayed.

Code:

10: kd> g
Breakpoint 0 hit
pci!PciBusInterface_GetBusData:
fffff800`47081ec0 48895c2408      mov     qword ptr [rsp+8],rbx
3: kd> r
rax=fffff80047081ec0 rbx=000000000000000f rcx=ffff808f5b5ea4b0
rdx=0000000000000000 rsi=0000000000ff0002 rdi=ffff808f5ebf3dd0
rip=fffff80047081ec0 rsp=ffffa4058c0ec5d8 rbp=000000000000ff00
 r8=ffffa4058c0ec610  r9=0000000000000fff r10=0000000000000000
r11=ffffa4058c0ec620 r12=ffff808f5eafbf00 r13=fffff8004d3ddbc0
r14=0000000000000000 r15=0000000000000fff
iopl=0         nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00040246
pci!PciBusInterface_GetBusData:
fffff800`47081ec0 48895c2408      mov     qword ptr [rsp+8],rbx ss:0018:ffffa405`8c0ec5e0=000000000000000f
3: kd> dq rsp+20h
ffffa405`8c0ec5f8  fffff800`00000003 00000000`00000004
ffffa405`8c0ec608  00000000`00040246 ffffa405`8c0ec620
ffffa405`8c0ec618  ffff4a4e`afdf0780 00000000`00000000
ffffa405`8c0ec628  00000000`00000000 00000000`00000000
ffffa405`8c0ec638  ffffa405`8c0ec6dc 00000000`00000fff
ffffa405`8c0ec648  fffff800`4c3379bc 00000000`00000000
ffffa405`8c0ec658  00000000`0008a000 ffff808f`5e12e000
ffffa405`8c0ec668  00000000`00000002 ffffffff`ffffffd1
3: kd> k
 # Child-SP          RetAddr               Call Site
00 ffffa405`8c0ec5d8 fffff800`4c0d2640     pci!PciBusInterface_GetBusData
01 ffffa405`8c0ec5e0 fffff800`4c3379bc     nvlddmkm+0xf2640
02 ffffa405`8c0ec650 fffff800`4c739167     nvlddmkm+0x3579bc
03 ffffa405`8c0ec6a0 fffff800`4c7363f7     nvlddmkm+0x759167
04 ffffa405`8c0ec710 fffff800`4c0eda01     nvlddmkm+0x7563f7
05 ffffa405`8c0ec740 fffff800`4c3edad4     nvlddmkm+0x10da01
06 ffffa405`8c0ec810 fffff800`4c1217d8     nvlddmkm+0x40dad4
07 ffffa405`8c0ec870 fffff800`4c0c7f9c     nvlddmkm+0x1417d8
08 ffffa405`8c0ec8b0 fffff800`4c0c7bc3     nvlddmkm+0xe7f9c
09 ffffa405`8c0ec940 fffff800`4d90e890     nvlddmkm+0xe7bc3
0a ffffa405`8c0eca30 fffff800`4d888084     nvlddmkm!nvDumpConfig+0x84ab20
0b ffffa405`8c0ecd50 fffff800`4d95651d     nvlddmkm!nvDumpConfig+0x7c4314
0c ffffa405`8c0ed330 fffff800`4d9666cb     nvlddmkm!nvDumpConfig+0x8927ad
0d ffffa405`8c0ed3f0 fffff800`4d70459e     nvlddmkm!nvDumpConfig+0x8a295b
0e ffffa405`8c0ed430 fffff800`4665d7dc     nvlddmkm!nvDumpConfig+0x64082e
0f ffffa405`8c0ed470 fffff800`4647af78     dxgkrnl!DpiDxgkDdiStartDevice+0x80
10 ffffa405`8c0ed4d0 fffff800`4647cecd     dxgkrnl!DpiFdoStartAdapter+0xac0
11 ffffa405`8c0ed6d0 fffff800`4647cb50     dxgkrnl!DpiFdoStartNonLdaAdapter+0x85
12 ffffa405`8c0ed720 fffff800`4647c473     dxgkrnl!DpiFdoStartAdapterThreadImpl+0x604
13 ffffa405`8c0ed8a0 fffff800`46623277     dxgkrnl!DpiFdoStartAdapterThread+0x33
14 ffffa405`8c0ed8d0 fffff800`46622e8a     dxgkrnl!DpiSessionCreateCallback+0x6b
15 ffffa405`8c0ed910 fffff800`4622442a     dxgkrnl!DxgkNotifySessionStateChange+0x5a
16 ffffa405`8c0ed960 fffff800`46154047     watchdog!SMgrNotifySessionChange+0x4a
17 ffffa405`8c0ed990 fffff800`4b15d883     win32k!SysEntrySMgrNotifySessionChange+0x17
18 ffffa405`8c0ed9c0 fffff800`4b15d815     win32kbase!SysEntrySMgrNotifySessionChange+0x33
19 ffffa405`8c0ed9f0 fffff800`4b15da85     win32kbase!DrvNotifySessionStateChange+0x45
1a ffffa405`8c0eda20 fffff800`4b28661c     win32kbase!InitializeGreCSRSS+0xd
1b ffffa405`8c0eda50 fffff800`46154ce4     win32kbase!Win32kBaseUserInitialize+0x21c
1c ffffa405`8c0edab0 fffff800`b48b8655     win32k!NtUserInitialize+0x34
1d ffffa405`8c0edae0 00007ffb`613ca5b4     nt!KiSystemServiceCopyEnd+0x25
1e 0000003b`c65bf808 00007ffb`61284a6a     0x00007ffb`613ca5b4
1f 0000003b`c65bf810 000001dd`cfaa08cc     0x00007ffb`61284a6a
20 0000003b`c65bf818 00000000`00000030     0x000001dd`cfaa08cc
21 0000003b`c65bf820 0000003b`c65bf879     0x30
22 0000003b`c65bf828 00000000`001f0003     0x0000003b`c65bf879
23 0000003b`c65bf830 00000000`00000000     0x1f0003
3: kd> r
rax=0000000000000000 rbx=000000000000000f rcx=0000000000000000
rdx=ffff808f590f0960 rsi=0000000000ff0002 rdi=ffff808f5ebf3dd0
rip=fffff8004708226b rsp=ffffa4058c0ec5d8 rbp=000000000000ff00
 r8=fffff800b460b150  r9=0000000000000000 r10=fffff800b4625150
r11=0000000000000072 r12=ffff808f5eafbf00 r13=fffff8004d3ddbc0
r14=0000000000000000 r15=0000000000000fff
iopl=0         nv up ei ng nz na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00040286
pci!PciBusInterface_GetBusData+0x3ab:
fffff800`4708226b c3              ret

Which kernel are you using on proxmox ?

number201724 · Wednesday at 19:13

adolfotregosa said:
Which kernel are you using on proxmox ?

I have tried it from 6.13, 6.14, 6.15.

adolfotregosa · Wednesday at 19:27

6.14.6 is the minimum required to have the vfio patch incorporated. You see the tianocore bios logo after running the nvidia utility, right?

number201724 · Wednesday at 20:06

adolfotregosa said:
6.14.6 is the minimum required to have the vfio patch incorporated. You see the tianocore bios logo after running the nvidia utility, right?

As long as I don't install the nvidia driver, everything works fine, I can enter the operating system and get to the desktop, once I install the nvidia driver, then my guest will not boot, even if I install the nvidia driver in the system, my mstsc works, but the system process has a thread at 100% cpu usage, but once the driver is installed and rebooted, the guest will not boot.

adolfotregosa · Wednesday at 20:13

number201724 said:
As long as I don't install the nvidia driver, everything works fine, I can enter the operating system and get to the desktop, once I install the nvidia driver, then my guest will not boot, even if I install the nvidia driver in the system, my mstsc works, but the system process has a thread at 100% cpu usage, but once the driver is installed and rebooted, the guest will not boot.

Sorry I can't help more. I don't use windows, I'm sorry. At least give a linux distro a try like cachyos just to have a reference. All I can say is that I do not bind vfio at all on boot, I have nvidia driver installed on proxmox and loaded with modeset=1. When I start my guest I unload the nvidia driver and then vfio takes over automatically and everything is fine. On guest shutdown the nvidia modules are reloaded to get proxmox cli back and also for the gpu to be able to go into P8 state and save power.

kenjiro310 · Thursday at 03:42

Hey I am having the similar problems. I just purchased a 5060ti a few days ago with the same intention and my 1st problem was getting it to load stuck at loading ram disk screen. I pre set my proxmox server up a few months ago with a quadro p620 no problems.

number201724 · Thursday at 10:29

I think this is a firmware bug/limitation, I tried spoofing the vendor-id, device-id to 4060ti, it still can't boot.

adolfotregosa · Thursday at 10:35

number201724 said:
I think this is a firmware bug/limitation, I tried spoofing the vendor-id, device-id to 4060ti, it still can't boot.

Did you test with a linux guest ?? You are gonna me try a windows guest lol .

adolfotregosa · Thursday at 10:48

kenjiro310 said:
Hey I am having the similar problems. I just purchased a 5060ti a few days ago with the same intention and my 1st problem was getting it to load stuck at loading ram disk screen. I pre set my proxmox server up a few months ago with a quadro p620 no problems.

Did you blacklist nouveau module ?
GRUB_CMDLINE_LINUX_DEFAULT="module_blacklist=nouveau"

number201724 · Thursday at 11:14

adolfotregosa said:
Did you blacklist nouveau module ?
GRUB_CMDLINE_LINUX_DEFAULT="module_blacklist=nouveau"

No, I recompiled the kernel and removed the graphics drivers directly from the kernel, so it has no nv related drivers.

adolfotregosa · Thursday at 12:50

No issue over here !! nvidia driver is installed in proxmox, working, I then unload the driver and start the vm.

Options I use for the proxmox nvidia driver in /etc/modprobe.d/nvidia-options.conf

options nvidia_modeset hdmi_deepcolor=0 conceal_vrr_caps=1
options nvidia_drm modeset=1

I leave attached my simple hookscript.

number201724 · Thursday at 13:17

adolfotregosa said:
No issue over here !! nvidia driver is installed in proxmox, working, I then unload the driver and start the vm.

Options I use for the proxmox nvidia driver in /etc/modprobe.d/nvidia-options.conf

options nvidia_modeset hdmi_deepcolor=0 conceal_vrr_caps=1
options nvidia_drm modeset=1

I leave attached my simple hookscript.

View attachment 86549
View attachment 86548

WTF????? What brand is your graphics card? ASUS? GIGABYTE?
What is the version of PVE? What is the version of the Linux kernel? What is the version of qemu? I want to try it. Maybe it is a bug in qemu 9.0?

adolfotregosa · Thursday at 14:22

number201724 said:
WTF????? What brand is your graphics card? ASUS? GIGABYTE?
What is the version of PVE? What is the version of the Linux kernel? What is the version of qemu? I want to try it. Maybe it is a bug in qemu 9.0?

I have experience with both MSI and Gigabyte but honestly it will not matter.
PVE latest, absolutely self compiled stock kernel 6.14.8, qemu latest. There's no bug as far I can tell. You simply have to find the combination that works. In theory my config should work for you. Rebar is enabled and UEFI boot is a must in both the host and VM. Also guest cpu has to be set to host ! Copy my print settings.

Are you sure the nvidia firmware tool updated your card successfully? I'm assuming your goal is to have direct image from the gpu like me.

number201724 · Thursday at 15:19

adolfotregosa said:
I have experience with both MSI and Gigabyte but honestly it will not matter.
PVE latest, absolutely self compiled stock kernel 6.14.8, qemu latest. There's no bug as far I can tell. You simply have to find the combination that works. In theory my config should work for you. Rebar is enabled and UEFI boot is a must in both the host and VM. Also guest cpu has to be set to host ! Copy my print settings.

Are you sure the nvidia firmware tool updated your card successfully? I'm assuming your goal is to have direct image from the gpu like me.

I solved it. It seems that the 5000 series needs

Code:

[device "ich9-pcie-port-1"]
  driver = "pcie-root-port"
  x-speed = "16"
  x-width = "32"
  multifunction = "on"
  bus = "pcie.0"
  addr = "1c.0"
  port = "1"
  chassis = "1"

adolfotregosa · Thursday at 15:26

number201724 said:
I solved it. It seems that the 5000 series needs

Code:

[device "ich9-pcie-port-1"] driver = "pcie-root-port" x-speed = "16" x-width = "32" multifunction = "on" bus = "pcie.0" addr = "1c.0" port = "1" chassis = "1"

Good. I have no idea what that means but good for you

EDIT: I remember that the following is also required. Without "PCI-Express" it did not work for me. (The "primary gpu" maybe is not required but it works so I left it alone)

Nvidia 5000 series passthrough , failing miserably.

Active Member

Active Member

Active Member

Member

Active Member

Active Member

Member

Active Member

Member

Active Member

New Member

Member

Active Member

Active Member

Member

Active Member

Attachments

Member

Active Member

Member

Active Member

We value your privacy