Gpu passthrough issue

D

devian34

New Member
Apr 15, 2024
9
0
1
i have issue with single-gpu system gpu passthrough
in monitor i see only black screen
my pc
Intel(R) Xeon(R) CPU E3-1230 V2 (ivy bridge)
motherboard: B75MA-E33 (with enabled vt-d)
1060 6gb (GV-N1060WF2OC-6GD)

Linux pve 6.5.11-8-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-8 (2024-01-30T12:27Z) x86_64 GNU/Linux

8.1.2 prox mox version

GRUB loader

BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)
IOMMU group 10 00:1f.0 ISA bridge [0601]: Intel Corporation B75 Express Chipset LPC Controller [8086:1e49] (rev 04)
IOMMU group 10 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04)
IOMMU group 10 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller [8086:1e22] (rev 04)
IOMMU group 11 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
IOMMU group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU group 2 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)
IOMMU group 3 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 [8086:1e3a] (rev 04)
IOMMU group 4 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)
IOMMU group 5 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)
IOMMU group 6 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4)
IOMMU group 7 00:1c.1 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 [8086:1e12] (rev c4)
IOMMU group 8 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)
IOMMU group 9 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a4)

BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,000A,000B,0002,0006,0009,0001,0003,0004
Boot0000* proxmox HD(2,GPT,9fc39af3-35aa-4ca8-b7c7-4ae3c5fbc778,0x800,0x200000)/File(\EFI\proxmox\shimx64.efi)
Boot0001* UEFI:CD/DVD Drive BBS(129,,0x0)
Boot0002 UEFI: Встроенный EFI Shell VenMedia(5023b95c-db26-429b-a648-bd47664c8012)AMBO
Boot0003* UEFI:Removable Device BBS(130,,0x0)
Boot0004* UEFI:Network Device BBS(131,,0x0)
Boot0006 Жесткий диск BBS(HD,,0x0)AMGOAMNO........o.H.G.S.T. .H.T.S.5.4.5.0.5.0.A.7.E.6.8.0....................A...........................>..Gd-.;.A..MQ..L. . . . . . .B.R.5.2.A.0.C.1.J.0.2.4.J.N......AMBO
Boot0009 Неизвестное устройство BBS(11,,0x0)AMGOAMNO........m. .U.S.B. .D.I.S.K. .2...0. .P.M.A.P....................A.............................:..Gd-.;.A..MQ..L. .U.S.B. .D.I.S.K. .2...0. .P.M.A.P......AMBO
Boot000A* UEFI OS HD(2,GPT,9fc39af3-35aa-4ca8-b7c7-4ae3c5fbc778,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)
Boot000B* UEFI: USB DISK 2.0 PMAP PciRoot(0x0)/Pci(0x1d,0x0)/USB(1,0)/USB(4,0)/HD(1,MBR,0xe0c6de61,0x800,0xe6d800)AMBO
 
I would suggest searching for vgpu_unlock and seeing if you cant just get vgpu profiles working, it will make it it so you dont have to pass the entire gpu to a vm and instead use profiles assigned to vms which also makes for being able to use your gpu in multiple vms.

But for single gpu set the display setting on the vm to none, you can read more about this solution and others related to passthrough here

https://www.linuxconsultant.org/imp...cie-passthrough-tweaks-fixes-and-workarounds/
 
zenowl77 said:
I would suggest searching for vgpu_unlock and seeing if you cant just get vgpu profiles working, it will make it it so you dont have to pass the entire gpu to a vm and instead use profiles assigned to vms which also makes for being able to use your gpu in multiple vms.

But for single gpu set the display setting on the vm to none, you can read more about this solution and others related to passthrough here

https://www.linuxconsultant.org/imp...cie-passthrough-tweaks-fixes-and-workarounds/
Click to expand...
on windows oc i always get code 43 and always blackscreen
i have set the display setting but it not help me
and i need working gpu at least 1 vm :)
 
Last edited:
Im not sure what else to do there if you need full passthrough for a monitor on the gpu for a desktop system, the display option set to none is basically all ive got, maybe also set gpu to primary? Not sure really beyond that as i always use vgpu profiles and a headless setup accessed from my laptop.

Just wanted to try to give you something to try that could work since i notice a lot of gpu passthrough posts never get any responses
 
zenowl77 said:
Im not sure what else to do there if you need full passthrough for a monitor on the gpu for a desktop system, the display option set to none is basically all ive got, maybe also set gpu to primary? Not sure really beyond that as i always use vgpu profiles and a headless setup accessed from my laptop.

Just wanted to try to give you something to try that could work since i notice a lot of gpu passthrough posts never get any responses
Click to expand...
i will test with unlocking vGPU maybe it will works for me
 
zenowl77 said:
Im not sure what else to do there if you need full passthrough for a monitor on the gpu for a desktop system, the display option set to none is basically all ive got, maybe also set gpu to primary? Not sure really beyond that as i always use vgpu profiles and a headless setup accessed from my laptop.

Just wanted to try to give you something to try that could work since i notice a lot of gpu passthrough posts never get any responses
Click to expand...
now i get stuck with mdevctl types doesnt return anything
and
nvidia-smi vgpu
No supported devices in vGPU mode
 
devian34 said:
now i get stuck with mdevctl types doesnt return anything
and
nvidia-smi vgpu
No supported devices in vGPU mode
Click to expand...
https://gitlab.com/polloloco/vgpu-proxmox

Try this guide specifically, its the best one ive found, it sounds like the patch wasn’t applied properly or something if unlock isn’t working and no vgpu mode devices are found. Youll also need specific drivers youll have to hunt down online somewhere, ive found them on archive.org and other places before.
 
  • Like
Reactions: devian34
zenowl77 said:
https://gitlab.com/polloloco/vgpu-proxmox

Try this guide specifically, its the best one ive found, it sounds like the patch wasn’t applied properly or something if unlock isn’t working and no vgpu mode devices are found. Youll also need specific drivers youll have to hunt down online somewhere, ive found them on archive.org and other places before.
Click to expand...
bro i love u IT WORKS spend 3 days to resolve it modify configs and only u resolve this ! i find drivers on this forum on threads
 
zenowl77 said:
https://gitlab.com/polloloco/vgpu-proxmox

Try this guide specifically, its the best one ive found, it sounds like the patch wasn’t applied properly or something if unlock isn’t working and no vgpu mode devices are found. Youll also need specific drivers youll have to hunt down online somewhere, ive found them on archive.org and other places before.
Click to expand...
but mdevctl types still doesnt return anything but nvidia-smi vgpu text my gpu
 
make sure you follow all the steps in the guide exactly if you need to just start at the top and go over the whole guide following it step by step (except the parts that do not apply to your gpu of course) and make sure you have installed the kvm driver on proxmox itself, it should be around 70mb in size unlike the other drivers.

also once you setup everything the unlock might need a reboot to apply, ive noticed this one so make sure once you're all done you reboot

then you vgpu should return the proper output (hoprfully anyways) here is mine for example with 2 vgpu profiles assigned to VMs

Code: 
root@prox:~# nvidia-smi vgpu
Tue Apr 16 23:01:34 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03                |
|---------------------------------+------------------------------+------------+
| GPU  Name                       | Bus-Id                       | GPU-Util   |
|      vGPU ID     Name           | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  Tesla P4                   | 00000000:04:00.0             |  26%       |
|      XXXXXXXXXX  GRID P4-2Q     | XXXX...  win-10-1,debug-t... |     25%    |
|      XXXXXXXXXX  GRID P4-2Q     | XXXX...  Win10-2,debug-th... |      0%    |
+---------------------------------+------------------------------+------------+

it is hard to say what exactly is going wrong in the process though as vgpu is extremely problematic and many things could be wrong but with enough trial and error you can get it working
 
zenowl77 said:
make sure you follow all the steps in the guide exactly if you need to just start at the top and go over the whole guide following it step by step (except the parts that do not apply to your gpu of course) and make sure you have installed the kvm driver on proxmox itself, it should be around 70mb in size unlike the other drivers.

also once you setup everything the unlock might need a reboot to apply, ive noticed this one so make sure once you're all done you reboot

then you vgpu should return the proper output (hoprfully anyways) here is mine for example with 2 vgpu profiles assigned to VMs

Code: 
root@prox:~# nvidia-smi vgpu
Tue Apr 16 23:01:34 2024     
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03                |
|---------------------------------+------------------------------+------------+
| GPU  Name                       | Bus-Id                       | GPU-Util   |
|      vGPU ID     Name           | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  Tesla P4                   | 00000000:04:00.0             |  26%       |
|      XXXXXXXXXX  GRID P4-2Q     | XXXX...  win-10-1,debug-t... |     25%    |
|      XXXXXXXXXX  GRID P4-2Q     | XXXX...  Win10-2,debug-th... |      0%    |
+---------------------------------+------------------------------+------------+

it is hard to say what exactly is going wrong in the process though as vgpu is extremely problematic and many things could be wrong but with enough trial and error you can get it working
Click to expand...
i recive exacly what u sent in nvidia-smi vgpu but in mdevctl types i didnt recive anything
can u sent me what u recive when text this command?
UPD
right now i recive Failed to initialize NVML: Unknown Error
but gpu on vm working
 
Last edited:
sure, here is what i receive from mdevctl types

first two are my intel iGPU then nvidia gpu is the rest, if this command returns nothing vgpu unlock likely is not working.
Code: 
root@prox:~# mdevctl types
0000:00:02.0
  i915-GVTg_V5_4
    Available instances: 0
    Device API: vfio-pci
    Name: GVTg_V5_4
    Description: low_gm_size: 128MB, high_gm_size: 512MB, fence: 4, resolution: 1920x1200, weight: 4
  i915-GVTg_V5_8
    Available instances: 0
    Device API: vfio-pci
    Name: GVTg_V5_8
    Description: low_gm_size: 64MB, high_gm_size: 384MB, fence: 4, resolution: 1024x768, weight: 2
0000:04:00.0
  nvidia-157
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=4
  nvidia-214
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2B4
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=4
  nvidia-243
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1B4
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=8
  nvidia-63
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=8
  nvidia-64
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=4
  nvidia-65
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=2
  nvidia-66
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=1
  nvidia-67
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=8
  nvidia-68
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=4
  nvidia-69
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=2
  nvidia-70
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=1
  nvidia-71
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=8

check your /etc/vgpu_unlock/config.toml make sure it says unlock = true and then check the psa at the bottom of the guide for pascal and older since you're using the 1060 (if you're not using an older driver)
 
zenowl77 said:
sure, here is what i receive from mdevctl types

first two are my intel iGPU then nvidia gpu is the rest, if this command returns nothing vgpu unlock likely is not working.
Code: 
root@prox:~# mdevctl types
0000:00:02.0
  i915-GVTg_V5_4
    Available instances: 0
    Device API: vfio-pci
    Name: GVTg_V5_4
    Description: low_gm_size: 128MB, high_gm_size: 512MB, fence: 4, resolution: 1920x1200, weight: 4
  i915-GVTg_V5_8
    Available instances: 0
    Device API: vfio-pci
    Name: GVTg_V5_8
    Description: low_gm_size: 64MB, high_gm_size: 384MB, fence: 4, resolution: 1024x768, weight: 2
0000:04:00.0
  nvidia-157
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=4
  nvidia-214
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2B4
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=4
  nvidia-243
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1B4
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=8
  nvidia-63
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=8
  nvidia-64
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=4
  nvidia-65
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=2
  nvidia-66
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=1
  nvidia-67
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=8
  nvidia-68
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P4-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=4
  nvidia-69
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=2
  nvidia-70
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=1
  nvidia-71
    Available instances: 0
    Device API: vfio-pci
    Name: GRID P4-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=8

check your /etc/vgpu_unlock/config.toml make sure it says unlock = true and then check the psa at the bottom of the guide for pascal and older since you're using the 1060 (if you're not using an older driver)
Click to expand...
oh i see in config unlock false
 
devian34 said:
oh i see in config unlock false
Click to expand...
change that to true and reboot, should be good to go hopefully

p.s. once its up go to datacenter > resource mappings > click add, check "use with mediated devices", check your gpu and add a name to it, this will help you assign profiles to vms easier
resource mappings.PNG
and don't forget to use the profile_override.toml in /etc/vgpu_unlock to assign different memory amounts to different vms otherwise you can only use profiles with the same amount so for example if you use 2gb all other vms must be set to 2gb profiles and then use the over ride to change that

here is one of mine from the file for example

Code: 
[vm.126]
display_width = 3840
display_height = 2160
max_pixels = 8294400
cuda_enabled = 1
framebuffer = 0x38000000
framebuffer_reservation = 0x8000000 # 1GB
 
Last edited:
zenowl77 said:
change that to true and reboot, should be good to go hopefully

p.s. once its up go to datacenter > resource mappings > click add, check "use with mediated devices", check your gpu and add a name to it, this will help you assign profiles to vms easier
View attachment 66469
and don't forget to use the profile_override.toml in /etc/vgpu_unlock to assign different memory amounts to different vms otherwise you can only use profiles with the same amount so for example if you use 2gb all other vms must be set to 2gb profiles and then use the over ride to change that

here is one of mine from the file for example

Code: 
[vm.126]
display_width = 3840
display_height = 2160
max_pixels = 8294400
cuda_enabled = 1
framebuffer = 0x38000000
framebuffer_reservation = 0x8000000 # 1GB
Click to expand...
oh yea now i recive all outputs what need thanks!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back