Tesla P4 massive performance problems

edit: do you have licensing setup? run & nvidia-smi | Select-String "License" and see the output if unsure. if "unlicensed" you're limited heavily and performance will suffer, you only get 15FPS, etc and you need to set up this server and set licenses from it for your p4. (not advisable to ever use that in corp settings of any kind, just home / personal use )

It's funny that you posted this because **right after** I submitted my post I was looking around and found the 15fps licensing issue. I noticed that on my Windows VM it was only hitting 15 on the passmark tests. Trying out my shiny new licenses now.
 
It's funny that you posted this because **right after** I submitted my post I was looking around and found the 15fps licensing issue. I noticed that on my Windows VM it was only hitting 15 on the passmark tests. Trying out my shiny new licenses now.
hope it helps
 
sadly it would be a lot of work to do that so its not exactly an option here, i can with vgpu and the fixes i have thanks to great devs keeping it supported use the new 550 drivers that are months old but if i go bare metal the support ended with 535 drivers and i think data center drivers are much older, plus i would have to setup a whole new vm or totally reinstall drivers, etc since it is a completely different driver to use it directly with passthrough and that also requires a reboot to get it back on proxmox for vgpu once used in that way. i might try a whole new vm if i get desperate for a fix but not sure i want it back enough to not use vgpu at all.


this is what i was thinking, i know i have some performance loss because of it but its not 90% lol, the card is still running on 8x, id be seeing a lot of impact on 3D and everything else if it was that bad, its definitely there but mostly only shows itself when everything in the system is stressed at once which is rare.


it has 8 chips, they are 16 and 8 gb chips i forget lay out i think its 16 8 16 8 16 8 16 8 i put them in slots relative to how the cpu handles them and i when through a lot of tweaking / benchmarking and they hit 55GB/s with good performance results.

yes it is a higher end board, i linked to it above its a gigabyte x299 ud4 board that claims Server-Class Digital Power Design and other features and the cpu is a i7-7820X skylake-x cpu.

Looking at the X299 product page I notice the last BIOS is from 2021, I'd not look much further.
One thing you could do is boot the machine with a live system or similar and benchmark hashcat that way on the GPU.

I'd not be surprised if the performance you get from this is as high as it is gonna be.

It may be worth poking at Gigabyte for a final BIOS update mentioning hardware vulnerabilities :-) while casually mentioning the GPU performance
 
Looking at the X299 product page I notice the last BIOS is from 2021, I'd not look much further.
One thing you could do is boot the machine with a live system or similar and benchmark hashcat that way on the GPU.

I'd not be surprised if the performance you get from this is as high as it is gonna be.

It may be worth poking at Gigabyte for a final BIOS update mentioning hardware vulnerabilities :-) while casually mentioning the GPU performance
I may have to try something like that just to test it

Also how exactly do you think the 2021 bios could be an issue? It is much newer than the tesla p4 so has full support for more than what is supported by the p4, it maxes out the newer intel arc card too along with even using resizable bar and passing it to the virtual machines just fine, it seems like it should handle anything the older p4 should need of it?
 
There are CPU mitigations in all modern OS that severely impact the performance of older systems, especially cheap desktop systems that nobody is testing against anymore. For more modern systems both UEFI updates and CPU microcode updates will mitigate that performance impact. The vendor no longer supports that device, so you won’t see any fixes.

As said, not having a valid license will also cap the performance for vGPU, but not sure if it’s worth it to you spending $10/y/VM for a vGPU license over just passing the entire card through (which should give you the near-native performance)
 
I'm not sure any of the x299 boards were cheap, this one i am pretty sure was in the $400-500 range on release and is still going for 250 today. still though it shouldn't impact GPU performance by 90% only in one specific area, that also kind of has nothing to do with the CPU? plus other devices would all be effected? and cpu performances would definitely be effected if it was CPU related. but i have also disabled all mitigations.

i have a license server setup, it is all working fine there and i would prefer not to pass the card because within vGPU with the fixes applied i can use the newer 550 driver without it the newest data center driver for the p4 is 472.50 from 2021 and that would also of course break vGPU functionality requiring a restart every time i release it from a vm and need it as a vgpu card again. i might have to setup a VM just for the data center driver purely as a test run but i don't think i would want to actually stick with that, kinda defeats the entire point of having a professional card with vGPU to not use any of it.
 
There is no more license server for NVIDIA vGPU, that ended 2 years ago, it’s all been converted to cloud subscription.

You’re also not using the regular 550 driver for a vGPU, it comes with a custom host and client driver and they have to match within 2 minor versions. The current one for your Pascal era hardware would be v16.10 (535) - 550 equivalent would be v17.2, there is no P4 support in the 550 drivers. I just checked on the NVIDIA Enterprise documentation, which is available with your subscription.

So if you have installed a 550 driver with a P4, you have the wrong drivers, my assumption is you used a generic public driver, which does not have vGPU support. And if you believe someone can hack them or provide you cracks, license servers or such things, my suggestion would be to scan for malware.

If you have the licenses, you can just login to NVIDIA’s website and download the proper drivers, they are not available publicly. Even in passthrough, the last driver series will be 535, ending support in 2026, not 472.
 
Last edited:
i know NVIDIA officially is now cloud, but i am still using a local server and i am using 17.5/550.144 with GPU unlock plus modifications to use it as a A5500 instead of a p4 to get past the limitation for drivers.

i do not have licenses through NVIDIA so i do not have access to all of that, i just have all the drivers backed up locally.

might look up to see how 17.6 - 18.0-2 do with kernel 6.14 + gpu unlock and see if theres any differences on those newer versions, i just don't want to not use vGPU
 
The 550 drivers do not support the P4. Without a subscription, you cannot unlock the license, although it will work, but the performance will be low.

The local server option died a while ago, the 550 drivers do not support it. Yes, the config feature is still there, but the 535 and 550 driver will ignore it, 525 was the last local server option I believe.

Per NVIDIA: vGPUs that require licensing run at a reduced capability until a license is acquired.
20 minutes
  • Frame rate is capped at 15 frames per second.
  • The performance of applications and processes that use CUDA is degraded.
24 hours
  • Frame rate is capped at 3 frames per second.
  • CUDA stops working and CUDA API function calls fail.
  • GPU resource allocations for a vGPU are limited, which will prevent some applications from rendering or running correctly, or will cause them to report errors when started.
 
screenshot of windows VM license + version with SSH into proxmox host showing driver version:
proxmox-plus-vm-license-plus-driver-versions.PNG
i do have it setup and working with the 550 driver, it works just fine with the modifications. my issue definitely isn't related to licensing
 
That is vApps, not vWS, different beast, so you may be passing the wrong card type (or, again, this driver is being weird because it is unsupported). Try nvidia-smi -q (for either platform). Also, those are not the correct drivers, those are the consumer card drivers, the Enterprise drivers aren’t at that version yet.
 
Last edited:
That is vApps, not vWS, different beast, so you may be passing the wrong card type (or, again, this driver is being weird because it is unsupported). Try nvidia-smi -q (for either platform). Also, those are not the correct drivers, those are the consumer card drivers, the Enterprise drivers aren’t at that version yet.
vAPPs claims it doesn't have cuda/opencl support or support for 4k and ive been using allof those things, so pretty sure that isn't a problem here, i am also passing the most performant profile type so i think the license is just listed wrong also because i am using a hacked together setup and not using the official licensing server i think all licenses passed to clients have maximum support for features.

these are definitely the grid drivers, i might try v18-18.2 though to see if those are any better, those are up to 570 now.

going to try some other profiles and whatever else i can figure out to tweak/change and see if anything helps.
 
Last edited:
The drivers have different minor version numbers, the output from nvidia-smi does not match up with my setup. They rarely line up with the online drivers. Whatever your hacked up version provides is just that, a hack, it is impossible to support by us.

You should open a ticket with whomever is selling you the hack. Or spend $10 on the official licensed version and open a ticket with NVIDIA.