Everything seems to be working, regardless of these warnings. I just don't remember seeing it before I started changing the kernels and updating the drivers.
Is there any real benefit to using the 17.x drivers and mocking an A5500? To apply the A5500 patch, I guess I need to reverse and reinstall the vgpu_unlock-rs but using GreenDamTan repo and instructions?
Follow up: this warning:
`WARNING: CPU: 1 PID: 560 at drivers/pci/msi/msi.c:888 __pci_enable_msi_range+0x1b3/0x1d0`
was because in my VM I did not have the nouveau driver blacklisted.
I am, however, still getting this warning on the host:
`WARNING: CPU: 5 PID: 24912 at ./include/linux/rwsem.h:85 remap_pfn_range_internal+0x4af/0x5a0`
I only get that warning when I start up a VM that has the vGPU passed to it. Either that 6.14 kernel, or the 17.5 drivers introduced that (or maybe the latest qemu that was installed with 8.4?) because I didn't have that before I updated everything.
@zenowl77 For the first time I tried to start up a 2nd VM and it failed when it had a different amount of memory. I set it to 4gb (the same as the one in use) and it failed with a different error:
Code:
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000210,enable-migration=on,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 00000000-0000-0000-0000-000000000210: error getting device from group 97: Input/output error
Verify all devices in group 97 are bound to vfio-<bus> or pci-stub and not already in use
stopping swtpm instance (pid 223324) due to QEMU startup error
waited 10 seconds for mediated device driver finishing clean up
actively clean up mediated device with UUID 00000000-0000-0000-0000-000000000210
TASK ERROR: start failed: QEMU exited with code 1
My card is an 8GB card. One VM has a 4GB vCpu passed to it. Maybe it doesn't like 2 4GB vCpus? When I tried with a 2GB vCpu it failed with some message about an mdev instance wasn't available.
Edit: I guess my 8GB card doesn't like 2x4GB vgpus. I used the profile override feature of the unlocker for 1 VM to force it to 2GB and it started up.
that is interesting, i haven't had that issue with mine. i am currently using 2x 4GB profiles one for windows 11 and one for ubuntu 22.04 for running stable diffusion in one ollama in the other.
you may of course also want to add frl_enabled = 0 to remove frame rate limits, i keep forgetting to do that in my AI profiles, not sure it will help with AI though.
here is my vram reference section too for anyone who wants it, i worked out a custom 7GB one just because sometimes i want to use almost all the vram while leaving enough for a 512mb profile or two. i just keep this at the end of my file to copy/paste from to change the amounts.
Yeah, not sure. I even overrode the [profile.nvidia-765] profile and set those same values in the framebuffer for 4GB just in case, but it didn't work.
The good thing is that it reminded me how to override one for a specific VM since I only needed 2GB anyway.
I'll leave that alone and see if I can figure out this warning that pops up whenever I start a VM that has a vgpu. I think it is related to the CPU type passed to the VM.
What CPU do you have in your host and what cpu type do you select for your VM?
Yeah, not sure. I even overrode the [profile.nvidia-765] profile and set those same values in the framebuffer for 4GB just in case, but it didn't work.
The good thing is that it reminded me how to override one for a specific VM since I only needed 2GB anyway.
I'll leave that alone and see if I can figure out this warning that pops up whenever I start a VM that has a vgpu. I think it is related to the CPU type passed to the VM.
What CPU do you have in your host and what cpu type do you select for your VM?
I am working with a i7-7820X and i always use the “host” setting as others do not expose all the same features properly and always end up with lower performance while seeming to simultaneously cause more cpu usage host side in my experience at least.
Also while not always the case, ive quite a few times had a lot of luck with just restarting the service then running the VMs by using this command
systemctl restart nvidia-vgpu-mgr.service
which if that is the fix then i usually have to do it every time after reboot but it will usually work in those instances at least.
I think i did run into that same error one time, not sure what caused it but the vm didn’t start and i tried it again and it started. I think mine might have just been from too much memory being occupied at the time though.
The P4 absolutely should work with two VMs set to 4GB each. It seems to actually be one of the most optimal settings, as I’ve noticed when setting 7-8GB to one VM its always 6.5, 7.5, etc where as 4Gb is 4Gb
that's interesting, my p4 shows 8192MB it should be exactly 8GB
here is what mine shows with the 2x4GB profiles running.
Code:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.02 Driver Version: 550.144.02 CUDA Version: N/A |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P4 On | 00000000:17:00.0 Off | Off |
| N/A 59C P0 47W / 68W | 8094MiB / 8192MiB | 73% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 3516431 C+G vgpu 4030MiB |
| 0 N/A N/A 3585720 C+G vgpu 4030MiB |
+-----------------------------------------------------------------------------------------+
it seems for some reason your total vram is not what it should be?
EDIT: i suppose you could always do a 4GB+3GB setup or 4GB+2GB+1GB or something. or even 4GB+3.5GB (would have to figure out the settings for 3.5GB as i did with the 7gb profile)
I would be too, so far i have not run into the same issue myself, i found another post on a blog in Chinese that the user was also using vgpu with proxmox using a P4 and they had the exact nvidia-smi output you have, so it seems like some kind of bios setting or issue on some of the P4s, have you noticed it saying 8192mb before at any point or always 7680mb?
Which is also weird cause its exactly 512mb less than it should be which seems very specific and also points to something allocating/reserving it, is it your primary gpu? Could that be the issue?
Yeah, I noticed the 512MB and was proceeding along the same thoughts about something reserving that. It is indeed the only GPU in the system, the board is "Supermicro H11SSL-NC 2.0". (all 3 nodes are the same).
I poked and prodded the BIOS and nothing stands out and I don't really remember if it reported 8192 before. This is the first time I started had a 2nd instance of a vgpu enabled vm.
Yeah, I noticed the 512MB and was proceeding along the same thoughts about something reserving that. It is indeed the only GPU in the system, the board is "Supermicro H11SSL-NC 2.0". (all 3 nodes are the same).
I poked and prodded the BIOS and nothing stands out and I don't really remember if it reported 8192 before. This is the first time I started had a 2nd instance of a vgpu enabled vm.
does the AMD cpu have an iGPU or anything? i am guessing no with that motherboard, etc, if so though enabling it could give you a second gpu and free up the rest of the tesla but then you would also lose som ram, but then you could also use the iGPU in ways.
i would be interested to know too just for the sake of future reference as thats kind of an annoyance.
one thing that can reduce ram is ECC but it appears to be off in your nvidia-smi and that should be 12.5% so 1024mb on 8gb, which would make it 7168mb and vGPU then shouldnt work if it was enabled. so it doesnt appear to be that at least. but then again your ECC portion says "0" where as mine says "off"
Has anyone been able to disable ECC memory on a Tesla GPU being passed through to a VM? Running a P4 which may only be able to use the full 8GB of memory if ECC is disabled, and I cannot figure out how to configure Nvidia-SMI in proxmox. Thanks!
Code:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.02 Driver Version: 550.144.02 CUDA Version: N/A |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P4 On | 00000000:81:00.0 Off | Off |
| N/A 36C P8 10W / 75W | 34MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
My understanding of ECC is really more for hardware faults, a little power blip or cosmic ray flipping a bit here or there. I definitely wanted it for my system memory with ZFS but for GPU workload (transcoding and codeprojectai for person/vehicle detection with blue iris) I can live with a flipped bit
My understanding of ECC is really more for hardware faults, a little power blip or cosmic ray flipping a bit here or there. I definitely wanted it for my system memory with ZFS but for GPU workload (transcoding and codeprojectai for person/vehicle detection with blue iris) I can live with a flipped bit
yeah, i definitely see the point with memory, it does seem to make a difference overall, like noticeably so with system stability, i do not currently have ECC ram, but years ago when using a windows 2012 R2 copy as a server OS, it definitely made an impact that was noticeable.
i wonder if it could potentially have any impact on LLMs, etc, but the benefit would likely be so minimal its just not worth the 512mb loss and the performance impact would probably also be something to consider. lol
