NVIDIA Tesla M10 Support

[firestreak]

Hello!

I've got proxmox install on an HPE Proliant DL 380 G10.

I purchased a M10, which is essentially 4 GPUs in one.

My goal is to create 4 gaming VMs, each with a GPU, and to have them each have Sunshine installed for streaming games.

According to the docs, the M10 is 4 Maxwell GPUS (https://www.nvidia.com/en-us/data-center/tesla-m10/)

So in order to get a compatible driver on the host side, I set the gpus to spoof Titan X which are also Maxwell.

Vender ID: 0x10de
Device ID: 0x17c2

I installed Windows 11 on the VM.

With this, I was able to install the drivers. However, whenever I try to start sunshine, it gives me an error:

Error: Failed to create D3D11 device [0x887A0004]

Which is meaning from my understanding, it is unable to run Direct X 11.

Not sure the next step here. I have already tried a similar strategy with turning the GPU into vGPUs based on https://gitlab.com/polloloco/vgpu-proxmox and overriding the vGPU to be that same Titan X, but not any better.

If someone knows a good next step I'd be very appreciative.

Edit: FYI, had also tried with a ubuntu VM, worked instantly with no configuration. But then, very few games to choose.

 

[the1010boy]

Hello!

I've got proxmox install on an HPE Proliant DL 380 G10.

I purchased a M10, which is essentially 4 GPUs in one.

My goal is to create 4 gaming VMs, each with a GPU, and to have them each have Sunshine installed for streaming games.

According to the docs, the M10 is 4 Maxwell GPUS (https://www.nvidia.com/en-us/data-center/tesla-m10/)

So in order to get a compatible driver on the host side, I set the gpus to spoof Titan X which are also Maxwell.

Vender ID: 0x10de
Device ID: 0x17c2

I installed Windows 11 on the VM.

With this, I was able to install the drivers. However, whenever I try to start sunshine, it gives me an error:

Error: Failed to create D3D11 device [0x887A0004]

Which is meaning from my understanding, it is unable to run Direct X 11.

Not sure the next step here. I have already tried a similar strategy with turning the GPU into vGPUs based on https://gitlab.com/polloloco/vgpu-proxmox and overriding the vGPU to be that same Titan X, but not any better.

If someone knows a good next step I'd be very appreciative.

Edit: FYI, had also tried with a ubuntu VM, worked instantly with no configuration. But then, very few games to choose.

Did you eventually get this working? I have the same goal and am hoping I don't run into these same issues. If you did get past this issue please share how. THANKS!

 

[the1010boy]

@firestreak ,

Thanks to your post I was able to learn of Pollo Loco's guide...

https://gitlab.com/polloloco/vgpu-proxmox

...and using that guide, I was able to get vGPU working for Windows 10 guest VMs in Proxmox 8 with a Tesla M10.

For me, however, I took the route of NOT patching the GPU. So instead of trying to "spoof" the Tesla to appear as some other GPU, I chose instead to use the Tesla M10 as is, since the Tesla M10 is natively supported for vGPU usage.

The only difficulty I had was getting NVIDIA to give me a trial license for their enterprise drivers. However, once I had obtained a copy of the native drivers for the M10, all worked according to that guide by Pollo Loco.

Here's what I did to obtain an NVIDIA trial license...

• Just as was mentioned in the Pollo Loco guide, the automated systems at NVIDIA ignored all request for a trial license when a free email type address was used (any gmail, yahoo mail, hotmail, or like email).
• Not having any non-free email available to me at the time, I decided to pay a nominal fee to a domain name service to create my own domain with an email.
• Using my new email handle (<my-name>@<my-new-domain>.net) I again applied for a 90-day evaluation license from NVIDIA using this link: https://www.nvidia.com/object/vgpu-evaluation.html
• This did NOT immediately result in being emailed a link for an evaluation / trial license. However, the next morning I received an email from NVIDIA.
• Yet, after registering my account at the nvid.nvidia.com portal using the link sent to my email by nvidia the online portal showed a message saying something like "your request is still under review", or something like that (apologies I forgot to take a screenshot of this).
• After a few more days someone from NVIDIA reached out to me with the following...
  "I wanted to follow up on your engagement with some of NVIDIA's online resources. If it's appropriate, would you like to hop on a brief call to discuss your current or future AI/ HPC projects? I'd be happy to connect you with my team if it made sense."
• I replied to the NVIDIA contact stating I wasn't able to use the 90 day evaluation yet because the portal said my request was still being reviewed. A few days after that, though I never received an email notifying me of any change, I happened to login to nvid.nvidia.com and low and behold, I had access to download drivers. FINALLY!

I then downloaded the 16.0 driver package using the method shown in the video guide at https://gitlab.com/polloloco/vgpu-proxmox in the "Obtaining the driver" section

The driver package included the host OS driver and guest OS drivers.

After installing the host OS drivers and rebooting I spun up a Windows 10 VM and then installed the guest OS drivers. And that did it.

 

[the1010boy]

A few more details of my setup, install & config in case it might help someone else down the road. ...or at least something for me to look back on in case I forget how I did it.

HARDWARE

• Motherboard
  • Machinist X99-D8-MAX
  • LGA2011-3 dual socket
  • Chipset: X99
  • Memory support: DDR4 ECC server memory and DDR4 desktop memory
  • Memory slot: DDR4*8 eight channels, support 2133/2400/2666/3000MHZ frequency memory, maximum support: 512G(desktop general memory/ECC memory)
  • Storage interface: SATA3.0*10, M.2 NVME protocol interface*2
  • Expansion interface: 4*PCI Expressx16
  • Onboard network card: 2*reaItek8111 Gigabit network card
  • Power interface: 24pin+8pin+8pin
  • Fan interface: 4pin*2, 3pin*2
  • PCB layers: 12-layer PCB+12-phase power supply
  • Motherboard size: 330mm*305mm
  • Motherboard specification: E-ATX
• CPU:
  • Dual Xeon E5 2699V3 (18 cores/36 threads)
  • Base frequency: 2.3GHz
  • Max Turbo Frequency: 3.6GHz
• Memory
  • 8*16GB DDR4 ECC RAM 2133mhz. (mechanic brand)
• GPU
  • AMD Radeon WX 7100 (for proxmox console access if every I need to troubleshoot, but I mostly just run the machine headless)
  • NVIDIA M10 (for vGPU usage by VMs)
• Storage
  • NVMe 2TB Silicon Power SSD
• Power Supply
  • SilverStone Technology 1200 watt fully modular 80 Plus Platinum ST1200-PTS

SOFTWARE

• Host OS
  • Proxmox 8.0-2
• Guest OS
  • Windows 10 x64
• NVIDIA Enterprise Drivers (http://nvid.nvidia.com)
  • Platform = Linux KVM 16.0
  • Platform Version = all supported
  • Product Version = 16.0
  • Release Date = 7/5/2023

CONFIG

Using this guide I did the following steps in the following order...​

• BIOS
  • Disabled CSM to force usage of UEFI BIOS only.
  • Set "Above 4G Decoding" to "Enabled".
    (this was required for my MoBo since it was set to "disabled" by default, and the MoBo would NOT complete the boot process or post anything with the Tesla M10 installed when this was disabled)
    
  • Verified Intel VT-D was enabled. (required for IOMMU)
• Packages
  • Added the community pve repo
    echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" >> /etc/apt/sources.list
  • Got rid of the enterprise repo
    rm /etc/apt/sources.list.d/pve-enterprise.list
  • Updated repositories
    apt update
  • Upgraded
    apt dist-upgrade
  • Installed specifically needed packages
    apt install -y git build-essential dkms pve-headers mdevctl
• Git repos and Rust compiler
  • Changed directory to /root/
  • cd /root
  • cloned vGPU Proxmox repo to /root/
    git clone https://gitlab.com/polloloco/vgpu-proxmox.git
  • Changed working directory to /opt
    cd /opt
  • Cloned vGPU unlock rs
    git clone https://github.com/mbilker/vgpu_unlock-rs.git
  • Installed the rust compiler
    curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal
  • Added rust binaries to your $PATH
    source $HOME/.cargo/env
  • Changed working directory to vgpu_unlock-rs
    cd vgpu_unlock-rs/
  • Compiled the library
    cargo build --release
• Create files for vGPU unlock
  • Made a director for vgpu unlock config
    mkdir /etc/vgpu_unlock
  • Created an empty config file
    touch /etc/vgpu_unlock/profile_override.toml
  • Created folders and files for systemd to load the vgpu_unlock-rs library when starting the nvidia vgpu services
    mkdir /etc/systemd/system/{nvidia-vgpud.service.d,nvidia-vgpu-mgr.service.d}
echo -e "[Service]\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so" > /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf
echo -e "[Service]\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so" > /etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf
  • DISABLED the unlock part since the Tesla M10 is a supported GPU for vGPU usage and "using the unlock stuff could introduce unnecessary complexity and more points of possible failure", or so says the guide, and I have no reason to disagree.
    echo "unlock = false" > /etc/vgpu_unlock/config.toml
• Enabling IOMMU
  • I had already verified earlier that Intel Vt-d was enabled in the BIOS.
  • Edited /etc/default/grub to include intel_iommu=on iommu=pt by adding them to the line with GRUB_CMDLINE_LINUX_DEFAULT="quiet" to look like this...
    GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
  • Saved changes to the file and updated grub by running this command...
    update-grub
• Kernel Modules
  • Loaded these kernel modules: vfio, vfio_iommu_type1, vfio_pci and vfio_virqfd needed for vGPU...
    echo -e "vfio\nvfio_iommu_type1\nvfio_pci\nvfio_virqfd" >> /etc/modules
  • Blacklisted the open source nvidia driver so that only the enterprise nvidia driver downloaded in my last post would be used by Proxmox for the Tesla M10...
    echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
  • Applied kernel configuration changes
    update-initramfs -u -k all
• REBOOT
  • Rebooted since some of the above changes require a reboot before those changes can be fully applied
    reboot
• Verified IOMMU was enabled
  • Ran the following command...
    dmesg | grep -e DMAR -e IOMMU
  • Looked for the line that said "IOMMU enabled" about the 3rd line down.
• NVIDIA Driver
  • 90 day free trial
    • Applied for a 90-day evaluation license from NVIDIA using this link: https://www.nvidia.com/object/vgpu-evaluation.html and made sure to use an email that was NOT a free email domain like gmail, hotmail, yahoo, etc.
  • Downloaded the Linux KVM driver package version 16.0, and extracted it.
    
  • Copied nvidia HOST driver file over to proxmox server using scp
  • scp NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run root@10.10.1.191:/root/.
  • ssh'd to my proxmox server
  • Changed the mode of the nvidia driver file so it can be run...
    chmod +x NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run
  • Installed the nvidia driver
    ./NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run --dkms
  • Rebooted the proxmox server
    reboot
• Finishing touches
  • Logged back into proxmox server via SSH and ran the nvidia-smi command...
    nvidia-smi
    ...which game me this output...
    
• Created a Windows 10 x64 VM
  • Created the VM with these settings...
    
    
  • Booted the VM and installed Windows 10 Pro x64 using the virtIO drivers to during install so the VirtIO block drive could be seen.
  • Logged into Windows and enabled remote desktop and made a note of the IP address the VM obtained from my DHCP server.
  • Downloaded, installed and configured Parsec for remote access that's gaming worthy.
• vGPU attach
  • Powered down the VM
  • Added one of the four Tesla M10 GPU's to my VM for use as a vGPU like so...
    
    ...I choose to go with the profile (MDev Type) that gave all 8GB of vRAM to my VM for that 1 of 4 GPUs on the Tesla M10. But I also tested it with the min profile that only allocated 2GB of VRAM which should allow me to assign that same vGPU to 3 other VMs also only using 2GB of vRAM thus allowing 4 VMs per GPU giving me a total of 16 VMs using a single Tesla card in a single x16 PCIe slot!!!
    
    I also made sure that the PCIe passthrough device was NOT YET set to "primary GPU" at this time...
    Then I booted the VM.
• NVIDIA Guest drivers
  • I then installed the nvidia guest drivers for the Tesla M10
    
    ...after which the device manager showed the vGPU as an NVIDIA GRID M10-8 GPU
    
• Disable the virtual display
  • Next I powered off the VM and set the Display > Graphics card = none
    
  • This was the best for I have found for making sure all apps use ONLY the NVIDIA vGPU passed through to the VM. And since I had remote desktop already enabled, as well as Parsec already installed and configured and had ways to get into the VM without needing to use the virtual display adapter proxmox adds to every VM at VM creation time.
  • At this time I also needed to set the PCI vGPU device to be the primary GPU like so...
    
  • After booting up the VM and connecting via Parsec, device manager now shows only one display adapter. yay! (combined that screenshot with the screenshot above because apparently there's a limit to the number of screenshots I can upload for this post ‍*shrug*)

All is working and I couldn't be happier!

 

[kingwilliam]

i think you have interest on below url

https://gitea.publichub.eu/oscar.krause/fastapi-dls

 

[mb69ad]

Hello,
I’ve successfully installed my Tesla M10 with the procedure described, bust since the 8.2 upgrade I’m losing the configuration and now I can’t proceed as usual assigning the GRID options. Does anyone is experimenting my problem?
Thanks

 

[codgician]

Hello,
I’ve successfully installed my Tesla M10 with the procedure described, bust since the 8.2 upgrade I’m losing the configuration and now I can’t proceed as usual assigning the GRID options. Does anyone is experimenting my problem?
Thanks

Unfortunately latest NVIDIA GRID drivers (16.5/17.1) is not compatible with 6.8. We will need to wait for Nvidia for an update.

 

[mb69ad]

Unfortunately latest NVIDIA GRID drivers (16.5/17.1) is not compatible with 6.8. We will need to wait for Nvidia for an update.

Thank you.

 

[the1010boy]

Hello,
I’ve successfully installed my Tesla M10 with the procedure described, bust since the 8.2 upgrade I’m losing the configuration and now I can’t proceed as usual assigning the GRID options. Does anyone is experimenting my problem?
Thanks

I also hit this issue after upgrading to PVE 8.2.

The solution that worked for me was to install the older kernel version used in PVE 8.1, kernel version 6.5 (I found this solution here).

These are the commands I ran to obtain and install kernel version 6.5 on proxmox PVE 8.2:

apt install proxmox-kernel-6.5.13-5-pve-signed

apt install proxmox-headers-6.5.13-5-pve

...then, run the following command to pin the newly installed older kernel version 6.5...

proxmox-boot-tool kernel pin 6.5.13-5-pve

Then reboot

After the system came back up I was able to successfully install NVIDIA drivers.