GPU virtualization in Proxmox (not passthrough)

[BQHKB]

Hello, and thanks in advance for any help/advice.

I'm interested in using a PCIe GPU in a Proxmox host as the primary hardware for rendering video for virtual machines. I've been having a hard time locating information about how to do this since search engines tend to return a ton of articles regarding passthrough - I actually did get passthrough working, but that's not what I'm looking to do. I'd like to subdivide a GPU and allocate portions of it to VMs similar to how the CPU is virtualized and subdivided. Is this even possible?

In a much more general sense, I'm interested how video rendering typically works in Proxmox. One of my hosts that could be a good example is using motherboard https://www.supermicro.com/en/products/motherboard/X11SSW-F - which has the BMC listed as the graphics controller. The CPU is a Xeon E3-1240 v6, which does not include any on-chip graphics controller. How would Proxmox render video for VMs on this hardware?

 

[Ramalama]

The ast2500 gpu cannot render anything. Forget this fake gpu.


Your options are:

- Intel gvt, that's the cheapest solutions, but requires an integrated intel gpu and newest supported platform is coffee lake.
- Provides multiple vgpus for passthrough.

- mxgpu s7150 (w7100 can be bios modded too). This is the easiest working solution, as this requires only a module to compile.
- you need to buy the gpu, it's not expensive, especially an w7100 with bios mod.

- nvidia consumer 10XX & 20XX cards (can be spoofed to quadro & tesla versions)
- this is the hardest of all options, and there is no official sources for an available license crack (it exists, just not publically).

- nvidia official quadro/tesla cards, most expensive option & not the best working one.

I would say, everyone should forget the nvidia way, the drivers are only available for red hat anyway, as far i know. But i seen some using this on proxmox, so im not sure here.
Get an s7150 or w7100, google for more information yourself and use mxgpu.
I personally tryed this and it works fine, just a bit compilation headaches.

But with ast2500/2600 you can do nothing. The hardware itself cannot render or anything.

Cheers

 

[BQHKB]

That info definitely helps. The w7100 might be within my budget, so I'll start considering that. I had taken a GTX970 out of my gaming desktop to try a similar setup, and your explanation clarifies why that didn't work.

I'm also interested in just knowing how would the system I have now is rendering video. I have a couple Windows Server VMs right now that work - I don't do anything graphics-intensive, but I RDP into them and can do basic stuff in the GUI. Would that video be getting rendered by the AST2500 or the Xeon CPU (that doesn't have integrated GPU)?

 

[Ramalama]

It is actually software emulated, there is no hardware acceleration at all. That means in short, your cpu is rendering the display output.

I think you could pass through the ast2500, to get your vm display output into bmc or on a real display.
But this wouldn't be 3d accelerated either, probably even slower as your cpu rendering.

Ast2500 is just made for simple display output or bmc, I don't even know if there are even any proper drivers for this chip for windows.

However the rendering performance, idk if you want to convert videos or something is comparable with an radeon rx460/gtx1060 at max.

 

[BQHKB]

Thanks! I'm glad to have it confirmed that without setting something particular up, the CPU renders video. I'm much happier with that being the case than having the AST2500 involved at all - I'm sure, just like you said, that it's really just there to drive the out-of-band management interface, and isn't well optimized for doing any real video processing.

Video conversion and also video streaming were the primary reasons I started looking into this in the first place. I do quite a bit of video conversion on my laptop, and if I could offload this to a VM running in Proxmox that would be great. I also have a VM running Plex Media Server on that system, and while it gets the job done, I'm sure it would perform better if I could offload the video streaming workload to a GPU. The last thought I've had right now is just that in general, if I had a powerful GPU in the Proxmox system, it'd be great to just have all the VMs use that to drive their GUIs.

 

[Droste]

Hello Ramalama,

it is the first time i read someone got the mxgpus S7150 or W7100 working on a proxmox environment with SR-IOV.

Probably you can share the steps how you got this working.
I'm sure nearly everybody in this forum should be glad to know more about your setup.

Greetings from a highly curious proxmox user with bad video performance in his vms

 

[Ramalama]

Im not having it actually right now build in...
That was my passthrough research and proxmox playing from mid december till mid january.
Since then my server is productive with waiting for a new gaming gpu, that I can't buy right now. We all know why... But we got lied, that bitcoins and whatever is the issue. It isn't. But that's a completely another topic.

However, mxgpu works, i said once that's probably the only way somewhere else in this forum. It doesn't means that it works amazingly good. Other solutions are much harder to realize (geforce spoofing, nvidia vGpu insane price's or intel gvt if your cpu doesn't have igp or you use amd).

The real and only issue is to compile gim with our kernel 5.4. But i got it working after sleepless nights and editing the gim sources to make it compatible.
I remember that my first tryes failed with some errors, after the module was compiled and loaded. But i got it working after some days.

Basically you need to get gim and patch it, one of those patches is directly in the gim github page, kernel > 5.0 merge something as far i remember and when you try later to compile you google all the error and you will find solutions + some things you have to correct yourself, but that wasn't that hard, just extremely time consuming for all the tryes to change stuff in gim.

However, it's possible. But it's not really an newbie friendly or a super stable solution to use it in production.
The main issue is only to compile gim, everything else was straight forward.

So i would probably say, that you can try to compile gim and try to get it loaded. Once you managed this, you can go ahead and buy an s7150 or an w7100 (didn't tryed the w7100, i didn't had it, but i readed that many had success with it).

Cheers

 

[Ramalama]

By the way, you should all probably wait.

Intel gvt is the most active maintained solution.
When the pcie based iris xe or however they name that cards, will get released, im pretty sure, those would support sr-iov.

Intel in general has the best linux/bsd support of the 3 big brands (amd/nvidia/intel).

For the background, even nvidia supports sr-iov on geforce rtx cards. (The hardware does)
But there is no official driver support available.

Amd has much never cards like v520 or basically all radeon pro's etc Support sr-iov.
But there aren't any drivers either.

They are closed source only for the big providers available like azure/red hat etc...

And this won't change, because it brings them a lot of money. So don't expect this ever even in future.

The only hope here, is intel.
But we will see xD

 

[Droste]

Thank you for your explanation.

At the moment the prices are little bit high. I saw the S7150v2 around 500€ a couple of weeks ago.
Maybe they dropping soon. Even the W7100 are extremley expensive.

I'll do some more research in this topic and if they become a little bit cheaper I give it a try.