AMD MxGPU on Proxmox cluster

[Vladimir Kharlashkin]

Good day!
In latest realese Proxmox 5.3 was declared vGPU technology from Intell & Nvidia. What about MxGPU with FirePro S7150 & S7150x2?
On GitHub created repository from AMD with GIM (GPU-IOV Module) and there is an instruction for Ubuntu Server 16.04.
Maybe someone has the necessary equipment, time and desire to try?

 

[mira]

In our tests here the AMD card was unstable and couldn't really be used for anything.

 

[Vladimir Kharlashkin]

In our tests here the AMD card was unstable and couldn't really be used for anything.

Is it possible more details about your tests?

 

[dcsapak]

Is it possible more details about your tests?

Hi,

yes of course.
The open source host driver (gim) is here: https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization
which compiled without problems on a current pve-kernel.

First try was with a consumer platform.
An Asus Z170-A with a i7 6600k
I could not get the MxGPU feature to work, the pci bus errored out with a 'not enough space' error, this was probably due to the mainboard
not implementing sr-iov completly correct (altough an intel i350 worked with sr-iov)
Passing through the complete card works, but under load the vm crashes and i get pci bus errors on the host. (Again i think this is due to bad platform implementation)

Second try was with an Supermicro X10SDV-6C-TLN4F (Bios 2.0) Xeon-D board.
At first, it did not work at all (host crashed on driver load), but this was due to a wrong bios setting (it has to load the 'legacy' oprom since the card
does not have a uefi bios). After correcting this, the driver loaded and the virtual functions appeared, and i could pass them through.
But still under load the vms crashed. I am not sure if this was due to the platform, a bad host / guest driver (i tried with 18.3Q from the amd site) or we maybe have a faulty card.

I will test again in a few weeks with an AMD Epyc Platform, if you want i can post my test results then. (Also we then probably see if the platform was the culprit)

 

[Vladimir Kharlashkin]

Thanks for your post. I saw you have this card on youtube video.
I have the opportunity to try this card, but there is no suitable server.

 

[pcgrubb]

Hello all,
I was hoping to hear if MxGpu has been enabled.
I know that it is difficult to get working on consumer motherboards.
but the proxmox solution is pretty much small business ready.
I was hoping to use mxGPu on a cisco server with a S7150x2
for a office installition but need to know if this is working before I pull
the trigger so to speak.
Anyways cheers.

 

[dcsapak]

the page is for an older pve version (5) but check this out: https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x
we currently don't test it anymore since we deemed it not worth it, as amd seemingly don't have any interest in maintaining this further

(also no clue if it works with current kernels)

an alternative could be the current intel datacenter flex gpus, but still running tests with it. i have no idea if/when we'll publish a page like the one above for the intel datacenter gpus, but wendel from level1techs posted a guide for it on his forums:
https://forum.level1techs.com/t/intel-i915-sr-iov-mode-for-flex-170-proxmox-pve-kernel-6-5/208294