Distributed Computing for LLM

[sjensen7]

Hey Proxmox Team,

You guys are smart as hell, and I am not, but I AM pretty visionary.

What would be absolutely HUGE for many enterprises (and homelabbers) right now would be to use distributed computing principles on their various devices to make their LLMs both faster (more resources/partitioning on inference) AND more accurate (more VRAM - or RAM and CPU).

There's a pretty new program out there called "exo" (https://github.com/exo-explore/exo) that is doing this.

I thought to myself: PVE folks already have clustering topology down... what if they applied that to having AI features in their Datacenter Manager software? Or even just in the datacenter of a cluster...

It'd be quite the challenge, I'm sure, but would be (as far as I know) something that competitors are not yet doing or capable of doing.

Anyway, hopefully this reaches some devs and you guys pick it up. If not, no worries, but it seemed like distributed AI in proxmox would be next level attractive (because it's a big need AND rare to see) to lots of enterprises.

 

[proxuser77]

What would be absolutely HUGE for many enterprises (and homelabbers) right now would be to use distributed computing principles on their various devices to make their LLMs both faster (more resources/partitioning on inference) AND more accurate (more VRAM - or RAM and CPU).

There's a pretty new program out there called "exo" (https://github.com/exo-explore/exo) that is doing this.

I haven't looked at the software in detail, but based on the description, I would say that it is possible to include Linux machines in such a cluster. In other words, if you have several Linux VMs with GPUs passed through on your Proxmox cluster, they should be able to become part of an Exo cluster. Not sure how Proxmox itself could help here, except to provide VMs or containers in which you can then could set up Exo. After all, that's what a hypervisor does, and the last time I checked, Proxmox VE was exactly that.

I thought to myself: PVE folks already have clustering topology down... what if they applied that to having AI features in their Datacenter Manager software? Or even just in the datacenter of a cluster.

Do you have any concrete ideas about how AI could meaningfully support users when managing their Proxmox servers, or in other words, real, practical use cases for AI in the Proxmox management interfaces? Or is this mainly about ticking the “AI” checkbox because everyone else is doing it?