Seeking Insights and Recommendations for a Standalone Compute Grid Solution inside proxmox


Active Member
Sep 9, 2019
I'm reaching here again insights and recommendations as I am currently in the process of reassessing our compute grid system. We've been relying on Hadoop/Spark/Yarn in isolated environments without the need for additional security measures. Unfortunately, they are no longer free and the cost has become significantly high (around $8,000 per server) to either add more nodes or upgrade to a newer version. So, I've decided to explore alternatives and install a new compute grid.

Here are our grid requirements:
  • Standalone compute process (CPU/RAM requirement known before execution).
  • No resource sharing between executions (when running a task with 100 jobs, each job is standalone).
  • Queue management for tasks when all resources are in use.
  • Multi-user support.
  • Ability to query task status and errors (though the latter is less critical as I write logs to a shared folder).
  • Easy addition of more nodes (even if it requires stopping and restarting the grid).
  • Support for GPU resources.
  • All hosts are Ubuntu 22.04.
  • All managers/workers will be managed by Proxmox.
  • Task can be executed on host directly (ubuntu 22) (docker wrapping not wanted to do time it needs to create )
We're aware of various enterprise solutions with heavy security measures, but our primary interest lies in a stable, simple solution, and free.

I was considering integrating Rancher and K3s. However, as they're only available via VMs and not LXC, I have some reservations. My experience is largely with LXC (we have around 100 different containers across the Proxmox cluster), and I assume the performance difference would be minimal.

If anyone has insights, suggestions, or alternatives to suggest, they would be greatly appreciated. Thank you in advance for your time and help.
Last edited:


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!