Best architecture for running 50+ VDSs each with its own Proxmox VE on a large Dedicated Server (2 TB RAM)

[alexskysilk]

t is okay offering, i see there is financial reason for it.

This is how I look at it.

You are offering to operate a bus for people. but instead of selling seats, you are putting busses inside your bus.

Whats the use case?! If you're trying to offer a customer resources that they can distribute between disparate vms, thats simple enough to do WITHOUT that additional complexity (hint- thats what ALL vps providers already do.)

 

[ness1602]

Okay you offer customers an hypervisor. It makes sense, not much but it does. I would always use CEPH for that.

 

[alexskysilk]

Each Proxmox VPS is placed on a different physical node, and every three VPSs are linked together as a cluster using replication (no Ceph).

Thats... absurd. you're wasting all those resources without any benefit; a cluster is of no value if all resources are HOSTED BY THE SAME DEVICE.

 

[ness1602]

I mean it makes sense if you sell something like proxmox with 2-4cpus and 64gb of ram, so that you can spin 20 pms

 

[alexskysilk]

I mean it makes sense if you sell something like proxmox with 2-4cpus and 64gb of ram, so that you can spin 20 pms

so the PRODUCT is 20 VMs, not a hypervisor.

 

[ness1602]

Yes, vm-1-hypervisor. Strange but okay if he got money for it. Why not?

 

[alexskysilk]

Yes, vm-1-hypervisor. Strange but okay if he got money for it. Why not?

because its dumb. YOU might convolute a way to make it make sense; in my world, customers pay me to AVOID complexity. again, the product is the VM.

 

[UK SPEED]

3 Physical Nodes (Bare Metal Hosts)
Each node runs Proxmox VE
────────────────────────────────────────────────────────

[Node 1 - Physical Proxmox]
├─ VPS1 (Proxmox VE) ───────────────┐
├─ VPS2 (Proxmox VE) ───────────────┐
├─ VPS3 (Proxmox VE) ───────────────┐
├─ VPS4 (Proxmox VE)
├─ VPS5 (Proxmox VE)
├─ VPS6 (Proxmox VE)
├─ VPS7 (Proxmox VE)
├─ VPS8 (Proxmox VE)
├─ VPS9 (Proxmox VE)
└─ VPS10 (Proxmox VE)

[Node 2 - Physical Proxmox]
├─ VPS1 (Proxmox VE) ───────────────┼──► Cluster A (Replication)
├─ VPS2 (Proxmox VE) ───────────────┼──► Cluster B (Replication)
├─ VPS3 (Proxmox VE) ───────────────┼──► Cluster C (Replication)
├─ VPS4 (Proxmox VE)
├─ VPS5 (Proxmox VE)
├─ VPS6 (Proxmox VE)
├─ VPS7 (Proxmox VE)
├─ VPS8 (Proxmox VE)
├─ VPS9 (Proxmox VE)
└─ VPS10 (Proxmox VE)

[Node 3 - Physical Proxmox]
├─ VPS1 (Proxmox VE) ───────────────┘
├─ VPS2 (Proxmox VE) ───────────────┘
├─ VPS3 (Proxmox VE) ───────────────┘
├─ VPS4 (Proxmox VE)
├─ VPS5 (Proxmox VE)
├─ VPS6 (Proxmox VE)
├─ VPS7 (Proxmox VE)
├─ VPS8 (Proxmox VE)
├─ VPS9 (Proxmox VE)
└─ VPS10 (Proxmox VE)



Each VPS (1–10) runs its own Proxmox VE instance.
VPS1 from Node1/Node2/Node3 forms a 3-node Proxmox cluster (replication only).
Same applies to VPS2, VPS3, etc.
No Ceph, virtual environments only.

 

[alexskysilk]

makes a lot more sense- but still doesnt answer why you want the vps to run proxmox. Also, in this configuration you really want to use shared storage and not local drives. you can still use nvmes with ceph.

 

[ness1602]

All of this is okay, but how will you run ports passthrough?

 

[UK SPEED]

@alexskysilk The goal is to give each customer a Proxmox environment that is as isolated as possible from the underlying physical nodes, not a shared VPS-style setup.

 

[alexskysilk]

@alexskysilk The goal is to give each customer a Proxmox environment that is as isolated as possible from the underlying physical nodes, not a shared VPS-style setup.

well then, I dont have any useful advice for you. go forth and prosper

 

[UK SPEED]

ness1602 I have 3 nodes, each one with 24 NVMes

 

[ness1602]

Then okay, give them one disk(ceph) for os and one for VM and that is that. No raid inside and other bs.