Best architecture for running 50+ VDSs each with its own Proxmox VE on a large Dedicated Server (2 TB RAM)

[alexskysilk]

t is okay offering, i see there is financial reason for it.

This is how I look at it.

You are offering to operate a bus for people. but instead of selling seats, you are putting busses inside your bus.

Whats the use case?! If you're trying to offer a customer resources that they can distribute between disparate vms, thats simple enough to do WITHOUT that additional complexity (hint- thats what ALL vps providers already do.)

 

[ness1602]

Okay you offer customers an hypervisor. It makes sense, not much but it does. I would always use CEPH for that.

 

[alexskysilk]

Each Proxmox VPS is placed on a different physical node, and every three VPSs are linked together as a cluster using replication (no Ceph).

Thats... absurd. you're wasting all those resources without any benefit; a cluster is of no value if all resources are HOSTED BY THE SAME DEVICE.

 

[ness1602]

I mean it makes sense if you sell something like proxmox with 2-4cpus and 64gb of ram, so that you can spin 20 pms

 

[alexskysilk]

I mean it makes sense if you sell something like proxmox with 2-4cpus and 64gb of ram, so that you can spin 20 pms

so the PRODUCT is 20 VMs, not a hypervisor.

 

[ness1602]

Yes, vm-1-hypervisor. Strange but okay if he got money for it. Why not?

 

[alexskysilk]

Yes, vm-1-hypervisor. Strange but okay if he got money for it. Why not?

because its dumb. YOU might convolute a way to make it make sense; in my world, customers pay me to AVOID complexity. again, the product is the VM.

 

[UK SPEED]

3 Physical Nodes (Bare Metal Hosts)
Each node runs Proxmox VE
────────────────────────────────────────────────────────

[Node 1 - Physical Proxmox]
├─ VPS1 (Proxmox VE) ───────────────┐
├─ VPS2 (Proxmox VE) ───────────────┐
├─ VPS3 (Proxmox VE) ───────────────┐
├─ VPS4 (Proxmox VE)
├─ VPS5 (Proxmox VE)
├─ VPS6 (Proxmox VE)
├─ VPS7 (Proxmox VE)
├─ VPS8 (Proxmox VE)
├─ VPS9 (Proxmox VE)
└─ VPS10 (Proxmox VE)

[Node 2 - Physical Proxmox]
├─ VPS1 (Proxmox VE) ───────────────┼──► Cluster A (Replication)
├─ VPS2 (Proxmox VE) ───────────────┼──► Cluster B (Replication)
├─ VPS3 (Proxmox VE) ───────────────┼──► Cluster C (Replication)
├─ VPS4 (Proxmox VE)
├─ VPS5 (Proxmox VE)
├─ VPS6 (Proxmox VE)
├─ VPS7 (Proxmox VE)
├─ VPS8 (Proxmox VE)
├─ VPS9 (Proxmox VE)
└─ VPS10 (Proxmox VE)

[Node 3 - Physical Proxmox]
├─ VPS1 (Proxmox VE) ───────────────┘
├─ VPS2 (Proxmox VE) ───────────────┘
├─ VPS3 (Proxmox VE) ───────────────┘
├─ VPS4 (Proxmox VE)
├─ VPS5 (Proxmox VE)
├─ VPS6 (Proxmox VE)
├─ VPS7 (Proxmox VE)
├─ VPS8 (Proxmox VE)
├─ VPS9 (Proxmox VE)
└─ VPS10 (Proxmox VE)



Each VPS (1–10) runs its own Proxmox VE instance.
VPS1 from Node1/Node2/Node3 forms a 3-node Proxmox cluster (replication only).
Same applies to VPS2, VPS3, etc.
No Ceph, virtual environments only.

 

[alexskysilk]

makes a lot more sense- but still doesnt answer why you want the vps to run proxmox. Also, in this configuration you really want to use shared storage and not local drives. you can still use nvmes with ceph.

 

[ness1602]

All of this is okay, but how will you run ports passthrough?

 

[UK SPEED]

@alexskysilk The goal is to give each customer a Proxmox environment that is as isolated as possible from the underlying physical nodes, not a shared VPS-style setup.

 

[alexskysilk]

@alexskysilk The goal is to give each customer a Proxmox environment that is as isolated as possible from the underlying physical nodes, not a shared VPS-style setup.

well then, I dont have any useful advice for you. go forth and prosper

 

[UK SPEED]

ness1602 I have 3 nodes, each one with 24 NVMes

 

[ness1602]

Then okay, give them one disk(ceph) for os and one for VM and that is that. No raid inside and other bs.

 

[Johannes S]

Why should one pay money to get a ProxmoxVE vserver if for the same (or lesser) amount one can get a regular vserver where I could install directly the OS of my choice (including PVE if one want to use it to host lxcs)? I can see a usecase where one might want to use such a thing as a lxc-hosting-plattform.
I can't imagine a reason, why anybody would want to take the performance penalty of nested virtualization for vms though. There are more than enough vserver providers, where I wouldn't get that performance penalty and I would also be isolated from other vms.

 

[UK SPEED]

Johannes ​

You pointed out an important aspect I should focus on.


To confirm:
If I run LXC containers inside Proxmox installed on a VPS/VDS, does this count as nested virtualization, or is it considered a normal and supported setup without significant overhead?


In other words, is using LXC inside Proxmox a recommended way to avoid nested KVM virtualisation?

 

[Johannes S]

If I run LXC containers inside Proxmox installed on a VPS/VDS, does this count as nested virtualization, or is it considered a normal and supported setup without significant overhead?

It has less overhead but i'ts also less isolated than a VM. A VM practically simulates a complete computer which comes with a performance penalty. With nested virtualization it gets even worse. An lxc doesn't has this problem but you can only install Linux systems with them and only of versions compatible with the kernel, systemd, cgroups etc base of the host. There are usecases, where this is fine and then there are usecases where this would absoluetly not be fine be it for security, compilance or simple technical reasons (e.g. if somebody wants to run Windows or BSD which isn't possible with Containers).

In other words, is using LXC inside Proxmox a recommended way to avoid nested KVM virtualisation?

This depends on the usecase, e.G. I would never host docker or podman containers from lxc.

I still don't get why somebody should rent a ProxmoxVE vserver from you if he could get a vserver without performance penalty from Hetzner, Netcup, Ionos, Amazon Google or some other cloud provider.

Sorry to be blunt: if you need to ask such questions you shouldn't do hosting, I wouldn't recommend your services. The main reason for renting a managed service is, that somebody else (who is hopefully more competent) is doing work you can't or don't want to do. If the service provider isn't actually more competent than the customer the service isn't needed.