Have you ever thought about the idea of providing cloud service with hundreds of PVE clusters?

[imoniker]

Have you ever thought about the idea of providing cloud service with hundreds of PVE clusters?
The only problem might be the network's scalability, and it might not be an issue with the new BGP-EVPN SDN
Suppose we have thousands of tenants, we might need thousands of zones (1 per tenant) in total and several hundred zones per cluster at max. Dealing with hundreds of VRF (zone) should not be a problem for Debian.
We also need several vNet per zone, several subnet per vNet. It shouldn't be a problem either for Debian.
FRR might have some pressure to keep track lots of rules for each cluster (hundreds of zones in a cluster x tens or hundreds of peers per zone).
I don't know the scalability of FRR but I guess it's not a problem.

PVE is very stable and it's much easier than OpenStack. The only thing left is writing lots of scripts to automate.

What do you think?

 

[gurubert]

AFAIK the max cluster size of Proxmox is around 32 nodes. This is a corosync limitation.

 

[itNGO]

Have you ever thought about the idea of providing cloud service with hundreds of PVE clusters?
The only problem might be the network's scalability, and it might not be an issue with the new BGP-EVPN SDN
Suppose we have thousands of tenants, we might need thousands of zones (1 per tenant) in total and several hundred zones per cluster at max. Dealing with hundreds of VRF (zone) should not be a problem for Debian.
We also need several vNet per zone, several subnet per vNet. It shouldn't be a problem either for Debian.
FRR might have some pressure to keep track lots of rules for each cluster (hundreds of zones in a cluster x tens or hundreds of peers per zone).
I don't know the scalability of FRR but I guess it's not a problem.

PVE is very stable and it's much easier than OpenStack. The only thing left is writing lots of scripts to automate.

What do you think?

That is how we do it already.... and yes, automation and scripting was the key. Still hoping for some more "multi-cluster" Management in the future. But overall... it works really "amazing".....

 

[imoniker]

amazing. I think this technology is going to revolutionize the cloud industry in the next decade.
Public cloud is so expensive.
Proxmox provides an important alternative.

 

[imoniker]

AFAIK the max cluster size of Proxmox is around 32 nodes. This is a corosync limitation.

20 nodes per cluster x 50 clusters = 1000 servers.
50 clusters shouldn't be a problem with SDN

 

[lifeboy]

AFAIK the max cluster size of Proxmox is around 32 nodes. This is a corosync limitation.

32 nodes with 2 x AMD 128core Epyc V4 processors = 8192 vCPU's per cluster. Pretty big. But as others have said, the inter-cluster scripts and automations are the key to making it work.