Managing Multiple Clusters

[hypsin]

I am currently running a single 6-node Proxmox cluster and looking to add 3 more nodes. There is really no need for me to be running all 9 nodes in a single cluster so I thinking about split it up into 3 clusters x 3 nodes each.

Couple questions if anyone has experience running multi-cluster setups:

1. Will I be able to administer all 3 clusters from a single node GUI?
2. Is there any limitation on how many clusters/nodes can reside on the same network? Or would it be best to allocate a separate VLAN to each cluster?
3. Does Proxmox allow offline/online VM migration between different clusters with identical node hardware/layout?

 

[Astraea]

1. I don't believe that you can. Each cluster would have a separate GUI.
2. I am sure you could put them on the same network, but I think it would be better to separate them out into their own network or VLAN.
3. I don't believe you can migrate VMs or containers between clusters.

I looked as doing a 3 and 3 node split on my setup but in the end, decided to do a 6 node cluster so that I could use 1 GUI and also move VMs and containers between all the nodes on the cluster. I have 3 different computer types/hardware setups in my cluster and I can move between all of them no problem in online and offline mode provided the allocated CPUs and RAM do not exceed the limits of the node I am moving to.

 

[ertanerbek]

Proxmox not have Vcenter type management system.

 

[hypsin]

So, let's say I want to scale to 24 nodes with a dedicated 24-port gigabit switch for corosync, would a single cluster be able to support it or would corosync still have a full meltdown?

It would be neat to be able to create "quorum groups", each with its own corosync domain, while still being part of the same cluster for ease of management. But not sure if that's possible with the way how pmxcfs works.

 

[ertanerbek]

At proxmox system, the cluster is exactly the host itself, it is mean cluster managign by all host.. But you can use diffrent ethernet for guest migration I think this feature was come with 6.**, for cluster system you really have very big host group it is like professional DC services... I think 24 host management can not be easy with proxmox standart system; so what you can ?

1. You can use oVirt instead of Proxmox, also oVirt better than proxmox for professional services... For little cluster for SOHO or SMB or for learning LAB, for Home LAB best of best Proxmox but wihtout Centrall managemenet system for cluster group can not be good for professional services...

2. if you are programmer or you can pay for centrall management software. Proxmox also API solition, like this " https://pve.proxmox.com/pve-docs/api-viewer/ " or "https://pve.proxmox.com/wiki/Proxmox_VE_API "

 

[hypsin]

Custom VM migration between clusters seems to be trivial and can probably be wrapped in a very simple shell script (SSH config/disks over).

But lack of centralized management for a large cluster or multiple small clusters is a bummer. If it bugs me too much I may indeed hook into API and write my own front end, but that's extra work I just don't have time for.

So, is the general consensus that Proxmox is for small business and hobby deployments and does not scale well for production use? Because you've could have fooled me - I've been running my 6 node cluster with ton of VMs (web hosting) for over a year now with literally 0 issues.

 

[guletz]

Hi,

If I remember I think that multi-data-center option is on the devs list. Many users had ask this option for many times.
As I see on others users post, some guys use several different clusters without problems. Yes live-migration between different cluster could not work. But with zfs storage, you can migrate(offline) any VM/CT with a small bash script.

Good luck!

 

[ertanerbek]

Custom VM migration between clusters seems to be trivial and can probably be wrapped in a very simple shell script (SSH config/disks over).

But lack of centralized management for a large cluster or multiple small clusters is a bummer. If it bugs me too much I may indeed hook into API and write my own front end, but that's extra work I just don't have time for.

So, is the general consensus that Proxmox is for small business and hobby deployments and does not scale well for production use? Because you've could have fooled me - I've been running my 6 node cluster with ton of VMs (web hosting) for over a year now with literally 0 issues.

If you have more time for manage HOST and GUEST whay not ?

For your purpose, best is management sofware for HOST and cluster... Also Management software not means only manage HOST / CLUSTER / GUEST; management software also means DRS, sDRS or similar system for easly total system source use without planing... Also proxmox have many diffrent storage soliton but can not spport SAN Shared storage system direcly, so SAN means IO king but PROXMOX have hyperconverged with local disk ( ZFS,LVM ), glusterFS, CEPH or remote system with Ethernet Based ISCSI, Gluster, RDB ( Ceph ) or Soft Link ZFS,NFS so can not create big IO like SAN.. also you tell to me I was manage my web server, web server means low BW high IO like DB what you use on storage side how you can support that IO request without SAN ...?

Brother I was work on this market more than 20 years and please do not tell to me Proxmox ok for DC management. I was manage more than hundreds Virtualization HOST and thousands thousands diffrent type GUEST with another virtualization software... If proxmox want enter enterprise virtualization market must be need centrall management ( Free or licanced... ) and need automize source management system ( Free or Licanced ) also must be fully support SAN Storage system

 

[hypsin]

also you tell to me I was manage my web server, web server means low BW high IO like DB what you use on storage side how you can support that IO request without SAN ...?

1. Reverse Proxy your webservers so you don't have to render or even hit storage with every request.
2. 4x SAS SSD in local RAID10 is more than enough to keep up with dozens of webserver VMs on same node.
3. DB - cache what you can, leverage columnstore compression and tune your indexes to keep DBMS IO down. If you need dedicated SAN to host webserver back-end then maybe overall architecture is the bottleneck, not storage.

 

[ertanerbek]

I talking about 12~20 gigabit real time traffic... moret han 14K session on only one country based TV Page.. I was survive that structure with dual A10 LB ...

 

[hypsin]

So, back to my question - can Proxmox 6.x easily handle 24 nodes on a single cluster without too much fiddling with corosync?

 

[ertanerbek]

https://pve.proxmox.com/wiki/Cluster_Manager

You can grow up your cluster until 32 Node.. For management ? What you want from Proxmox ? What is your requiriment or expentations and what is you storage infrastructure, after all node menas CPU adn RAM and KVM easly transfer memory from network but disk not like that ...

 

[Dominic]

So, back to my question - can Proxmox 6.x easily handle 24 nodes on a single cluster without too much fiddling with corosync?

You might be interested in this thread.

 

[hypsin]

You might be interested in this thread.

Perfect, thank you sir!

I'll keep scaling it up until it breaks. Hopefully by then we'll have knet multicast support or, even more ideal, multi-cluster management!