Hyper-converged PVE and CEPH, Single PVE cluster with multiple CEPH clusters?

[dmulk]

Hey all,

JOOC, can PVE be configured so that there is a single PVE Cluster with multiple CEPH clusters. In other words, I want to run all my PVE nodes in one cluster but have half serving one CEPH cluster and the other half serving another, separate, CEPH cluster. (I'm aware that I can define different pools in a single cluster...I'm talking about different clusters..).

Thanks,
<D>

 

[aaron]

I not sure but I think our (GUI) tooling around Ceph cannot deal with two clusters.

What is your use case?

 

[dmulk]

@aaron Sorry for the delayed response. Somehow taking 2 weeks off during the holidays = 2 months behind at work....

In summary, I want some redundancy at the CEPH cluster level while at the same time allowing me to freely migrate compute (VM/Containers) around all nodes as needed. Simply put as the CEPH clusters scale, it would be nice to upgrade one, while leveraging the other to minimize risk.

Another way to accomplish this would be to have two separate PVE/CEPH hyper-converged clusters if there was the ability to migrate VMs\Containers between separate PVE clusters. (although the single PVE / multiple CEPH clusters would be preferred).

Thoughts?

 

[Alwin]

In summary, I want some redundancy at the CEPH cluster level while at the same time allowing me to freely migrate compute (VM/Containers) around all nodes as needed. Simply put as the CEPH clusters scale, it would be nice to upgrade one, while leveraging the other to minimize risk.

With PVE+Ceph cluster, you have the redundancy on upgrade. You migrate the VM/CT to other nodes and upgrade one node at a time.

Another way to accomplish this would be to have two separate PVE/CEPH hyper-converged clusters if there was the ability to migrate VMs\Containers between separate PVE clusters. (although the single PVE / multiple CEPH clusters would be preferred).

ATM, there is no build-in way to replicate VM/CT to different clusters. You can use rbd-mirror to replicate rbd images between clusters, but you need to take care about the configs yourself.
https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring

 

[dmulk]

Thanks for the tip on the CEPH, RBD Mirroring, Alwin, I'll give that a read.

With regards to "cluster redundancy", sorry if I wasn't more clear. While it's true that there is "some" redundancy INSIDE a ceph cluster, I'm looking for more redundancy one level up. So, two Ceph clusters. Yes I can run multiple CEPH clusters already but it means that I would need to run multiple PVE clusters also (If I'm hyper-converged). I'm looking for a way to run multiple CEPH clusters with a single (hyper-converged) PVE environment. I think you've confirmed that there isn't a way to do this right now.

Thanks for the time, guys!

<D>

 

[Alwin]

With regards to "cluster redundancy", sorry if I wasn't more clear. While it's true that there is "some" redundancy INSIDE a ceph cluster, I'm looking for more redundancy one level up. So, two Ceph clusters. Yes I can run multiple CEPH clusters already but it means that I would need to run multiple PVE clusters also (If I'm hyper-converged). I'm looking for a way to run multiple CEPH clusters with a single (hyper-converged) PVE environment. I think you've confirmed that there isn't a way to do this right now.

No there isn't.

I fail to see the benefit of multiple Ceph clusters on the same hardware. Please explain. Thanks.

 

[dmulk]

No there isn't.

I fail to see the benefit of multiple Ceph clusters on the same hardware. Please explain. Thanks.

I want to avoid down time with my workloads. I think there is a benefit at a certain scale if we had the ability to balance compute, in total, regardless of what is happening with CEPH environment(s).

There's always risk when changes are made to a CEPH cluster (say during an upgrade). If a business has minimal maintenance opportunities to assume an upgrade risk, then it would be extremely beneficial to be able to keep a "stable" storage environment in tact, live migrate all compute (vm's/containers) to the "stable" ceph nodes, roll through upgrading the nodes running the "other" CEPH cluster and then once it's confirmed things are "good" rebalance the workload to take advantage of the upgraded compute.

Some of this can be accomplished inside a single PVE/single CEPH cluster (using CEPH pools, etc) or multiple single / single clusters but it's laborious at a certain scale.

If there were a supported way to live migrate compute between separate PVE's then that would probably also accomplish the same thing (as we can already do that with storage obviously). I'm not sure what the lift for the devs would be to accomplish this in the PVE, but thinking about it, this might be the "best" way to do it.



<D>

 

[dmulk]

Something like this would probably accomplish what I'm looking for:

https://forum.proxmox.com/threads/multi-datacenter.35043/

(Assuming live migration for compute was a feature)

<D>

 

[Alwin]

What you describe is a multi cluster management. It is on the list of things to implement, but as you can imagine it is not a trivial task.

 

[dmulk]

What you describe is a multi cluster management. It is on the list of things to implement, but as you can imagine it is not a trivial task.

I can only imagine. That said, it's definitely a feature that would take Proxmox to the next level.

 

[Volker Lieder]

Hi,
we have running a single proxmox cluster with 2 Ceph clusters. But its not possible to maintain ceph via GUI, so we installed the CEPH Clusters from cli ont he relevant nodes. Works good but is more work from cli

 

[Robert.H]

I would also like to connect proxmox to 2 external ceph clusters so I can do live storage migration between them for example.
Thanks.

 

[Volker Lieder]

Hi Robert,
its possible, even if you already have the other two ceph clusters configured. If you need some assistence on this dont hesitate to contact me with more details.
Best regards,
Volker

 

[Robert.H]

Yes, got it working simply using the GUI
The catch is in the "key" field where you don't paste juste the key but the whole block like:
[client.admin]
key = .....
And make sure to put all the ceph mon space delimited.
Thanks

 

[Lephisto]

You might want to talk a look at:

https://github.com/lephisto/crossover

I use this to have incremental DR cold-standby copies in separate clusters, do (near) live migration with minimum downtime between different clusters and so on.