Mix cluster with CEPH + ZFS nodes

[DynFi User]

Hello,

We will migrate a working cluster based on four nodes (ZFS) into a larger cluster with three nodes based on CEPH.

We wanted to know if it would be "ok" to join all nodes on a mixed Proxmox cluster with :

• standalone nodes based on ZFS (Intel)
• working cluster nodes based on CEPH (AMD)

Of course the CEPH cluster network (private and public) will be on a dedicated network separated from the cluster network.

Main idea is to be able to recover things in case something does not go as expected.

 

[aaron]

If the ZFS nodes have the same network config to access the Ceph Public network, then that should be no problem. If the ZFS storage is only available on those 4 nodes, make sure to limit it in the storage config so PVE knows on which nodes it can expect that storage.

If you have some HA guests that are the ZFS storage, create a new HA group with the restricted checkbox enabled and limit that group to only these nodes. This way you can avoid that some VMs will end up on a node without the needed storage available.

 

[DynFi User]

CEPH public and private networks are totally separated from the other part of the storage.

My question was more related to the PVE cluster itself (corosync).
Would It cause any problem to have a mixed Corosync cluster with both CEPH based nodes (running on AMD proc) and ZFS based ones (running on Intel proc)?

 

[aaron]

Okay, well, the PVE cluster will not have any issue with different CPU vendors. Live migration between an Intel and AMD CPU will most likely crash the VM though.

The other thing that will be a problem, if both clusters are actively in use: you cannot add a node to another cluster if it is not empty as all configuration on the node (storage, guests, ssl certs, ...) will be overwritten.

This means, you will have to plan the migration a bit differently. You could back up the VMs on the standalone nodes, restore them on the Ceph nodes, do a clean install of the standalone nodes and add them to the Ceph cluster.

How easy that is, depends on how large the VMs are and how much space / downtime you have available during the migration phase. After that, you can use a mixed storage approach within one cluster. But as I have mentioned, depending on what you want to use, you will have to put in a few fail safes.

I hope that helps

 

[DynFi User]

Thanks for your reply.

We were more or less planning what you described on the the target scenario that we have.

Backup with our PBS of all VM + CT
Restore on the new Ceph Cluster

The only risky point is if the migration fails for one reason or another…
Then we will have to roll-back to the old cluster…

Large scale migration is always complex !