Ceph Implementation

[CameronMG]

Hi Everyone,

Looking for some advice here. Been looking at the docs but im struggling to get my head around this.

So I've got a two node cluster that I'm about to flatten and set back up, as I was never happy with the intiial implementation that I had done.

Effectively, I'd like to implement CEPH in this new iteration, however I have some questions regarding this.

1) Both of my servers have 2 x 1GB NIC in them. Is there any way I can set it up so CEPH uses that, instead of going across my core switch?
2) Will this allow me to dynamically move VMs across hosts within the cluster at will?

Thanks in advance

 

[aaron]

Ceph is only useful with at least 3 nodes. Before that, it really doesn't make a lot of sense.

If you need to be able to recover a VM via HA to another node, consider ZFS + Replication or another shared storage.

2) Will this allow me to dynamically move VMs across hosts within the cluster at will?

You can already do that, though moving a local disk image around might take some time.

Again, another kind of shared storage (e.g. network share) or ZFS + VM replication will reduce live-migration times considerably.

 

[CameronMG]

Ceph is only useful with at least 3 nodes. Before that, it really doesn't make a lot of sense.

If you need to be able to recover a VM via HA to another node, consider ZFS + Replication or another shared storage.


You can already do that, though moving a local disk image around might take some time.

Again, another kind of shared storage (e.g. network share) or ZFS + VM replication will reduce live-migration times considerably.

Many thanks for your reply to this!

Just curious here, can CEPH still be set up and configured within a 2 node cluster or is it not possible and a minmum of three is required?

Could you also elaborate as to why its not worth it for a two node cluster? I'm just trying to wrap my head around this at the moment.

Cheers.

 

[aaron]

Just curious here, can CEPH still be set up and configured within a 2 node cluster or is it not possible and a minmum of three is required?

To use Ceph as intended and to learn how it works, you want to run a cluster under normal circumstances. That means, at least 3 nodes, so that Ceph can store 3 replicas, each on a different node.
Ceph will be even nicer if run on larger clusters. The rule-of-thumb is, the more resources you give it (disks, nodes) the better it will perform and recover from failures.

 

[brucekornfeld]

To use Ceph as intended and to learn how it works, you want to run a cluster under normal circumstances. That means, at least 3 nodes, so that Ceph can store 3 replicas, each on a different node.
Ceph will be even nicer if run on larger clusters. The rule-of-thumb is, the more resources you give it (disks, nodes) the better it will perform and recover from failures.

Another option for storage is StorMagic SvSAN. Here are the published docs on how to install it as the underlying storage layer in a 2 server node Proxmox cluster: https://support.stormagic.com/hc/en-gb/sections/14959498948381-SvSAN-Proxmox-7

For small environments, it provides the HA you want but doesn't require a lot of resources (1vCPU, 1GB memory, 1 drive)....

SvSAN free trial: https://stormagic.com/trial/

 

[pveuser113]

Please be transparent that your solution is NOT a 2-node solution and that providing complete high availability in a 2-node-only solution is impossible. A 3rd vote is always required.
As discussed previously, your solution requires a witness to be installed on a 3rd host (virtual or physical). Be advised that a 2-node Proxmox cluster is not a recommended or supported configuration. See this thread for additional info:
https://forum.proxmox.com/threads/c...th-stormagic-software-defined-storage.142472/