2 Node Proxmox Ceph Cluster

[sternbergsysadmin]

Hello,

we recently got new Servers for our organization and havent really had experience with Proxmox before. We migrated everything from VMWare to Proxmox. Worked really well.

Now we are facing the problem that we dont have Node failover since our Pool min_size is 2.
A little bit about our Cluster:

- Nodes: 2
- OSDs per Node: 11
- Single OSD Size: 3,84 TB

Main Pool Configuration:
- Size: 2
- min_size: 2
- Crush Rule: replicated_rule (default)
- Number of PGs: 128
- PG Autoscale Mode: yes
- Available Storage about 38 TB

So now i need a little help from you guys what our best options are

Buying another Server for a 3 Node Cluster would be perfect but we really cant afford it so we have to stick with 2 Nodes.

Since we have alot of OSDs per Node, would it be possible to get good redundance?

Maybe replicate the data on both Nodes and on each Nodes also on different OSDs. So even with if 1 Node fails we still have the data replicated on more then 1 OSDs on a single Node. Is erasure coding maybe an option?

Sounds safe to me if we can configure it that way. Or am i missing something?

Losing a bit of available storage isnt a problem either since we dont even come close to the now available 38 TB.

Our main goal is to get failover working.

Thank you in advance for your help

 

[ness1602]

If you can't afford three nodes, go back to zfs, it will save you a lot of sleepless nights.

 

[UdoB]

Ceph with 3/2 copies needs three Nodes - alive. If you want to be able to lose one Node and stay comfortable then you need four Nodes.

It works great as long as it runs. Problems with Ceph are painful - especially with a small cluster.

Disclaimer: I am NOT a Ceph specialist.

 

[sb-jw]

Maybe replicate the data on both Nodes and on each Nodes also on different OSDs. So even with if 1 Node fails we still have the data replicated on more then 1 OSDs on a single Node.

But that doesn't work with replica 2, because CEPH distributes the data to two servers and there to the OSDs. So you would have to use Replica 3 or even 4, but you would also lose storage space.

Is erasure coding maybe an option?

If you get 3 more servers, erasure coding would definitely be an option. It doesn't go below that. It is also best to start with 11 nodes, otherwise the performance will be extremely poor.

Sounds safe to me if we can configure it that way. Or am i missing something?

Yes, you forgot a lot of things. Your data is in great danger with Replica 2, so if you don't care about this, then you can do Replica 2. Otherwise, Replica 3 and at least 3 nodes are mandatory.

The reason is quite simple: if one node fails, e.g. due to a motherboard defect, then you have lost your redundancy. If even one more hard drive in the intact node fails, then that's it, your data is completely destroyed. You will no longer be able to achieve an intact condition after 3 days until the replacement part is delivered.

By the way, it's the same game even if both are still running. If one disk fails, the other server may not fail until CEPH is halthy again. Otherwise same game.

Buying another Server for a 3 Node Cluster would be perfect but we really cant afford it so we have to stick with 2 Nodes.

What is the reason for that? For example, the Dell Gen 13 servers (E5 v3/v4) are available for around 200 EUR. The Gen 14 (Scalable v1/2) is also available for around 800 EUR. The third server does not have to have the same specs as the others. You can also view it purely as a storage node and not shift any computing load to it.

 

[sternbergsysadmin]

But that doesn't work with replica 2, because CEPH distributes the data to two servers and there to the OSDs. So you would have to use Replica 3 or even 4, but you would also lose storage space.

So it would be possible that i change Replica to 4 or even 6 and that would keep 2/3 copies on each Node?
I know at cost of Storage space but as i said thats not a Problem.
So OSDs could fail and Ceph would still be functional?
That would be a temporary solution for us till we get a third node.

What is the reason for that? For example, the Dell Gen 13 servers (E5 v3/v4) are available for around 200 EUR. The Gen 14 (Scalable v1/2) is also available for around 800 EUR. The third server does not have to have the same specs as the others. You can also view it purely as a storage node and not shift any computing load to it.

Oh thats a good idea that i havent tought about. Thank you

 

[sternbergsysadmin]

If you can't afford three nodes, go back to zfs, it will save you a lot of sleepless nights.

i read a bit through the docs and it sounds like a good option if we cant get ceph working like ceph wants it

 

[ness1602]

CEPH minimum,as in absolute is 3 nodes,usually whatever size ,ram, etc ,and 2.5g. But ideally 5 nodes, 10g networking.
So, if you have two beefy nodes, get just one small node you can afford, and add disks on it.

 

[sternbergsysadmin]

CEPH minimum,as in absolute is 3 nodes,usually whatever size ,ram, etc ,and 2.5g. But ideally 5 nodes, 10g networking.

we got 20g networking for ceph.

how much cpu/ram should the storage server have for it to functions properly for ceph needs?
I read something online that ceph recommends about 32 GB for small clusters.

So, if you have two beefy nodes, get just one small node you can afford, and add disks on it.

yes thats the plan i think. small server with same or half the amount of osds but i total the same storage space.

 

[ness1602]

I would say 64gb if you plan vm/cts on them,but of course i have a few nodes with as low as 16gb in production. Just not with that much data on them.

 

[sb-jw]

As a rule, an OSD needs at least 4 GB of RAM, but you shouldn't run it that way, I recommend going for at least 6 GB per OSD. When it comes to the CPU, however, CEPH no longer gives any real information, but you should plan at least 2 cores for an SSD; with an NVMe, 8 can also be used. It's better to use fewer cores and higher clock speeds. The energy saving mode should also be deactivated, it costs a lot of latency and that directly kills performance.