Had to remove 1 Node from Proxmox/Ceph

[liszca]

Once I had 4 nodes running with Ceph and one could fail. Now that one has failed and is removed.

Is it possible to establish redundancy with the remaining 3 nodes?

From ceph -s

  data:
    volumes: 2/2 healthy
    pools:   7 pools, 193 pgs
    objects: 139.48k objects, 538 GiB
    usage:   1.6 TiB used, 1.2 TiB / 2.8 TiB avail
    pgs:     193 active+clean

how to read the numbers on "usage"?

 

[UdoB]

Once I had 4 nodes running with Ceph and one could fail.

My understanding: the reason for only one could fail was the default rule of "size=3/min=2". This rule is valid for any number of nodes. (As long as you do not change it.)

This rule stays active when you lose (or add) a node. Regarding redundany nothing changed: one node (of the surviving three now) can fail.

What has changed is the OSD usage: it has increased a little bit because the data formerly stored on the dead node now has to be stored on the three survivors.

Is it possible to establish redundancy with the remaining 3 nodes?

Usually: YES. Three is the minimum number. As soon as one of these fails you are "degraded" but everything should continue to work.

Just remove the daemons (OSD, Manager, Monitors) from that now unavailable node to clean up.

how to read the numbers on "usage"?

You have used 1.6 TiB, there is 1.2 TiB free. This sums up to 2.8 TiB. Easy, no?

 

[liszca]

Thanks for your help, I figured out there is redundancy exactly as described by you.

But what is this command used for:

ceph osd ok-to-stop 2

I thought it is useful to check if the redundancy is actually working, what is it really used for?

 

[UdoB]

I thought it is useful to check if the redundancy is actually working, what is it really used for?

Sorry, I have no idea, this is everything I know: https://docs.ceph.com/en/quincy/api/mon_command_api/#osd-ok-to-stop