Expanding Cluster and ceph with other node

[YmeY]

Hi ,
i have a 3-node cluster , with ceph
all current nodes have 4x 3,4TB Nve in the ceph OSD
i want to add a node the host is the same hardware but with 3x 1.9TB Nvme

my questions are;

What is the best way of expanding the OSD and can it be done with different storage, or is it better to get the same storage on the new node,
and what is best practice then with out creating a massive I/o outrage while expanding the OSD

thanks

 

[alexskysilk]

i want to add a node the host is the same hardware but with 3x 1.9TB Nvme

This is not a good idea. see below.

can it be done with defend storage

I imagine this is a typo but I cant guess what you mean here.

is it better to get the same storage on the new node,

In some ways yes, but generally speaking you should be allright as long as the OSDs in each node are roughly of equivalent performance characteristics AND EACH NODE HAS ROUGHLY THE SAME CAPACITY.

and what is best practice then with out creating a massive I/o outrage while expanding the OSD

ahh thats the real question It really depends on your cluster load. The default nice values are pretty reasonable but if you have a very busy filesystem you may need to adjust. More importantly, since you are not bringing to bear an additional 14.4TB (the capacity on an existing node) what you would want to do is divide, in real time, the total disk capacity you will end up with between the 4 nodes that will comprise your cluster. This is pretty painful and require a lot of handholding on a loaded cluster, and would be far easier if you have the option of blowing away the storage and start over.

Alternatively, bring sufficient disk space to bear so you dont need to remove existing OSDs, in which case, you can use the following strategies on minimizing rebalance impact:

1. bring the OSDs up and in, and immediately set their weight to 0. then increase gradually.
2. tunables:
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_recovery_op_priority 3

just be aware that at such a priority the rebalance will take a long time.

 

[YmeY]

Thanks ,
i think i fixed the typo but clear answer.. thank you again