Add new disks

[Mn1sh]

We have a 3 node proxmox/ceph cluster ... each with 4 x4 TB disks

1) If we want to add more disks , what are the things that we need to be careful about?


Will the following steps automatically add it to ceph.conf?
ceph-disk zap /dev/sd[X]
pveceph createosd /dev/sd[X] -journal_dev /dev/sd[Y]

where X is new disk and Y is the journal disk.

2) Is it safe to run different number of OSDs in the cluster, say one server with 5 OSD and other two servers with 4OSD ? Though we have plan to add one OSD to each server.


3) How do we safely add the new OSD to an existing storage pool?

 

[t.lamprecht]

Will the following steps automatically add it to ceph.conf?
ceph-disk zap /dev/sd[X]
pveceph createosd /dev/sd[X] -journal_dev /dev/sd[Y]

Yes, although the zap-disk is not normally not needed here, pveceph should do that itself (so attention which block device you specify).

2) Is it safe to run different number of OSDs in the cluster, say one server with 5 OSD and other two servers with 4OSD ? Though we have plan to add one OSD to each server.

Yes ceph can handle that quite good, but it still prefers an uniform storage architecture over that.

Oh and if you add an OSD the cluster normally re-balances itself, that can put quite a load on the network/IO/servers. So you could reweight the new OSD to 0 at first and then increment it a little over the next time so that there is a more balanced load which ensures that everything stays up and running, you want to increment it up to 1.
For that exec

ceph osd reweight <newosdid> 0.0

Directly after you added the osd, <newosdid> is an integer and the id of the new osd, count simply one up from your highest osd id, which should be 11 (ids are zero indices) e.g. do:

pveceph createosd /dev/sd[X] -journal_dev /dev/sd[Y] && ceph osd reweight 12 0.0

All pools have the additional space then available. You may want to increase the placement group number a bit

ceph osd pool set {pool-name} pg_num {pg_num}

See: http://docs.ceph.com/docs/master/rados/operations/placement-groups/#a-preselection-of-pg-num for more details.