Reinstall Proxmox node with Ceph storage

[igloo]

Hello,

I'v got this setup: Proxmox 3, 4x nodes with Ceph hammer storage. OS is on one HDD, while Ceph is using multiple additional disks on each node (sda - OS, sdb and sdc - osds). One of the nodes went down because of the failed system disk. So at the moment it's 3x nodes. I suppose I need to follow this: Proxmox_VE_4.x_Cluster#Re-installing_a_cluster_node, it's about VE 4, and I assume it's still applicable. But I confused about Ceph here. I think the safest way would be to reinstall it from scratch as a new node (new hostname and IP). But what the routine to reintegrate old osds to the ceph cluster? Should I just do the following for each of the old osds one by one:

ceph osd out osd.X
/etc/init.d/ceph stop osd.X
ceph osd crush remove osd.X
ceph auth del osd.X
ceph osd rm osd.X

then

ceph-disk zap /dev/sdx
pveceph createosd /dev/sdx
ceph osd out osd.Y
/etc/init.d/ceph stop osd.Y
ceph osd crush add osd.Y 0.14 host=node01
/etc/init.d/ceph start osd.Y
ceph osd in osd.Y


to re-add osd as new ones one after one or there is a better way? Any input is welcome. Thank you!

 

[Mikepop]

Hello,

I'v got this setup: Proxmox 3, 4x nodes with Ceph hammer storage. OS is on one HDD, while Ceph is using multiple additional disks on each node (sda - OS, sdb and sdc - osds). One of the nodes went down because of the failed system disk. So at the moment it's 3x nodes. I suppose I need to follow this: Proxmox_VE_4.x_Cluster#Re-installing_a_cluster_node, it's about VE 4, and I assume it's still applicable. But I confused about Ceph here. I think the safest way would be to reinstall it from scratch as a new node (new hostname and IP). But what the routine to reintegrate old osds to the ceph cluster? Should I just do the following for each of the old osds one by one:

ceph osd out osd.X
/etc/init.d/ceph stop osd.X
ceph osd crush remove osd.X
ceph auth del osd.X
ceph osd rm osd.X

then

ceph-disk zap /dev/sdx
pveceph createosd /dev/sdx
ceph osd out osd.Y
/etc/init.d/ceph stop osd.Y
ceph osd crush add osd.Y 0.14 host=node01
/etc/init.d/ceph start osd.Y
ceph osd in osd.Y


to re-add osd as new ones one after one or there is a better way? Any input is welcome. Thank you!

Hello igloo, I've just follow the same guide yesterday with a similar setup and issue after an update and I had not to recreate osds again, were detected and brought in again after I restarted the node after install ceph.
The problem I have is with the monitor, I cannot recreate the monitor with the importing conf, https://forum.proxmox.com/threads/reinstall-monitor.41908/, this happened to you also? How did you sorted it?

 

[igloo]

Hello Mikepop,

Sorry, I am few steps behind you.. Thank you for your reply!