Just added a node after 9.1.4 > 9.1.5 update and Ceph keys weren't installed

[Howard A2 Labs]

Went through the normal process of adding a node to the cluster and installing Ceph, then went to run ceph-volume lvm batch and it errored out for missing config. Figured no big deal, symlinked /etc/ceph/ceph.conf -> /etc/pve/ceph.conf, I've had to do that before.

Tried again, still errors out, turns out /var/lib/ceph/bootstrap-osd is empty... uh oh. Thought about just scping contents from another node, but I don't want to keep playing whackamole with a semi-busted node.

Is there a way to rerun the entire Ceph setup process on this node to ensure everything is populated?

PS: The cluster status page always showing the big red X and "You have at least one node without subscription." nag when you have a subscribed node down is really infuriating when you're already dealing with an irritating issue.

 

[Howard A2 Labs]

So I ran into this again and tried some more things and figured something out.

While ceph-volume lvm batch still did not work initially, it did work after adding one (temporary in my case) OSD via the webUI. Some part of Ceph setup necessary for the new node to add NVMe drives sliced up as multiple OSDs does not get done during Ceph install but only during first webUI OSD creation.