[SOLVED] CEPH OSD(s) failing to initialize after a hardware change

[ResidentDataHoarder]

no matter what amount of cleanup I do, ceph will not let go of those OSD IDs that don't even exist

Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5b712fc3-97ba>
Running command: vgcreate --force --yes ceph-bf8aeb77-12ec-43b7-b530-2eef86c8124c /dev/sdc
 stdout: Physical volume "/dev/sdc" successfully created.
 stdout: Volume group "ceph-bf8aeb77-12ec-43b7-b530-2eef86c8124c" successfully created
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.8 --yes->
 stderr: purged osd.8
--> No OSD identified by "8" was found among LVM-based OSDs.
--> Proceeding to check RAW-based OSDs.
No OSD were found.

 

[ResidentDataHoarder]

I figured it out, unfortunately it was the thing I wanted to avoid doing, rewiring it to not use the stock wiring it comes with (already full of drives and not a lot of clearance to plug them in). I guess let this be a lesson, sometimes the correct solution is the thing you want to avoid doing the most.
Sorry for the quote that sounds like I pulled it out of a fortune cookie being the resolution to all this, but that's genuinely how I feel about it.

Hi,


You should take a look at the manual of your hardware because if your backplane doesn't have an expander, then your single cable may not work like you think.

If you have doubts, you can post the part number.

Best regards,

You were on the right track, and I had a feeling the whole time, subconsciously I didn't want it to be the case. When you wire it with the default configuration I think it uses the expanded, and when you wire each 4 to its own HBA instead it doesn't.