Ceph OSD disk replacement

[Paspao]

Hello,

I am looking for updated documentation on correct procedure for Ceph OSD disk replacement.

Current Proxmox docs only cover OSD creation but is lacking of management procedures using pveceph commands.

I found this Ceph document but some commands are not giving the same output (ie:
service ceph status ) and I am not sure if it is 100% Proxmox version compatible.

In general can we mix procedures using traditional ceph commands with pveceph ?

Thanks to everyone who can spread some light.

P.

 

[alexskysilk]

In general can we mix procedures using traditional ceph commands with pveceph ?

yes. pveceph isnt an actual command binary, its a wrapper for ceph commands.

replacing a disk is super simple, and can be performed even from the gui:

1. down (stop) and out an osd (will probably already be in this state for a failed drive)
2. remove it from the tree and crush map ("destroy" in the gui)
3. replace disk
4. create new osd
5. profit

 

[Alwin]

I found this Ceph document but some commands are not giving the same output (ie:
service ceph status ) and I am not sure if it is 100% Proxmox version compatible.

We use upstream packages (+ some extra patches) and just provide our management stack around it. In general the ceph tools should work as expected.

 

[Waywatcher]

So glad I found this post. I was working on all the command lines for this. This was so much easier.

A note from my experience. The disk was failed but the mount point still existed on my server. Running 'destroy' from gui did not unmount the drive. After swapping the drive it was going to set the new mount point as sdm. I did a umount sdf, popped the new drive out and back in. Then adding the disk in GUI used the sdf mount point.

 

[dmulk]

So glad I found this post. I was working on all the command lines for this. This was so much easier.

A note from my experience. The disk was failed but the mount point still existed on my server. Swapping the drive it was going to set the new mount point as sdm. I did a umount sdf, popped the new drive out and back in. Then adding the disk in GUI used the sdf mount point.

Seems like this is a bug with Proxmox's GUI wrapper...? Seems like "Destroy" should also unmount the OSD's mount point, no?

 

[elimus_]

One more note for those that might look at this thread for HOWTO in future as I did now.

To unmount/disable/power down the sata device, IF system hasn't already done that for you
> echo 1 > /sys/block/(whatever)/device/delete

https://unix.stackexchange.com/ques...fely-remove-a-sata-disk-from-a-running-system

 

[readyspace]

Hi,

Is this the same procedure if I want to replace the SSD with a larger SSD than the original?

David

 

[Rasmus Linden]

Hi David!

Just noticed nobody answered your question.
Yes you can use this to swop to larger drives.
And there is a trick, remapping takes quite a long time, and you can greatly speedup the process by :

$ceph tell 'osd.*' injectargs '--osd-max-backfills 16'

All the best

Ras

 

[Adam Smith]

$ceph tell 'osd.*' injectargs '--osd-max-backfills 16'

Do you have to run this for every swap or is just the once sufficient?

 

[sb-jw]

The command is only persistent if you write it into the config. Otherwise you have to set the value again after restarting the specific service.

 

[Adam Smith]

The command is only persistent if you write it into the config. Otherwise you have to set the value again after restarting the specific service.

Perfect. Thank you.

Also, I did a little lookup. I had more luck with

# ceph tell 'osd.*' injectargs --osd-max-backfills 3 --osd-recovery-max 9

as described here.