Ceph OSDs marked out, but still rebalance when I remove them.

aarcane

Member
Jul 28, 2015
14
0
21
Okay, so reading the manual for Ceph tells me to mark an OSD as "Out" and then allow the pool to rebalance the data away from it before removing it, and then removing it should not require any rebalancing. Okay, so I did this. I marked an OSD as OUT. I added two OSDs to replace it. I let the rebalance run. For two days, I let it run, until everything was 100% active+clean, the new OSDs were up and in, the old OSD was up and OUT. I deleted the old OSD using the GUI. Immediately another day long OSD rebalance occured. What did I do wrong, or is this something wrong with ceph or is my understanding wrong, or what?
 

wigor

Member
Dec 5, 2019
40
5
8
Hey aarcane,

iirc you better set the crush weight of the "leaving" osd to zero. So the weight of the host is also altered.
Otherwise the second rebalancing occured due to altered host weight after destroying osd. Setting osd out on it´s own does not alter the host weight (iirc).
 

aarcane

Member
Jul 28, 2015
14
0
21
Hey aarcane,

iirc you better set the crush weight of the "leaving" osd to zero. So the weight of the host is also altered.
Otherwise the second rebalancing occured due to altered host weight after destroying osd. Setting osd out on it´s own does not alter the host weight (iirc).
That makes an odd sort of sense, and sounds like a bug, or something that should be warned about in the UI somewhere. Anyway, I know what to do next time.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!