Ceph crush retention after reboot

[Spiros Pap]

Hi all,

I have a problem with my ceph installation, where my crush table changes after a reboot.
I have configured my ceph crush like this (i am giving the relevant config):
host px1 {
item osd.1 weight 1.637
item osd.2 weight 1.637
item osd.3 weight 1.637
item osd.4 weight 1.637
}
host px1-ssd {
item osd.6 weight 0.830
item osd.7 weight 0.830
item osd.5 weight 0.830
}

which means that a have created two hosts, where px1 contains the SAS disks and px1-ssd contains the SSD disks. As you can guess all disks in px1,px1-ssd belong to the same physical host. This config works for me, in general. The problem is that when I reboot the host, all OSDs in px1-ssd, go under px1, like this:
host px1 {
item osd.1 weight 1.637
item osd.2 weight 1.637
item osd.3 weight 1.637
item osd.4 weight 1.637
item osd.6 weight 0.830
item osd.7 weight 0.830
item osd.5 weight 0.830
}
host px1-ssd {
}

'px1' is the name of the host.

Is that normal behaviour? I want my crush to stay as it is and not change, on every reboot....
How can i fix this?

Thanx,
Sp

 

[Alwin]

Not that I would know of. Is the new crush map distributed to all MONs/OSDs? An alternative approach to the separation by host are the device classes. Way less work for separation by devices.
https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_ceph_crush_amp_device_classes

 

[Spiros Pap]

Hi,

Yes, all nodes, report the same crush file. When i make a change, it is updated to all hosts (as far as i can tell by getting the crushmap from each host). The nodes are in a cluster and I guess the crush is distributed across all nodes.

I'll see into device classes.

Sp