Proxmox Ceph multiple roots that has same host item but different weights

[lalaman]

I'm trying to do a two ceph pools with 3 nodes (Site A) and 3 nodes (Site B) with two different rules. The first rule is a stretched Ceph with Site A and Site B and the second rule is just a standalone ceph rule on Site A.

Each of the 6 nodes has 3 OSDs. So total there's 18 OSDs.

I'm wondering if you can put the same hosts (Node 1, 2, 3) with different weightages under different roots in the CRUSH map? Or is there another way to do it?

For example, what I was thinking:

Spoiler: Example CRUSH config

#All the node are similar to each other
host node1 {
    id -3        # do not change unnecessarily
    id -4 class ssd        # do not change unnecessarily
    # weight 10.47926
    alg straw2
    hash 0    # rjenkins1
    item osd.0 weight 3.49309
    item osd.1 weight 3.49309
    item osd.2 weight 3.49309
}
. 
.
.
.
root siteA_project1 {
    id -21        # do not change unnecessarily
    id -2 class ssd        # do not change unnecessarily
    alg straw2
    hash 0    # rjenkins1
    item node1 weight 6.98618
    item node2 weight 6.98618
    item node3 weight 6.98618
}

root siteB_project1 {
    id -22        # do not change unnecessarily
    id -1 class ssd        # do not change unnecessarily
    alg straw2
    hash 0    # rjenkins1
    item node4 weight 6.98618
    item node5 weight 6.98618
    item node6 weight 6.98618
}

root siteA_project2 {
    id -21        # do not change unnecessarily
    id -2 class ssd        # do not change unnecessarily
    alg straw2
    hash 0    # rjenkins1
    item node1 weight 3.49309
    item node2 weight 3.49309
    item node3 weight 3.49309
}

rule replicated_rule_project1 {
    id 0
    type replicated
    step take siteA_project1
    step chooseleaf indep 2 type host
    step emit
    step take siteB_project1
    step chooseleaf indep 2 type host
    step emit
}
rule replicated_rule_project2 {
    id 1
    type replicated
    step take siteA_project2
    step chooseleaf firstn 3 type host
    step emit
}

 

[gurubert]

Just put two buckets of type room (or datacenter if you like) under the root and the hosts under the room buckets.

The second rule would then take the room bucket for site A as starting point. No need for two root buckets.

 

[lalaman]

Just put two buckets of type room (or datacenter if you like) under the root and the hosts under the room buckets.

The second rule would then take the room bucket for site A as starting point. No need for two root buckets.

Got it. So my understanding is that you don't have to assign weightage at the site levels?
Should the storage be capped at the ceph pool level? (E.g the target size/ratio) Because the two pools are using the same disks in site A.

Thanks

Followed this guide btw for the CRUSH map rules for the stretched ceph.
https://docs.ceph.com/en/reef/rados/operations/stretch-mode/

 

[gurubert]

So my understanding is that you don't have to assign weightage at the site levels?
Should the storage be capped at the ceph pool level? (E.g the target size/ratio) Because the two pools are using the same disks in site A.

The weight of the buckets is always the sum of their contained items. Why would you want to change that?
All pools always use all OSDs (except when you use device classes). This makes Ceph very flexible wrt to disk usage.

If you want the second pool to use its "own" OSDs you need to assign a different device class to them and change the CRUSH rule for that pool.