CEPH replication to different hosts instead of OSDs (PVE 6.2)

KORG

Renowned Member
Dec 25, 2012
28
0
66
Hello,

Could you please advice on how to safely change the replica's to be on different hosts, instead of OSDs for next crush map (PVE 6.2):

Code:
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class ssd
device 4 osd.4 class ssd
device 5 osd.5 class ssd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
device 9 osd.9 class ssd
device 10 osd.10 class ssd
device 11 osd.11 class ssd
device 12 osd.12 class hdd
device 13 osd.13 class hdd
device 14 osd.14 class hdd
device 15 osd.15 class ssd
device 16 osd.16 class ssd
device 17 osd.17 class ssd
device 18 osd.18 class hdd
device 19 osd.19 class hdd
device 20 osd.20 class hdd
device 21 osd.21 class ssd
device 22 osd.22 class ssd
device 23 osd.23 class ssd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root

# buckets
host cloud1 {
    id -3        # do not change unnecessarily
    id -9 class ssd        # do not change unnecessarily
    id -2 class hdd        # do not change unnecessarily
    # weight 24.449
    alg straw2
    hash 0    # rjenkins1
    item osd.3 weight 0.873
    item osd.4 weight 0.873
    item osd.5 weight 0.873
    item osd.0 weight 7.277
    item osd.1 weight 7.277
    item osd.2 weight 7.277
}
host cloud2 {
    id -5        # do not change unnecessarily
    id -10 class ssd        # do not change unnecessarily
    id -4 class hdd        # do not change unnecessarily
    # weight 24.449
    alg straw2
    hash 0    # rjenkins1
    item osd.9 weight 0.873
    item osd.10 weight 0.873
    item osd.11 weight 0.873
    item osd.6 weight 7.277
    item osd.7 weight 7.277
    item osd.8 weight 7.277
}
host cloud3 {
    id -7        # do not change unnecessarily
    id -11 class ssd        # do not change unnecessarily
    id -6 class hdd        # do not change unnecessarily
    # weight 24.449
    alg straw2
    hash 0    # rjenkins1
    item osd.15 weight 0.873
    item osd.16 weight 0.873
    item osd.17 weight 0.873
    item osd.12 weight 7.277
    item osd.13 weight 7.277
    item osd.14 weight 7.277
}
host cloud4 {
    id -13        # do not change unnecessarily
    id -14 class ssd        # do not change unnecessarily
    id -15 class hdd        # do not change unnecessarily
    # weight 24.449
    alg straw2
    hash 0    # rjenkins1
    item osd.18 weight 7.277
    item osd.19 weight 7.277
    item osd.20 weight 7.277
    item osd.21 weight 0.873
    item osd.22 weight 0.873
    item osd.23 weight 0.873
}
root default {
    id -1        # do not change unnecessarily
    id -12 class ssd        # do not change unnecessarily
    id -8 class hdd        # do not change unnecessarily
    # weight 97.796
    alg straw2
    hash 0    # rjenkins1
    item cloud1 weight 24.449
    item cloud2 weight 24.449
    item cloud3 weight 24.449
    item cloud4 weight 24.449
}

# rules
rule replicated_rule {
    id 0
    type replicated
    min_size 1
    max_size 10
    step take default
    step chooseleaf firstn 0 type host
    step emit
}
rule ssd-pool-rule {
    id 1
    type replicated
    min_size 1
    max_size 10
    step take default class ssd
    step choose firstn 0 type osd
    step emit
}
rule hdd-pool-rule {
    id 2
    type replicated
    min_size 1
    max_size 10
    step take default class hdd
    step choose firstn 0 type osd
    step emit
}

# end crush map
 
As I correctly understood, next steps should be taken:

1. ceph osd getcrushmap -o /tmp/mycrushmap
2. crushtool -d /tmp/mycrushmap > /tmp/mycrushmap.txt
3. Change ssd-pool-rule and hdd-pool-rule with (step chooseleaf firstn 0 type host):

Code:
rule ssd-pool-rule {
    id 1
    type replicated
    min_size 1
    max_size 10
    step take default class ssd
    step chooseleaf firstn 0 type host
    step emit
}
rule hdd-pool-rule {
    id 2
    type replicated
    min_size 1
    max_size 10
    step take default class hdd
    step chooseleaf firstn 0 type host
    step emit
}

4. crushtool -c /tmp/mycrushmap.txt -o /tmp/mycrushmap.new
5. ceph osd setcrushmap -i /tmp/mycrushmap.new

Can anyone please confirm/correct me if I wrong?

Thank you guys
 
Last edited:
Guys could you please briefly look if steps I've mentioned are correct?
 
Here is current tree from crushmap:

Code:
# ceph osd crush tree --show-shadow
ID  CLASS WEIGHT   TYPE NAME           
-12   ssd 10.47583 root default~ssd   
 -9   ssd  2.61896     host cloud1~ssd
  3   ssd  0.87299         osd.3       
  4   ssd  0.87299         osd.4       
  5   ssd  0.87299         osd.5       
-10   ssd  2.61896     host cloud2~ssd
  9   ssd  0.87299         osd.9       
 10   ssd  0.87299         osd.10     
 11   ssd  0.87299         osd.11     
-11   ssd  2.61896     host cloud3~ssd
 15   ssd  0.87299         osd.15     
 16   ssd  0.87299         osd.16     
 17   ssd  0.87299         osd.17     
-14   ssd  2.61896     host cloud4~ssd
 21   ssd  0.87299         osd.21     
 22   ssd  0.87299         osd.22     
 23   ssd  0.87299         osd.23     
 -8   hdd 87.32391 root default~hdd   
 -2   hdd 21.83098     host cloud1~hdd
  0   hdd  7.27699         osd.0       
  1   hdd  7.27699         osd.1       
  2   hdd  7.27699         osd.2       
 -4   hdd 21.83098     host cloud2~hdd
  6   hdd  7.27699         osd.6       
  7   hdd  7.27699         osd.7       
  8   hdd  7.27699         osd.8       
 -6   hdd 21.83098     host cloud3~hdd
 12   hdd  7.27699         osd.12     
 13   hdd  7.27699         osd.13     
 14   hdd  7.27699         osd.14     
-15   hdd 21.83098     host cloud4~hdd
 18   hdd  7.27699         osd.18     
 19   hdd  7.27699         osd.19     
 20   hdd  7.27699         osd.20     
 -1       97.79596 root default       
 -3       24.44899     host cloud1     
  0   hdd  7.27699         osd.0       
  1   hdd  7.27699         osd.1       
  2   hdd  7.27699         osd.2       
  3   ssd  0.87299         osd.3       
  4   ssd  0.87299         osd.4       
  5   ssd  0.87299         osd.5       
 -5       24.44899     host cloud2     
  6   hdd  7.27699         osd.6       
  7   hdd  7.27699         osd.7       
  8   hdd  7.27699         osd.8       
  9   ssd  0.87299         osd.9       
 10   ssd  0.87299         osd.10     
 11   ssd  0.87299         osd.11     
 -7       24.44899     host cloud3     
 12   hdd  7.27699         osd.12     
 13   hdd  7.27699         osd.13     
 14   hdd  7.27699         osd.14     
 15   ssd  0.87299         osd.15     
 16   ssd  0.87299         osd.16     
 17   ssd  0.87299         osd.17     
-13       24.44899     host cloud4     
 18   hdd  7.27699         osd.18     
 19   hdd  7.27699         osd.19     
 20   hdd  7.27699         osd.20     
 21   ssd  0.87299         osd.21     
 22   ssd  0.87299         osd.22     
 23   ssd  0.87299         osd.23
 
Crush rules was created with commands:

Code:
ceph osd crush rule create-replicated ssd-pool-rule default osd ssd
ceph osd crush rule create-replicated hdd-pool-rule default osd hdd
 
Is it safe change chooseleaf type to host from osd?
Now i delployed test cluster on one node and add three nodes to him. Now i want to bring it back to recommended configuration.
Can i change crushmap to host from osd back without safely?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!