Creating/Using Multiple Ceph Pools

[jeff Wabik]

Hardware configuration:

• 6 systems in a ProxMox cluster. Works fine.
• Each system has two 6TB "storage" disks, for a total of 12 disks.
• One 6TB physical disk on each system is of storage class "A" (e.g. Self-encrypting fast spinner)
• One 6TB physical disk on each system is of storage class "B" (e.g. Normal slow-spinner)

Desired storage configuration:

• I want to create two Ceph pools, one comprised of disks from storage class "A" (Ceph pool "A"), and a second comprised of disks from storage class "B" (Ceph pool "B")
• Ceph pool "A" needs to be configured with size 3, while "B" is configured with size 2
• I will then create VM's and allocate storage as needed from each appropriate pool per the application.

I have tried to create this environment several different ways, but each has failed for some reason:

• When I try to create it from the ProxMox GUI, I cannot find a way to add a second crush ruleset. As such, any OSDs I add are put into the single initial first pool
• When I create a second ruleset by hand-writing and installing a new crushmap with "class A" disks in one ruleset, and "class B" in a different ruleset (e.g. ceph osd setcrushmap -i <file>), I can then add the OSDs to the new ruleset.. however, upon reboot, I find the crushmap has been altered such that both class "A" and class "B" disks are now in one pool.. and any VM that has a reference to the second pool now fails (because it can't find its object storage)
• I have read the documentation such as: https://pve.proxmox.com/wiki/Ceph_Server#Ceph_Pools, however, in practice, each time I create an OSD it is automatically assigned to the first pool

I am willing to start over if necessary, but there must be a way to achieve what I have described. Any help is appreciated. Step-by-step instructions would be amazing.

Thanks!

 

[wolfgang]

Hi,

you have to edit the crush map by hand to achieve this.
can you send the decompiled crushmap what you have running?

 

[markusd]

however, upon reboot, I find the crushmap has been altered such that both class "A" and class "B" disks are now in one pool..

Have you add

osd crush update on start = false

in /etc/pve/ceph.conf
?

Markus

 

[jeff Wabik]

Have you add

osd crush update on start = false

in /etc/pve/ceph.conf
?

Markus

Yes.

 

[jeff Wabik]

Hi,

you have to edit the crush map by hand to achieve this.
can you send the decompiled crushmap what you have running?

I have edited the crushmap by hand (it's attached below).

Here's the scenario:

1. I edit the crushmap (either by hand and import it with "set osd crushmap", or by using the right sequence of "ceph osd crush <...>" commands) and get it to what looks correct.
2. I create a new pool (with suitable parameters) pointing to the new crush ruleset (i.e. ruleset 1)
3. I create new storage pointing to the new pool.

With the same wrong/bad results:

1. If the OSDs are down, when I try to mark them up/in from the ProxMox GUI, I am greeted by "Connection error 500: proxy loop detected" error. And the operation fails.
2. If the OSDs are UP or I mark them UP from cli (e.g. "ceph osd in <ids>"), the OSDs get added to the pool described in ruleset 0.
3. If I do not have crush_update_on_start=false, the crushmap is changed to reflect what is described in #2.

I've tried creating new root buckets and new rulesets. Same effect. I'm trying to use ruleset 0 and ruleset 1.

Here's the Crushmap followed by "ceph osd tree". OSD.[0-5] should be in ruleset 0, and OSD[6-11] in ruleset 1. Some of them are in the wrong bucket right now -- don't worry about that.

------------------------------------------------------------------------------------------------------------------------------------------
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable straw_calc_version 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host prox5 {
id -2 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.4 weight 5.460
}
host prox6 {
id -3 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.5 weight 5.460
}
host prox4 {
id -4 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.3 weight 5.460
}
host prox3 {
id -5 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.2 weight 5.460
}
host prox1 {
id -6 # do not change unnecessarily
# weight 10.920
alg straw
hash 0 # rjenkins1
item osd.0 weight 5.460
item osd.6 weight 5.460
}
host prox2 {
id -7 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.1 weight 5.460
}
root default {
id -1 # do not change unnecessarily
# weight 38.220
alg straw
hash 0 # rjenkins1
item prox5 weight 5.460
item prox6 weight 5.460
item prox4 weight 5.460
item prox3 weight 5.460
item prox1 weight 10.920
item prox2 weight 5.460
}
host prox1-cam {
id -9 # do not change unnecessarily
# weight 0.000
alg straw
hash 0 # rjenkins1
}
host prox2-cam {
id -10 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.7 weight 5.460
}
host prox3-cam {
id -11 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.8 weight 5.460
}
host prox4-cam {
id -12 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.9 weight 5.460
}
host prox5-cam {
id -13 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.10 weight 5.460
}
host prox6-cam {
id -14 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.11 weight 5.460
}
root cameras {
id -8 # do not change unnecessarily
# weight 27.300
alg straw
hash 0 # rjenkins1
item prox1-cam weight 0.000
item prox2-cam weight 5.460
item prox3-cam weight 5.460
item prox4-cam weight 5.460
item prox5-cam weight 5.460
item prox6-cam weight 5.460
}
root main {
id -15 # do not change unnecessarily
# weight 0.000
alg straw
hash 0 # rjenkins1
}

# rules
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
rule camrule {
ruleset 1
type replicated
min_size 1
max_size 10
step take cameras
step chooseleaf firstn 0 type host
step emit
}
rule mainrule {
ruleset 2
type replicated
min_size 1
max_size 10
step take main
step chooseleaf firstn 0 type host
step emit
}

# end crush map
-----------------------------------------------------------------------------------------------------------------------
root@prox1:~/crush# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-15 0 root main
-8 27.29988 root cameras
-9 0 host prox1-cam
-10 5.45998 host prox2-cam
7 5.45998 osd.7 up 0 1.00000
-11 5.45998 host prox3-cam
8 5.45998 osd.8 up 0 1.00000
-12 5.45998 host prox4-cam
9 5.45998 osd.9 up 0 1.00000
-13 5.45998 host prox5-cam
10 5.45998 osd.10 up 0 1.00000
-14 5.45998 host prox6-cam
11 5.45998 osd.11 down 0 1.00000
-1 38.21994 root default
-2 5.45999 host prox5
4 5.45999 osd.4 up 1.00000 1.00000
-3 5.45999 host prox6
5 5.45999 osd.5 up 1.00000 1.00000
-4 5.45999 host prox4
3 5.45999 osd.3 up 1.00000 1.00000
-5 5.45999 host prox3
2 5.45999 osd.2 up 1.00000 1.00000
-6 10.91998 host prox1
0 5.45999 osd.0 up 1.00000 1.00000
6 5.45999 osd.6 up 1.00000 1.00000
-7 5.45999 host prox2
1 5.45999 osd.1 down 0 1.00000

 

[markusd]

which ceph-vesion are you using?

 

[jeff Wabik]

which ceph-vesion are you using?

ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)