Creating/Using Multiple Ceph Pools

J

jeff Wabik

New Member
Nov 7, 2016
4
0
1
60
Hardware configuration:
  • 6 systems in a ProxMox cluster. Works fine.
  • Each system has two 6TB "storage" disks, for a total of 12 disks.
  • One 6TB physical disk on each system is of storage class "A" (e.g. Self-encrypting fast spinner)
  • One 6TB physical disk on each system is of storage class "B" (e.g. Normal slow-spinner)

Desired storage configuration:

  • I want to create two Ceph pools, one comprised of disks from storage class "A" (Ceph pool "A"), and a second comprised of disks from storage class "B" (Ceph pool "B")
  • Ceph pool "A" needs to be configured with size 3, while "B" is configured with size 2
  • I will then create VM's and allocate storage as needed from each appropriate pool per the application.
I have tried to create this environment several different ways, but each has failed for some reason:

  • When I try to create it from the ProxMox GUI, I cannot find a way to add a second crush ruleset. As such, any OSDs I add are put into the single initial first pool
  • When I create a second ruleset by hand-writing and installing a new crushmap with "class A" disks in one ruleset, and "class B" in a different ruleset (e.g. ceph osd setcrushmap -i <file>), I can then add the OSDs to the new ruleset.. however, upon reboot, I find the crushmap has been altered such that both class "A" and class "B" disks are now in one pool.. and any VM that has a reference to the second pool now fails (because it can't find its object storage)
  • I have read the documentation such as: https://pve.proxmox.com/wiki/Ceph_Server#Ceph_Pools, however, in practice, each time I create an OSD it is automatically assigned to the first pool
I am willing to start over if necessary, but there must be a way to achieve what I have described. Any help is appreciated. Step-by-step instructions would be amazing. :)

Thanks!
 
Hi,

you have to edit the crush map by hand to achieve this.
can you send the decompiled crushmap what you have running?
 
wolfgang said:
Hi,

you have to edit the crush map by hand to achieve this.
can you send the decompiled crushmap what you have running?
Click to expand...

I have edited the crushmap by hand (it's attached below).

Here's the scenario:
  1. I edit the crushmap (either by hand and import it with "set osd crushmap", or by using the right sequence of "ceph osd crush <...>" commands) and get it to what looks correct.
  2. I create a new pool (with suitable parameters) pointing to the new crush ruleset (i.e. ruleset 1)
  3. I create new storage pointing to the new pool.
With the same wrong/bad results:
  1. If the OSDs are down, when I try to mark them up/in from the ProxMox GUI, I am greeted by "Connection error 500: proxy loop detected" error. And the operation fails.
  2. If the OSDs are UP or I mark them UP from cli (e.g. "ceph osd in <ids>"), the OSDs get added to the pool described in ruleset 0.
  3. If I do not have crush_update_on_start=false, the crushmap is changed to reflect what is described in #2.

I've tried creating new root buckets and new rulesets. Same effect. I'm trying to use ruleset 0 and ruleset 1.

Here's the Crushmap followed by "ceph osd tree". OSD.[0-5] should be in ruleset 0, and OSD[6-11] in ruleset 1. Some of them are in the wrong bucket right now -- don't worry about that.

------------------------------------------------------------------------------------------------------------------------------------------
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable straw_calc_version 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host prox5 {
id -2 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.4 weight 5.460
}
host prox6 {
id -3 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.5 weight 5.460
}
host prox4 {
id -4 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.3 weight 5.460
}
host prox3 {
id -5 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.2 weight 5.460
}
host prox1 {
id -6 # do not change unnecessarily
# weight 10.920
alg straw
hash 0 # rjenkins1
item osd.0 weight 5.460
item osd.6 weight 5.460
}
host prox2 {
id -7 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.1 weight 5.460
}
root default {
id -1 # do not change unnecessarily
# weight 38.220
alg straw
hash 0 # rjenkins1
item prox5 weight 5.460
item prox6 weight 5.460
item prox4 weight 5.460
item prox3 weight 5.460
item prox1 weight 10.920
item prox2 weight 5.460
}
host prox1-cam {
id -9 # do not change unnecessarily
# weight 0.000
alg straw
hash 0 # rjenkins1
}
host prox2-cam {
id -10 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.7 weight 5.460
}
host prox3-cam {
id -11 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.8 weight 5.460
}
host prox4-cam {
id -12 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.9 weight 5.460
}
host prox5-cam {
id -13 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.10 weight 5.460
}
host prox6-cam {
id -14 # do not change unnecessarily
# weight 5.460
alg straw
hash 0 # rjenkins1
item osd.11 weight 5.460
}
root cameras {
id -8 # do not change unnecessarily
# weight 27.300
alg straw
hash 0 # rjenkins1
item prox1-cam weight 0.000
item prox2-cam weight 5.460
item prox3-cam weight 5.460
item prox4-cam weight 5.460
item prox5-cam weight 5.460
item prox6-cam weight 5.460
}
root main {
id -15 # do not change unnecessarily
# weight 0.000
alg straw
hash 0 # rjenkins1
}

# rules
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
rule camrule {
ruleset 1
type replicated
min_size 1
max_size 10
step take cameras
step chooseleaf firstn 0 type host
step emit
}
rule mainrule {
ruleset 2
type replicated
min_size 1
max_size 10
step take main
step chooseleaf firstn 0 type host
step emit
}

# end crush map
-----------------------------------------------------------------------------------------------------------------------
root@prox1:~/crush# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-15 0 root main
-8 27.29988 root cameras
-9 0 host prox1-cam
-10 5.45998 host prox2-cam
7 5.45998 osd.7 up 0 1.00000
-11 5.45998 host prox3-cam
8 5.45998 osd.8 up 0 1.00000
-12 5.45998 host prox4-cam
9 5.45998 osd.9 up 0 1.00000
-13 5.45998 host prox5-cam
10 5.45998 osd.10 up 0 1.00000
-14 5.45998 host prox6-cam
11 5.45998 osd.11 down 0 1.00000
-1 38.21994 root default
-2 5.45999 host prox5
4 5.45999 osd.4 up 1.00000 1.00000
-3 5.45999 host prox6
5 5.45999 osd.5 up 1.00000 1.00000
-4 5.45999 host prox4
3 5.45999 osd.3 up 1.00000 1.00000
-5 5.45999 host prox3
2 5.45999 osd.2 up 1.00000 1.00000
-6 10.91998 host prox1
0 5.45999 osd.0 up 1.00000 1.00000
6 5.45999 osd.6 up 1.00000 1.00000
-7 5.45999 host prox2
1 5.45999 osd.1 down 0 1.00000
 
You must log in or register to reply here.
Top Bottom