Hello, I've been testing ceph in my dev cluster for a bit now and I've finally added a few SSDs. I edited the CRUSH map to force the default replicated rule to use HDD. I created a new SSD rule for type SSD and created a new pool. If I write any data to the pool I instantly get a health warning of objects misplaced. Any thoughts? Code: ceph -s cluster: id: e791fb44-9c38-4e86-8edf-cdbc5a3f7d63 health: HEALTH_WARN 5/567160 objects misplaced (0.001%) services: mon: 3 daemons, quorum FEDERATION,DS9,isolinear mgr: isolinear(active), standbys: FEDERATION, DS9 osd: 10 osds: 10 up, 10 in; 64 remapped pgs data: pools: 2 pools, 314 pgs objects: 184k objects, 738 GB usage: 2224 GB used, 22920 GB / 25145 GB avail pgs: 5/567160 objects misplaced (0.001%) 250 active+clean 64 active+clean+remapped io: client: 25361 kB/s rd, 71561 B/s wr, 7 op/s rd, 8 op/s wr Edit: there is only one host with storage. It is configured for single node. The other monitors were added to use storage from isolinear (star trek nerd here). Code: # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class ssd device 9 osd.9 class ssd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 region type 10 root # buckets host isolinear { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily id -5 class ssd # do not change unnecessarily # weight 24.556 alg straw2 hash 0 # rjenkins1 item osd.0 weight 1.819 item osd.1 weight 1.819 item osd.2 weight 2.728 item osd.3 weight 2.728 item osd.4 weight 3.638 item osd.5 weight 3.638 item osd.6 weight 3.638 item osd.7 weight 3.638 item osd.8 weight 0.455 item osd.9 weight 0.455 } root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily id -6 class ssd # do not change unnecessarily # weight 24.556 alg straw2 hash 0 # rjenkins1 item isolinear weight 24.556 } # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default class hdd step choose firstn 0 type osd step emit } rule FastStoreage { id 1 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 0 type host step emit } # end crush map Server View Logs
Never mind I found my mistake: "step chooseleaf firstn 0 type host" should be " step choose firstn 0 type osd" Sometimes you just need to see your mistake in a forum =
Naturally! I'm currently testing. I was running a pure ZFS three way mirror but I finally got a decent system with enough removable drive caddies to give this a test run. This might sound strange but I love how the hashing makes all the drive activity lights blink. Anyways, I backup all data to external storage so data loss potential has been mitigated a bit.