Journals missing or in the wrong place?

[jkirker]

Good morning all,

On each of my Ceph nodes I have 2 SSD's for journals. /dev/sdj & /dev/sdk

While upgrading from Hammer -> Jewel I noticed something that I think is odd, but I'm not sure. It appears that some of my OSD's either may not have journals, or the journal is not set to one of the SSD's they should be.

Is the below raeadlink list normal? (And if it is, is there a "simple" way to reassign an OSD so that the journal is moved to one of the SSD's?)

Both times when upgrading from both Hammer -> Jewel and Jewel -> Luminous, I went from 100% healthy to having a status like this:

root@moxstore1:~# ceph status
cluster:
id: 53d58faa-b12b-4fdb-a131-21370562f573
health: HEALTH_WARN
3047409/6951392 objects misplaced (43.839%)
Degraded data redundancy: 50/6951392 objects degraded (0.001%), 2 pgs degraded
application not enabled on 1 pool(s)
services:
mon: 3 daemons, quorum 4,3,2
mgr: mox1(active), standbys: mox3, mox2
osd: 18 osds: 18 up, 18 in; 452 remapped pgs
data:
pools: 1 pools, 512 pgs
objects: 3394k objects, 13539 GB
usage: 27160 GB used, 23115 GB / 50276 GB avail
pgs: 50/6951392 objects degraded (0.001%)
3047409/6951392 objects misplaced (43.839%)
443 active+remapped+backfill_wait
60 active+clean
7 active+remapped+backfilling
2 active+recovery_wait+degraded+remapped
io:
client: 290 kB/s rd, 195 kB/s wr, 61 op/s rd, 10 op/s wr
recovery: 169 MB/s, 42 objects/s

It's re-balancing and rebuilding just fine - but something seems odd. Again, the readlink's look like.

Example Ceph Node
readlink -f /var/lib/ceph/osd/ceph-0/journal
/dev/sdj1

readlink -f /var/lib/ceph/osd/ceph-2/journal
/dev/sdj2

readlink -f /var/lib/ceph/osd/ceph-4/journal
/dev/sdj3

readlink -f /var/lib/ceph/osd/ceph-5/journal
/var/lib/ceph/osd/ceph-5/journal

readlink -f /var/lib/ceph/osd/ceph-6/journal
/dev/sdk1

readlink -f /var/lib/ceph/osd/ceph-7/journal
/var/lib/ceph/osd/ceph-7/journal

readlink -f /var/lib/ceph/osd/ceph-8/journal
/dev/sdk2

readlink -f /var/lib/ceph/osd/ceph-9/journal
/var/lib/ceph/osd/ceph-9/journal

readlink -f /var/lib/ceph/osd/ceph-10/journal
/var/lib/ceph/osd/ceph-10/journal

readlink -f /var/lib/ceph/osd/ceph-11/journal
/var/lib/ceph/osd/ceph-11/journal

readlink -f /var/lib/ceph/osd/ceph-12/journal
/var/lib/ceph/osd/ceph-12/journal

readlink -f /var/lib/ceph/osd/ceph-13/journal
/dev/sdk3

readlink -f /var/lib/ceph/osd/ceph-14/journal
/dev/sdk4

readlink -f /var/lib/ceph/osd/ceph-15/journal
/dev/sdk5

readlink -f /var/lib/ceph/osd/ceph-17/journal
/dev/sdj4

 

[jkirker]

Can anyone tell me if the above is normal? It appears if there are journals, they are assigned to the OSD and not the SSD. My concern is that the last two upgrades required significant rebuilding/re-balancing even though I did everything per doc which expressly says re-balancing shouldn't be necessary.