OSD wont start after Ceph upgrade from Hammer to Jewel

D

Dan Nicolae

Renowned Member
Apr 27, 2016
78
6
73
45
I just updated one of our Ceph nodes using this tutorial, from Hammer to Jewel version. Unfortunately after upgrade OSD's wont start. We use Proxmox 4.4.5. OSD's have the journal mounted on SSD. Error is,

root@ceph03:~# systemctl status ceph-osd@2.service
● ceph-osd@2.service - Ceph object storage daemon
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
Drop-In: /lib/systemd/system/ceph-osd@.service.d
└─ceph-after-pve-cluster.conf
Active: failed (Result: start-limit) since Wed 2017-01-18 00:30:11 EET; 16s ago
Process: 4471 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Process: 4423 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
Main PID: 4471 (code=exited, status=1/FAILURE)

Jan 18 00:30:11 ceph03 systemd[1]: Unit ceph-osd@2.service entered failed state.
Jan 18 00:30:11 ceph03 systemd[1]: ceph-osd@2.service holdoff time over, scheduling restart.
Jan 18 00:30:11 ceph03 systemd[1]: Stopping Ceph object storage daemon...
Jan 18 00:30:11 ceph03 systemd[1]: Starting Ceph object storage daemon...
Jan 18 00:30:11 ceph03 systemd[1]: ceph-osd@2.service start request repeated too quickly, refusing to start.
Jan 18 00:30:11 ceph03 systemd[1]: Failed to start Ceph object storage daemon.
Jan 18 00:30:11 ceph03 systemd[1]: Unit ceph-osd@2.service entered failed state.

Any help is appreciated. Thanks.
 
Last edited:
Hi,

can you send the output of this commands?

Code: 
ls -hal /var/lib/ceph/osd/ceph-2/
ls -hal /dev/<journaldev>
lsblk
 
Sure,

root@ceph03:~# ls -hal /var/lib/ceph/osd/ceph-2/
total 60K
drwxr-xr-x 3 ceph ceph 217 Jan 18 00:54 .
drwxr-xr-x 4 ceph ceph 4.0K Jan 27 2016 ..
-rw-r--r-- 1 ceph ceph 892 Nov 26 16:30 activate.monmap
-rw-r--r-- 1 ceph ceph 3 Nov 26 16:30 active
-rw-r--r-- 1 ceph ceph 37 Nov 26 16:30 ceph_fsid
drwxr-xr-x 308 ceph ceph 8.0K Jan 17 22:40 current
-rw-r--r-- 1 ceph ceph 37 Nov 26 16:30 fsid
lrwxrwxrwx 1 ceph ceph 58 Nov 26 16:30 journal -> /dev/disk/by-partuuid/48b6bce4-d41a-402b-8397-b479bf8a63a3
-rw-r--r-- 1 ceph ceph 37 Nov 26 16:30 journal_uuid
-rw------- 1 ceph ceph 56 Nov 26 16:30 keyring
-rw-r--r-- 1 ceph ceph 21 Nov 26 16:30 magic
-rw-r--r-- 1 ceph ceph 6 Nov 26 16:30 ready
-rw-r--r-- 1 ceph ceph 4 Nov 26 16:30 store_version
-rw-r--r-- 1 ceph ceph 53 Nov 26 16:30 superblock
-rw-r--r-- 1 ceph ceph 0 Jan 18 00:54 systemd
-rw-r--r-- 1 ceph ceph 2 Nov 26 16:30 whoami




first osd journal

root@ceph03:~# ls -hal /dev/sdd1
brw-rw---- 1 root disk 8, 49 Jan 18 00:46 /dev/sdd1

and the second osd journal,
root@ceph03:~# ls -hal /dev/sdd2
brw-rw---- 1 root disk 8, 50 Jan 18 00:46 /dev/sdd2
root@ceph03:~#
 
Ok, I run,

chown ceph: -R /dev/disk/by-partlabel/journal-2
chown ceph: -R /dev/disk/by-partlabel/journal-3
(location of the osd journals)

Everything was ok. OSD's where up. I rebooted the server and permissions are back,

root@ceph03:~# ls -hal /dev/disk/by-partlabel/journal-2
lrwxrwxrwx 1 root root 10 Jan 18 10:13 /dev/disk/by-partlabel/journal-2 -> ../../sdd1
root@ceph03:~# ls -hal /dev/disk/by-partlabel/journal-3
lrwxrwxrwx 1 root root 10 Jan 18 10:13 /dev/disk/by-partlabel/journal-3 -> ../../sdd2
 
I changed back the permissions to ceph, wait for a while, start the osd's and they are OK. I'm sure that if I reboot the server permissions are switched back to root.

root@ceph03:~# systemctl status ceph-osd@3.service
● ceph-osd@3.service - Ceph object storage daemon
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
Drop-In: /lib/systemd/system/ceph-osd@.service.d
└─ceph-after-pve-cluster.conf
Active: active (running) since Wed 2017-01-18 11:00:27 EET; 9min ago
Process: 4252 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
Main PID: 4300 (ceph-osd)
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@3.service
└─4300 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph

Jan 18 11:00:27 ceph03 systemd[1]: Starting Ceph object storage daemon...
Jan 18 11:00:27 ceph03 ceph-osd-prestart.sh[4252]: create-or-move updated item name 'osd.3' weight 1.8136 at location {host=ceph03,root=default} to crush map
Jan 18 11:00:27 ceph03 systemd[1]: Started Ceph object storage daemon.
Jan 18 11:00:27 ceph03 ceph-osd[4300]: starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /dev/disk/by-partlabel/journal-3
Jan 18 11:00:40 ceph03 ceph-osd[4300]: 2017-01-18 11:00:40.222835 7f8e5f9a6800 -1 osd.3 33355 log_to_monitors {default=true}
 
If you set the permission on /dev/disk/by-partlabel/journal-2
this path will be create every boot form udev.
This is the reason why it is back to root

The problem is you do not what the HowTo tell you to do.
readlink -f follows the link to the end to the block dev.
And you set the permission on the block dev and not on a symlink.
 
root@ceph03:~# readlink -f /var/lib/ceph/osd/ceph-2/journal
/dev/sda2

Where /dev/sda is the HDD with one of the OSD.

Journal is declared in ceph.conf as,

[osd.2]
osd journal = /dev/disk/by-partlabel/journal-2 osd
ournal size = 10240

And is on a SSD drive (/dev/sdd1 for one OSD and /dev/sdd2 for the second OSD).

If I set permissions like in the how to, it wil not point on the real path of the journal.

So, I guess I have to change the path to journal like in the ceph.conf.

root@ceph03:~# readlink -f /dev/disk/by-partlabel/journal-2
/dev/sdd1
root@ceph03:~# readlink -f /dev/disk/by-partlabel/journal-3
/dev/sdd2

and then set permissions to ceph
chown ceph: -R /dev/sdd1
chown ceph: -R /dev/sdd2

and it will be ok?
 
After reboot, permissions are back to root

root@ceph03:~# ls -hal /dev/sdd1
brw-rw---- 1 root disk 8, 49 Jan 18 11:40 /dev/sdd1
root@ceph03:~# ls -hal /dev/sdd2
brw-rw---- 1 root disk 8, 50 Jan 18 11:40 /dev/sdd2

and

root@ceph03:~# ls -hal /dev/disk/by-partlabel/journal-2
lrwxrwxrwx 1 root root 10 Jan 18 11:40 /dev/disk/by-partlabel/journal-2 -> ../../sdd1
root@ceph03:~# ls -hal /dev/disk/by-partlabel/journal-3
lrwxrwxrwx 1 root root 10 Jan 18 11:40 /dev/disk/by-partlabel/journal-3 -> ../../sdd2

I wonder if Ceph can't reach the journal path in ceph.conf because the symlink is also under root permission.

Aparently only after I changed the permission from root to ceph on /dev/disk/by-partlabel/journal... I was able to start the OSDs.

root@ceph03:~# chown ceph: -R /dev/disk/by-partlabel/journal-3
root@ceph03:~# chown ceph: -R /dev/disk/by-partlabel/journal-2
 
Last edited:
The problem is that the udev do not set the permission correct normal this is be done by /lib/udev/rules.d/95-ceph-osd.rules

I think your journal has not the correct attributes so udev rule does not match on your journal dev.
 
I found a workaround for the journal/udev problem.

First,

rm -f /var/lib/ceph/osd/<osd-id>/journal
ln -s /var/lib/ceph/osd/<osd-id>/journal /dev/<ssd-partition-for-your-journal>

then,
sgdisk --typecode=1:45b0969e-9b03-4f30-b4c6-b4b80ceff106 /dev/<ssd-drive-for-your-journal>

(where 1 here ...typecode=1:45b... is the number of the partition used as journal for that osd)

Source: here and here.
 
Last edited:
You must log in or register to reply here.
Top Bottom