Cannot create jewel OSDs in 4.4.5

athompso · Jan 3, 2017

Not unless I put a filesystem on /dev/sdc2. What type should it be?

Matthieu Le Corre · Jan 4, 2017

you don't need to peed any filesytem on /dev/sdc2 ( ceph will do it for you)
but the partition type must be "45b0969e-9b03-4f30-b4c6-b4b80ceff106"

-> https://github.com/ceph/ceph/blob/jewel/src/ceph-disk/ceph_disk/main.py#L52

Code:

sgdisk --change-name="2:ceph journal" \
       --typecode="2:45b0969e-9b03-4f30-b4c6-b4b80ceff106" \
       /dev/sdc

athompso · Jan 4, 2017

I mean there no symlink to /dev/sdc2 exists in /dev/disk/by-partuuid to /dev/sdc2 until I put a filesystem on /dev/sdc2.
Should I just change the symlink to (literal) "/dev/sdc2" ? Right now the symlink is to a non-existent entry in /dev/disk/by-partuuid.

Matthieu Le Corre · Jan 4, 2017

This is strange, you should have a partuuid even if you don't have a FS on a partition.

what's he output of

Code:

 sgdisk --info=2 /dev/sdc

?

Ok let's try to generate the OSD from scratch the manuel way :

Code:

// check that the table is GPT and generate 2 partitions, 1= total of disque - 5 Go , 2=5Go
cfdisk /dev/sdc

/ the data partition
//generate a uuid
uuidgen

//give this uuid to the data partition  (1)
sgdisk --change-name="1:ceph data" \
                        --partition-guid=1:<THE_UUID_YOU_GET_JUST_BEFORE> \
                        --typecode=1:45b0969e-9b03-4f30-b4c6-b4b80ceff106 \
                        /dev/sdc

//checking everythink is ok
lsblk  -o /dev/sdc NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID

// the journal partition
//generate a uuid
uuidgen

//give this uuid to the journal partition (2)
sgdisk --change-name="2:ceph journal" \
            --partition-guid=2:<THE_NEW_UUID_YOU_GET_JUST> 
            --typecode=2:4fbd7e29-9d25-41b8-afd0-062c0ceff05d 
            /dev/sdc

//checking everythink is ok
lsblk  -o /dev/sdc NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID

//for now patition should be ok
//let's prepare the osd

ceph-disk prepare --osd-uuid=<THE_UUID_OF_DATA_PART> \
             --fs-type xfs \
             --cluster ceph -- /dev/sdc1 /dev/sdc2

//check the result
ceph-disk list

// and finally activate the osd
ceph-disk activate /dev/sdc1

And let's rock !

athompso · Jan 4, 2017

At the last step (ceph-activate), I get:

Code:

root@pve1:~# ceph-disk activate /dev/sdc1
got monmap epoch 1
mount_activate: Failed to activate
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'2', '--monmap', '/var/lib/ceph/tmp/mnt.4Y75ui/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.4Y75ui', '--osd-journal', '/var/lib/ceph/tmp/mnt.4Y75ui/journal', '--osd-uuid', u'30515e35-b5f2-4d55-8998-85820baa04f0', '--keyring', '/var/lib/ceph/tmp/mnt.4Y75ui/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2017-01-04 10:32:57.537037 7f9cc47bd800 -1 filestore(/var/lib/ceph/tmp/mnt.4Y75ui) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.4Y75ui/journal: (13) Permission denied
2017-01-04 10:32:57.537053 7f9cc47bd800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13
2017-01-04 10:32:57.537110 7f9cc47bd800 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.4Y75ui: (13) Permission denied

Matthieu Le Corre · Jan 4, 2017

Check the directory /var/lib/ceph it-self , it seems that it can't create the tmp dir !

athompso · Jan 5, 2017

Nothing obviously wrong to me:

Code:

root@pve1:~# ls -ld /var/lib/ceph/
drwxr-x--- 8 ceph ceph 4096 Dec 30 13:07 /var/lib/ceph/

The error more tends to indicate (to me, anyway) that the mountpoint wasn't chown'd after being mounted there. (All disks under linux AFAIK mount as root... is there a chown somewhere in the ceph-disk activate code?)

Matthieu Le Corre · Jan 5, 2017

can you check the permittion of the /dev/sdc2 it should be ceph/ceph.

"Ceph-disk " is taking care of that
-> https://github.com/ceph/ceph/blob/jewel/src/ceph-disk/ceph_disk/main.py#L1224

athompso · Jan 5, 2017

Nope. /dev/sdc1 is owned by ceph/ceph, but /dev/sdc2 is owned by root/root.

udo · Jan 5, 2017

athompso said:
Nope. /dev/sdc1 is owned by ceph/ceph, but /dev/sdc2 is owned by root/root.

Hi,
the journal (/dev/sdc2) must owned by ceph too. See here:
http://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel (Set permission)

Udo

athompso · Jan 6, 2017

udo said:
See here:
http://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel (Set permission)

Granted, but recall this was a fresh install of Jewel, Hammer was never installed here.
I'll try following some of that anyway (once the site is back up...) and see what happens.

athompso · Jan 7, 2017

Matthieu Le Corre said:
Can you symlink the journal to right partition (/dev/sdc2 in your case I think) ?

what the result of :

Code:

lsblk /dev/sdc -o NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID

Matthieu, lsblk never reports anything for the second partition. There is no filesystem on the second partition. Partitions to not get UUIDs by themselves; only whole disks and filesystems get UUIDs, so it makes sense that there is no UUID on the second partition.
In any case, I can't realistically support the manual creation of 4 x 5 = 20 OSDs on a production cluster - I'll have to wait until this bug gets fixed by someone else, unfortunately. Sheepdog, here I come :-(.

UPDATE: I figured out how to manually uninstall jewel and install hammer, I'll try that next.

Matthieu Le Corre · Jan 9, 2017

that's weird, lsblk sould report the uuid for the partition even if there's no FS on it !

I can assure you that partition get UUID

here is the output from my cluster :

Code:

root@pve03:~# lsblk  /dev/sdp -o NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID
NAME   FSTYPE MOUNTPOINT                PARTTYPE                             PARTLABEL    PARTUUID
sdp                                                                                      
├─sdp1 xfs    /var/lib/ceph/osd/ceph-41 4fbd7e29-9d25-41b8-afd0-062c0ceff05d ceph data    33a21cd4-682e-4500-a955-5b276aba60c4
└─sdp2                                  45b0969e-9b03-4f30-b4c6-b4b80ceff106 ceph journal 7f9c9c56-ebb2-4914-aed6-2c6279f5a393
root@pve03:~# sgdisk --info=1 /dev/sdp
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 33A21CD4-682E-4500-A955-5B276ABA60C4
First sector: 2048 (at 1024.0 KiB)
Last sector: 1939867647 (at 925.0 GiB)
Partition size: 1939865600 sectors (925.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'
root@pve03:~# 
root@pve03:~# 
root@pve03:~# sgdisk --info=2 /dev/sdp
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 7F9C9C56-EBB2-4914-AED6-2C6279F5A393
First sector: 1939867648 (at 925.0 GiB)
Last sector: 1953525134 (at 931.5 GiB)
Partition size: 13657487 sectors (6.5 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'
root@pve03:~#

I think there is a permission issue on you system, maybe partition already existe on sdc.
In that case, ceph-disk won't chang the owner.

Search

Search

Cannot create jewel OSDs in 4.4.5

athompso

Member

Matthieu Le Corre

Renowned Member

athompso

Member

Matthieu Le Corre

Renowned Member

athompso

Member

Matthieu Le Corre

Renowned Member

athompso

Member

Matthieu Le Corre

Renowned Member

athompso

Member

udo

Distinguished Member

athompso

Member

athompso

Member

Matthieu Le Corre

Renowned Member