Cannot create jewel OSDs in 4.4.5

I mean there no symlink to /dev/sdc2 exists in /dev/disk/by-partuuid to /dev/sdc2 until I put a filesystem on /dev/sdc2.
Should I just change the symlink to (literal) "/dev/sdc2" ? Right now the symlink is to a non-existent entry in /dev/disk/by-partuuid.
 
This is strange, you should have a partuuid even if you don't have a FS on a partition.

what's he output of
Code: 
 sgdisk --info=2 /dev/sdc
?

Ok let's try to generate the OSD from scratch the manuel way :

Code: 
// check that the table is GPT and generate 2 partitions, 1= total of disque - 5 Go , 2=5Go
cfdisk /dev/sdc

/ the data partition
//generate a uuid
uuidgen

//give this uuid to the data partition  (1)
sgdisk --change-name="1:ceph data" \
                        --partition-guid=1:<THE_UUID_YOU_GET_JUST_BEFORE> \
                        --typecode=1:45b0969e-9b03-4f30-b4c6-b4b80ceff106 \
                        /dev/sdc

//checking everythink is ok
lsblk  -o /dev/sdc NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID

// the journal partition
//generate a uuid
uuidgen

//give this uuid to the journal partition (2)
sgdisk --change-name="2:ceph journal" \
            --partition-guid=2:<THE_NEW_UUID_YOU_GET_JUST> 
            --typecode=2:4fbd7e29-9d25-41b8-afd0-062c0ceff05d 
            /dev/sdc

//checking everythink is ok
lsblk  -o /dev/sdc NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID

//for now patition should be ok
//let's prepare the osd

ceph-disk prepare --osd-uuid=<THE_UUID_OF_DATA_PART> \
             --fs-type xfs \
             --cluster ceph -- /dev/sdc1 /dev/sdc2

//check the result
ceph-disk list

// and finally activate the osd
ceph-disk activate /dev/sdc1

And let's rock !
 
At the last step (ceph-activate), I get:
Code: 
root@pve1:~# ceph-disk activate /dev/sdc1
got monmap epoch 1
mount_activate: Failed to activate
ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'2', '--monmap', '/var/lib/ceph/tmp/mnt.4Y75ui/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.4Y75ui', '--osd-journal', '/var/lib/ceph/tmp/mnt.4Y75ui/journal', '--osd-uuid', u'30515e35-b5f2-4d55-8998-85820baa04f0', '--keyring', '/var/lib/ceph/tmp/mnt.4Y75ui/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2017-01-04 10:32:57.537037 7f9cc47bd800 -1 filestore(/var/lib/ceph/tmp/mnt.4Y75ui) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.4Y75ui/journal: (13) Permission denied
2017-01-04 10:32:57.537053 7f9cc47bd800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13
2017-01-04 10:32:57.537110 7f9cc47bd800 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.4Y75ui: (13) Permission denied
 
Nothing obviously wrong to me:
Code: 
root@pve1:~# ls -ld /var/lib/ceph/
drwxr-x--- 8 ceph ceph 4096 Dec 30 13:07 /var/lib/ceph/

The error more tends to indicate (to me, anyway) that the mountpoint wasn't chown'd after being mounted there. (All disks under linux AFAIK mount as root... is there a chown somewhere in the ceph-disk activate code?)
 
Matthieu Le Corre said:
Can you symlink the journal to right partition (/dev/sdc2 in your case I think) ?

what the result of :
Code: 
lsblk /dev/sdc -o NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID
Click to expand...
Matthieu, lsblk never reports anything for the second partition. There is no filesystem on the second partition. Partitions to not get UUIDs by themselves; only whole disks and filesystems get UUIDs, so it makes sense that there is no UUID on the second partition.
In any case, I can't realistically support the manual creation of 4 x 5 = 20 OSDs on a production cluster - I'll have to wait until this bug gets fixed by someone else, unfortunately. Sheepdog, here I come :-(.

UPDATE: I figured out how to manually uninstall jewel and install hammer, I'll try that next.
 
Last edited:
that's weird, lsblk sould report the uuid for the partition even if there's no FS on it !

I can assure you that partition get UUID ;)

here is the output from my cluster :
Code: 
root@pve03:~# lsblk  /dev/sdp -o NAME,FSTYPE,MOUNTPOINT,PARTTYPE,PARTLABEL,PARTUUID
NAME   FSTYPE MOUNTPOINT                PARTTYPE                             PARTLABEL    PARTUUID
sdp                                                                                      
├─sdp1 xfs    /var/lib/ceph/osd/ceph-41 4fbd7e29-9d25-41b8-afd0-062c0ceff05d ceph data    33a21cd4-682e-4500-a955-5b276aba60c4
└─sdp2                                  45b0969e-9b03-4f30-b4c6-b4b80ceff106 ceph journal 7f9c9c56-ebb2-4914-aed6-2c6279f5a393
root@pve03:~# sgdisk --info=1 /dev/sdp
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 33A21CD4-682E-4500-A955-5B276ABA60C4
First sector: 2048 (at 1024.0 KiB)
Last sector: 1939867647 (at 925.0 GiB)
Partition size: 1939865600 sectors (925.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'
root@pve03:~# 
root@pve03:~# 
root@pve03:~# sgdisk --info=2 /dev/sdp
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 7F9C9C56-EBB2-4914-AED6-2C6279F5A393
First sector: 1939867648 (at 925.0 GiB)
Last sector: 1953525134 (at 931.5 GiB)
Partition size: 13657487 sectors (6.5 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'
root@pve03:~#

I think there is a permission issue on you system, maybe partition already existe on sdc.
In that case, ceph-disk won't chang the owner.
 
You must log in or register to reply here.
Top Bottom