Strange ceph error

[Norman Uittenbogaart]

Ceph Jewel was acting strange so I rebooted my servers and then Ceph didn't want to come up.
The OSD's where down (except for one)
And starting the OSD with systemctl didn't work, it seems to hang on authentication.
The error was,

** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-3: (2) No such file or directory

Now I checked the paths with ceph-disk list and mounted the drives directly and started the OSD's manually with ceph-osd -i <id> --osd-journal <journal disk> and everything is working fine.
But restarting still won't start the OSD's, what could this be?
It used to work without a issue.

 

[wolfgang]

Hi,

have you an extra journal disk?

I would check the disk meta information if they are correct.

 

[Norman Uittenbogaart]

Hi sorry for the late reply!
I have one SSD on which I keep the journals for 3 sata disks. (so yes a external journal disk)

How would I check the disk meta info?

In the meantime I got the system up, by manually mounting the OSD's to the correct folders and starting the OSD's by hand by doing
ceph-osd -i 6 --osd-journal /dev/sda4 for all OSD's.

Then everything works and communicates.
But then still ceph can't restart the osd's by itself, so I have to do that everytime.

 

[udo]

Hi sorry for the late reply!
I have one SSD on which I keep the journals for 3 sata disks. (so yes a external journal disk)

How would I check the disk meta info?

In the meantime I got the system up, by manually mounting the OSD's to the correct folders and starting the OSD's by hand by doing
ceph-osd -i 6 --osd-journal /dev/sda4 for all OSD's.

Then everything works and communicates.
But then still ceph can't restart the osd's by itself, so I have to do that everytime.

Hi,
how do you have created your jornal and the ceph-disk?

Are the GUID right?

See as example two ceph-disks (with internal journal):

oot@pve04:~# sgdisk --info=1 /dev/sdc
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 9F7F7BB7-AA85-47D3-B389-C9AED6386744
First sector: 20482048 (at 9.8 GiB)
Last sector: 584843230 (at 278.9 GiB)
Partition size: 564361183 sectors (269.1 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'
root@pve04:~# sgdisk --info=2 /dev/sdc
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 71CA2B73-CB79-493B-8834-3EB267FAF4C0
First sector: 2048 (at 1024.0 KiB)
Last sector: 20482047 (at 9.8 GiB)
Partition size: 20480000 sectors (9.8 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'

root@pve06:~# sgdisk --info=1 /dev/sdc
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 4692C2FD-F121-4000-BCA4-D368952E558B
First sector: 20482048 (at 9.8 GiB)
Last sector: 975699934 (at 465.2 GiB)
Partition size: 955217887 sectors (455.5 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'
root@pve06:~# sgdisk --info=2 /dev/sdc
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 0997C62E-B797-4BF4-8DCC-072311EA324E
First sector: 2048 (at 1024.0 KiB)
Last sector: 20482047 (at 9.8 GiB)
Partition size: 20480000 sectors (9.8 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'

is the journal path correct in ceph.conf?

Udo

 

[Norman Uittenbogaart]

Hi Udo,

there is no path in ceph.conf, but this is what those commands give me,

root@nod1:~# sgdisk --info=1 /dev/sdc
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 74654208-0338-440A-99E7-665B2596FBB1
First sector: 2048 (at 1024.0 KiB)
Last sector: 1952448478 (at 931.0 GiB)
Partition size: 1952446431 sectors (931.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'

but the paths I think where never in ceph.conf even wen it worked.
I think it is rights issue...
Not completely sure but I think it happened not long after upgrading to Jewel. (I followed https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel)

 

[udo]

Hi Udo,

there is no path in ceph.conf, but this is what those commands give me,

root@nod1:~# sgdisk --info=1 /dev/sdc
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 74654208-0338-440A-99E7-665B2596FBB1
First sector: 2048 (at 1024.0 KiB)
Last sector: 1952448478 (at 931.0 GiB)
Partition size: 1952446431 sectors (931.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'

but the paths I think where never in ceph.conf even wen it worked.
I think it is rights issue...
Not completely sure but I think it happened not long after upgrading to Jewel. (I followed https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel)

Hi,
and how look the journal?
sgdisk --info=2 /dev/sdc

Udo

 

[Nikolay]

You should check udev rules, try to run afeter reboot:

udevadm trigger --action=add

and look at your journals.

 

[Norman Uittenbogaart]

Hi,
and how look the journal?
sgdisk --info=2 /dev/sdc

Udo

root@nod1:~# sgdisk --info=2 /dev/sda
Partition GUID code: C12A7328-F81F-11D2-BA4B-00A0C93EC93B (EFI System)
Partition unique GUID: 71E851B2-EA93-4352-97C6-80EB79A9D4BB
First sector: 4096 (at 2.0 MiB)
Last sector: 266239 (at 130.0 MiB)
Partition size: 262144 sectors (128.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'EFI-System-Partition'

 

[udo]

root@nod1:~# sgdisk --info=2 /dev/sda
Partition GUID code: C12A7328-F81F-11D2-BA4B-00A0C93EC93B (EFI System)
Partition unique GUID: 71E851B2-EA93-4352-97C6-80EB79A9D4BB
First sector: 4096 (at 2.0 MiB)
Last sector: 266239 (at 130.0 MiB)
Partition size: 262144 sectors (128.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'EFI-System-Partition'

Hi,
sda is your boot-disk. Your journal should be on the ceph-disk, or?

Udo

 

[Norman Uittenbogaart]

Hi,
sda is your boot-disk. Your journal should be on the ceph-disk, or?

Udo

I made partitions on my bootdisk (SSD) for journal info for the ceph disks.