[SOLVED] CEPH OSD delay in service startup after reboot.

[itNGO]

We are investigating an issue, where after a NODE-Reboot in a 3-Node-Cluster the OSDs are delayed in startup.
We need to set "NOOUT"-Flag to prevent CEPH from mixing data around, cause the OSD just take several minutes to get started.
They always start on their own and work normal after that.

Is this delay normal? We are on PVE 7.2-4 with CEPH 16.2.9. It takes 10 Minutes until last OSD is available again from 4 OSDs per Node.

Anyone had this also?

 

[mira]

Is this only after a reboot, or also when you restart an OSD?
Could you provide both the journal from the boot until all OSDs are up and running and the Ceph log (/var/log/ceph/ceph.log) and OSD logs for that timeframe?

 

[itNGO]

We updated all Nodes yesterday to 7.2-7 with latest Kernel from "Enterprise Repository".
The reboot went without issues and all OSDs came up on their own after a few minutes.

So I guess this is no longer an issue.

 

[spirit]

I have see that on 1 cluster, where osd omap size was very big (700-800MB)

#ceph osd df

-> check "OMAP" column

osd need to replay them.

This was on pacific, upgraded from octopus, and after omap format convertion,
https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific#Data_corruption_on_OMAP_conversion
"ceph config set osd bluestore_fsck_quick_fix_on_mount true"

The boot was fast again.