Proxmox 6.0 CEPH OSD's don't show down

[zacko11288]

Hi Everyone,

We have been running proxmox 5.4 with CEPH for awhile now without any issues. We recently built a brand new 5 node cluster using Proxmox 6.0. We had no issues getting our OSD's up and running, However whenever we physically pull a drive the associated OSD does not show down. Has anyone else noticed this behavior? We noticed that the new version of CEPH seems to reference devices differently. Any help would be greatly appreciated.

 

[Alwin]

It will take up to 10 min till Ceph will mark an OSD down & out.

 

[zacko11288]

Thanks for the response, we have our OSD's down/Out set to 30 seconds, I pulled 2 drives last night and this morning when I came back into the office the OSD's still show up and in. We've never seen this behavior in our 5.4 environments.

 

[Alwin]

Are the OSDs created on a single VG with a singel PV? Or did you customize the OSD creation?

 

[Jarek]

Is there any slow request in the ceph status?
If not, and osd is still up, there wasn't any I/O operation on that drive, so ceph doesn't know if disk is still here.

It will take up to 10 min till Ceph will mark an OSD down & out.

Ceph mark an OSD down immediately after I/O error, and mark out 10 min later.