Proxmox 6.0/CEPH OSD disk hot-plug detection

[maff1989]

I'm running a Dell R710 with disk hot-plug capability, running the latest Proxmox 6.0 and Ceph Nautilus. All of the disks are configured as OSDs via the latest GUI tools.

The problem I'm seeing, however, is that when unplugging a drive from the hot-plug bay, the OSD is not detected as "down". It isn't until I manually `down` a different disk that the original hot-plugged disk is detected as "down". Waiting the default 10m for an OSD to detect a down disk never actually sets the OSD as down. Even after waiting multiple hours, the OSD doesn't automatically "down" itself.

I'm curious why the OSD would not detect when a hot-plugged disk is removed and set itself "down" after the default 10m interval. I notice that there are "Input/Output errors" and that the device-mapper (i.e. `/dev/mapper/*`) devices aren't automatically removed to accommodate the hot-plug event. The old OSD format (i.e. using `ceph-disk`) didn't have these issues since the OSD interfaced directly with disk partitions, and not through device-mapper/LVM, which I think is part of the issue here.

Any help and/or insight is appreciated.

 

[dcsapak]

i just tested this here and it works without problems...
do you have any activity on the ceph cluster while you are doing those tests?
it is possible that with the replacement of ceph-disk with ceph-volume it may be that such errors are only detected when there is activity on the disks

 

[Alwin]

See this thread.
https://forum.proxmox.com/threads/proxmox-6-0-ceph-osds-dont-show-down.56310/#post-259689

 

[maff1989]

do you have any activity on the ceph cluster while you are doing those tests?
it is possible that with the replacement of ceph-disk with ceph-volume it may be that such errors are only detected when there is activity on the disks

Thank you for the reply. This was the exact reason for my issue. I did some further testing yesterday after creating and running a VM and the disks were indeed marked "down" immediately after unplugging the drive due to the I/O error. It makes sense that an OSD will only be marked "down" once I/O fails.