Ceph: OSD down for changed disk path

corsinvest

Renowned Member
Good morning,
in a 3 node Proxmox - Ceph cluster in some situations (disk fail, host badly stopped, etc.) disks path (/dev/sda, /dev/sdb, etc.) changed after reboot; but in this cases the related OSD was down (I suppose because OSD was built in the gui choosing hdd and optional ssd journal with their device path /dev/sdb, /dev/sdc and so on).

Is there a manner to create OSD using physical disk id (/dev/disk/by-id/*) instead of device path?

Thanks,
best regards
 
Hi,
this happened to me as well at one point. However, it seems that basically, OSDs do not rely on their device paths. After another reboot of that node they came back, even though the device paths were different. I think I had the journals on the same disks at that time. However, when I look at one of my OSDs in /var/lib/ceph/osd/ceph-<ID> I see the journal symlink references its disk by partition id:
Code:
# ls -ld /var/lib/ceph/osd/ceph-12/journal
lrwxrwxrwx 1 ceph ceph 58 Mar 24 17:52 journal -> /dev/disk/by-partuuid/fabe9f66-eef6-47c1-811b-2171c5c69a64
So this should be fine as well.

I am taking a wild guess here, but check the above link for the OSD in question. Maybe older versions of Ceph / pveceph did not use the partition ID? If this is the case, you can change the journal link, but you must be careful (in a nutshell: stop the OSD, flush the journal, change link and (re-)create journal. I did this for testing purposes a few times, so if you want more details on the process I might be able to dig them out of my notes.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!