Create Ceph OSD on SAN in Proxmox 6

[Mino]

Hello,

With previous proxmox 5 version, i was able to create ceph OSD on a multipath LUN with the command ceph-disk.
Now this command disappeared in proxmox 6.

I need to test ceph in my Lab, which only has SAN as storage.
How could i create OSD with a /dev/mapper/xxxxxx device. The ceph-volume command return errors when i try.

 

[t.lamprecht]

ceph-disk was depreacate, use ceph-volume instead.

See https://docs.ceph.com/docs/nautilus/ceph-volume/ and https://docs.ceph.com/docs/nautilus/install/manual-deployment/#adding-osds

Which exact command did you used?

 

[Mino]

For example, with multipath device /dev/mapper/LUN_ceph_bay1_01, the underlying paths are /dev/sdc and /dev/sdd.

This is the way i could create OSD before on Proxmox 5 / Ceph 12 :

ceph-disk prepare /dev/mapper/LUN_ceph_bay1_01 --cluster-uuid CLUSTER_UUID_HERE
ceph-disk activate /dev/mapper/LUN_ceph_bay1_01
ceph-disk suppress-activate /dev/sdc
ceph-disk suppress-activate /dev/sdd

Now with ceph-volume on Proxmox 6 / Ceph 14 :

ceph-volume lvm prepare --data /dev/mapper/LUN_ceph_bay1_01

Result is :

Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new OSD_UUID_HERE
stderr: [errno 22] error connecting to the cluster
--> RuntimeError: Unable to create a new OSD id

The file /var/lib/ceph/bootstrap-osd/ceph.keyring doesn't exist, so I don't see how this can work.

 

[Alwin]

The file /var/lib/ceph/bootstrap-osd/ceph.keyring doesn't exist, so I don't see how this can work.

Get the keyring from the cluster ceph auth get-or-create client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring. Usually it is created on the beginning of the installation.

 

[Mino]

Ok thanks for the reply.

I generated the new keyring but still had an error :

root@server:~# ceph-volume lvm prepare --data /dev/mapper/LUN_ceph_baie1_01
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0e901ab2-c308-4e66-8fbf-9f656baa45c9
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
 stderr: 2019-09-04 09:21:07.399 7fe1c1678700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2019-09-04 09:21:07.399 7fe1c1678700 -1 AuthRegistry(0x7fe1bc07f818) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: purged osd.0
-->  RuntimeError: Cannot use device (/dev/mapper/LUN_ceph_baie1_01). A vg/lv path or an existing device is needed

So i copied the keyring in `/etc/pve` and the first error disappeared :

root@server:~# cp /var/lib/ceph/bootstrap-osd/ceph.keyring /etc/pve/priv/ceph.client.bootstrap-osd.keyring

root@server:~# ceph-volume lvm prepare --data /dev/mapper/LUN_ceph_baie1_01
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 55e5188b-7b09-4d79-ae14-30a1f0684492
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
 stderr: purged osd.0
-->  RuntimeError: Cannot use device (/dev/mapper/LUN_ceph_baie1_01). A vg/lv path or an existing device is needed

I tried creating a VG/LV on the multipath device, but with no success :

root@server:~# vgcreate vg_ceph_baie1_01 /dev/mapper/LUN_ceph_baie1_01
  Physical volume "/dev/mapper/LUN_ceph_baie1_01" successfully created.
  Volume group "vg_ceph_baie1_01" successfully created

root@server:~# lvcreate -n lv_ceph vg_ceph_baie1_01 -l+100%FREE
  Logical volume "lv_ceph" created.

root@server:~# ceph-volume lvm prepare --data /dev/mapper/vg_ceph_baie1_01-lv_ceph
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new bed4b417-9972-4b1c-a858-b2a641d44acd
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
 stderr: purged osd.0
-->  RuntimeError: Cannot use device (/dev/mapper/vg_ceph_baie1_01-lv_ceph). A vg/lv path or an existing device is needed

Do you have any idea why it's not working ?

 

[Alwin]

ceph-volume wants physical devices or partitions. You may try with vg_ceph_baie1_01/lv_ceph.

In general, please don't use foreign devices for OSDs, it has higher latency and may be unstable for Ceph usage.

 

[Mino]

It doesn't work with `vg_ceph_baie1_01/lv_ceph`. I got the same error.

I know CEPH is supposed to work only with physical disks, but as i said, this is a lab environment where i only have SAN storage.
So there is no way to install an OSD on a /dev/mapper device ?

 

[Alwin]

It doesn't work with `vg_ceph_baie1_01/lv_ceph`. I got the same error.

Does this mean, you tried without the /dev/mapper in front?