[SOLVED] Create OSD using GPT partitions

[Firm]

Hi,
I'm trying to create Bluestore-backed OSDs using partitions not complete disk. Assuming that /dev/sda4 will be future OSD (placed on HDD drive), I created separate partition on SSD for DB/WAL. First try:

# pveceph osd create /dev/sda4 -db_dev /dev/sdc1 -db_size 54
unable to get device info for '/dev/sda4'

Ok, looks as pveceph requires complete disk, let's try with ceph-volume:

# ceph-volume lvm create --data /dev/sda4 --block.db /dev/sdc1
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 12c49b8f-f26c-4849-be9e-f38a306e3171
 stderr: 2021-08-01T12:01:12.415+0300 7fd86e01f700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
 stderr: 2021-08-01T12:01:12.415+0300 7fd86e01f700 -1 AuthRegistry(0x7fd8680597a0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: 2021-08-01T12:01:12.419+0300 7fd86cdbd700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-08-01T12:01:12.419+0300 7fd86cdbd700 -1 AuthRegistry(0x7fd8680597a0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2021-08-01T12:01:12.419+0300 7fd86cdbd700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-08-01T12:01:12.419+0300 7fd86cdbd700 -1 AuthRegistry(0x7fd86805be40) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2021-08-01T12:01:12.419+0300 7fd86cdbd700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-08-01T12:01:12.419+0300 7fd86cdbd700 -1 AuthRegistry(0x7fd86cdbc120) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: [errno 2] RADOS object not found (error connecting to the cluster)
-->  RuntimeError: Unable to create a new OSD id

I copied content of /etc/pve/priv/ceph.client.admin.keyring to /etc/pve/priv/ceph.client.bootstrap-osd.keyring and /var/lib/ceph/bootstrap-osd/ceph.keyring but the last command still reports the error:

# ceph-volume lvm create --data /dev/sda4 --block.db /dev/sdc1
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d54be965-1c10-45e2-9d58-5deb16851de1
stderr: 2021-08-01T11:39:02.584+0300 7f17cd6ce700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
 stderr: [errno 1] RADOS permission error (error connecting to the cluster)
-->  RuntimeError: Unable to create a new OSD id

What could went wrong?

# pveversion
pve-manager/6.4-13/9f411e79 (running kernel: 5.4.128-1-pve)
# ceph -v
ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)

Regards,
Alex

 

[Firm]

Upd. Looks as I need to carefully read Cephx Config Reference.

 

[Firm]

Looks as I solved it. Key problem was to place correct data into /etc/pve/priv/ceph.client.bootstrap-osd.keyring and /var/lib/ceph/bootstrap-osd/ceph.keyring files. So, you just need to get auth data for client.bootstrap-osd (real key is replaced):

# ceph auth ls
// skipped and screened
client.bootstrap-osd
    key: ABCD==
    caps: [mon] allow profile bootstrap-osd
// skipped

Now, create both keyring files mentioned above with correct data:

[client.bootstrap-osd]
    key = ABCD==
    caps mon = "allow profile bootstrap-osd"

Don't forget to change owner/group for /var/lib/ceph/bootstrap-osd/ceph.keyring:

chown ceph:ceph /var/lib/ceph/bootstrap-osd/ceph.keyring

Now re-run initial command.

P.S. You may need to specify --block-db-size for your DB/WAL partition - not sure if it's required.

Regards,
Alex