ceph-volume lvm create => error connecting to the cluster

[nowrap]

Hello,
i am trying to get ceph up and running again after migrating proxmox v5 to v6 on 3 nodes.

After the upgrade the old lumious setup didn't work anymore due to 2 "ghost" monitors (mon.0 and mon.1).
I couldn't remove them and the GUI says "mon_command failed - command not known (500) ".

So i purged ceph and start from scratch like in this thread:
https://forum.proxmox.com/threads/reinstall-ceph-on-proxmox-6.57691/page-2#post-298616

Now i have ceph nautilus installed and the can add monitors, managers and mds.

But i cannot recreate my osds:

ceph-volume lvm create --filestore --data /dev/sdc2 --journal /dev/sda3

Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 744dd792-33a4-4d8f-a495-33eb0992afbf
 stderr: [errno 1] error connecting to the cluster
-->  RuntimeError: Unable to create a new OSD id

This command worked before upgrading.

The file /var/lib/ceph/bootstrap-osd/ceph.keyring was missing after purging and reinstalling.
So recreated it:

ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd'
ceph-authtool /etc/pve/priv/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring

But i didn't make a difference.

Thx for any hints and regards
nowrap

 

[Alwin]

The new ceph cluster should already have bootstrap keys. Run a ceph auth list. You should see them there.

And to completely remove Ceph, you can run pveceph purge.

ceph-volume lvm create --filestore --data /dev/sdc2 --journal /dev/sda3

Best use our tooling for it, pveceph osd create.

 

[nowrap]

Hello Alwin,

thanks for your reply.
I purged ceph with these commans:

apt purge ceph ceph-osd ceph-mds ceph-mon ceph-mgr ceph-base
pveceph purge

pveceph install
dpkg-reconfigure ceph-common
dpkg-reconfigure ceph-fuse

truncate -s 0 /etc/pve/ceph.conf
pveceph init --network 192.168.0.0/24
pveceph mon create

But this doesn't creates a /var/lib/ceph/bootstrap-osd/ceph.keyring
The self generated osd keyring via ceph-authtool differs from the output of ceph auth list .

I cannot use pveceph osd create because it works only with complete disks not partitions.
Someone has written a patch for it. But ceph-volume lvm create worked before.

I'll change the osd keyring and try again.

// EDIT
It works But i am getting new errors:

stderr: Volume group name  has invalid characters

Regards
nowrap

 

[Alwin]

stderr: Volume group name has invalid characters

Did you zap the drives before trying again?

 

[nowrap]

Manually i did it this.
Giving the lvm vgs names again, fixed it.
Two OSD are up and running

Thx 4 your help
nowrap

 

[alexskysilk]

Best use our tooling for it, pveceph osd create.

I want to point out that this tool does not offer all functionality that ceph-volume does- especially not the batch functionality which makes it SO much more useful.

edit: it works just fine. problems with that specific node

Since I do want to have that function, can you point me at how to overcome the keyring not found issue?

ceph-volume lvm batch --bluestore /dev/sdc /dev/sdd /dev/sde /dev/sdj /dev/sdg /dev/nvme0n1
--> DEPRECATION NOTICE
--> You are using the legacy automatic disk sorting behavior
--> The Pacific release will change the default to --no-auto
--> passed data devices: 5 physical, 0 LVM
--> relative data size: 1.0
--> passed block_db devices: 1 physical, 0 LVM


Total OSDs: 5


  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  data            /dev/sdc                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sdd                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sde                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sdj                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sdg                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) y
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0577f974-1ee9-4660-b2b7-17d6adcfe891
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 AuthRegistry(0x7fad3805adf8) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 AuthRegistry(0x7fad380603e0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 AuthRegistry(0x7fad3e1b10e0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: [errno 2] RADOS object not found (error connecting to the cluster)
-->  RuntimeError: Unable to create a new OSD id

# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.124-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-4
pve-kernel-helper: 6.4-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-4.15: 5.4-19
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-30-pve: 4.15.18-58
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.3-1
proxmox-backup-client: 1.1.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1