ceph-volume lvm create => error connecting to the cluster

nowrap

Member
Nov 2, 2016
9
0
6
46
Hello,
i am trying to get ceph up and running again after migrating proxmox v5 to v6 on 3 nodes.

After the upgrade the old lumious setup didn't work anymore due to 2 "ghost" monitors (mon.0 and mon.1).
I couldn't remove them and the GUI says "mon_command failed - command not known (500) ".

So i purged ceph and start from scratch like in this thread:
https://forum.proxmox.com/threads/reinstall-ceph-on-proxmox-6.57691/page-2#post-298616

Now i have ceph nautilus installed and the can add monitors, managers and mds.

But i cannot recreate my osds:
Bash:
ceph-volume lvm create --filestore --data /dev/sdc2 --journal /dev/sda3

Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 744dd792-33a4-4d8f-a495-33eb0992afbf
 stderr: [errno 1] error connecting to the cluster
-->  RuntimeError: Unable to create a new OSD id

This command worked before upgrading.

The file /var/lib/ceph/bootstrap-osd/ceph.keyring was missing after purging and reinstalling.
So recreated it:
Bash:
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd'
ceph-authtool /etc/pve/priv/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring

But i didn't make a difference.

Thx for any hints and regards
nowrap
 
The new ceph cluster should already have bootstrap keys. Run a ceph auth list. You should see them there.

And to completely remove Ceph, you can run pveceph purge.

ceph-volume lvm create --filestore --data /dev/sdc2 --journal /dev/sda3
Best use our tooling for it, pveceph osd create.
 
Hello Alwin,

thanks for your reply.
I purged ceph with these commans:
Bash:
apt purge ceph ceph-osd ceph-mds ceph-mon ceph-mgr ceph-base
pveceph purge

pveceph install
dpkg-reconfigure ceph-common
dpkg-reconfigure ceph-fuse

truncate -s 0 /etc/pve/ceph.conf
pveceph init --network 192.168.0.0/24
pveceph mon create

But this doesn't creates a /var/lib/ceph/bootstrap-osd/ceph.keyring
The self generated osd keyring via ceph-authtool differs from the output of ceph auth list .

I cannot use pveceph osd create because it works only with complete disks not partitions.
Someone has written a patch for it. But ceph-volume lvm create worked before.

I'll change the osd keyring and try again.

// EDIT
It works ;) But i am getting new errors:
Code:
stderr: Volume group name  has invalid characters

Regards
nowrap
 
Last edited:
Manually i did it this.
Giving the lvm vgs names again, fixed it.
Two OSD are up and running ;)

Thx 4 your help
nowrap
 
Best use our tooling for it, pveceph osd create.
I want to point out that this tool does not offer all functionality that ceph-volume does- especially not the batch functionality which makes it SO much more useful.

edit: it works just fine. problems with that specific node :)

Since I do want to have that function, can you point me at how to overcome the keyring not found issue?

Code:
ceph-volume lvm batch --bluestore /dev/sdc /dev/sdd /dev/sde /dev/sdj /dev/sdg /dev/nvme0n1
--> DEPRECATION NOTICE
--> You are using the legacy automatic disk sorting behavior
--> The Pacific release will change the default to --no-auto
--> passed data devices: 5 physical, 0 LVM
--> relative data size: 1.0
--> passed block_db devices: 1 physical, 0 LVM


Total OSDs: 5


  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  data            /dev/sdc                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sdd                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sde                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sdj                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
----------------------------------------------------------------------------------------------------
  data            /dev/sdg                                                3.64 TB         100.00%
  block_db        /dev/nvme0n1                                            186.30 GB       20.00%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) y
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0577f974-1ee9-4660-b2b7-17d6adcfe891
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 AuthRegistry(0x7fad3805adf8) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 AuthRegistry(0x7fad380603e0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2021-07-14T10:27:25.428-0700 7fad3e1b2700 -1 AuthRegistry(0x7fad3e1b10e0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: [errno 2] RADOS object not found (error connecting to the cluster)
-->  RuntimeError: Unable to create a new OSD id

Code:
# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.124-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-4
pve-kernel-helper: 6.4-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-4.15: 5.4-19
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-30-pve: 4.15.18-58
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.3-1
proxmox-backup-client: 1.1.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!