new install, cannot create Ceph OSDs bc of keyring error

starkruzr

Well-Known Member
hi - I have a new install of PVE with 4 nodes, all with Ceph installed. when I go to create OSDs, specifically two on one 2TB NVMe, I get this:

Code:
root@riogrande:/etc/pve/priv# ceph-volume lvm batch --osds-per-device 2 /dev/nvme0n1
--> DEPRECATION NOTICE
--> You are using the legacy automatic disk sorting behavior
--> The Pacific release will change the default to --no-auto
--> passed data devices: 1 physical, 0 LVM
--> relative data size: 0.5

Total OSDs: 2

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  data            /dev/nvme0n1                                            953.86 GB       50.00%
----------------------------------------------------------------------------------------------------
  data            /dev/nvme0n1                                            953.86 GB       50.00%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) yes
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 950219b9-c16d-4115-893d-9dae1297f064
 stderr: 2022-12-12T02:42:29.623-0600 7f2d074bc700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
 stderr: 2022-12-12T02:42:29.623-0600 7f2d074bc700 -1 AuthRegistry(0x7f2d00060800) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 AuthRegistry(0x7f2d00060800) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 AuthRegistry(0x7f2d000657d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 AuthRegistry(0x7f2d062590d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: [errno 2] RADOS object not found (error connecting to the cluster)
-->  RuntimeError: Unable to create a new OSD id
root@riogrande:/etc/pve/priv#

it's right that there are no keyrings that match those names. there is ceph.client.admin.keyring and ceph.mon.keyring in /etc/pve/priv, but it's not looking for either of those.

contents of ceph.conf:

Code:
[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 192.168.9.10/24
     fsid = a817096f-27b2-4213-90ad-5eddeff5af87
     mon_allow_pool_delete = true
     mon_host = 192.168.9.10 192.168.9.13 192.168.9.11
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 2
     public_network = 192.168.9.10/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.localhost]
     public_addr = 192.168.9.10

[mon.orinoco]
     public_addr = 192.168.9.13

[mon.riogrande]
     public_addr = 192.168.9.11


ETA: it also seems to think one of my four nodes is offline, even though everything with respect to it works. it has a red X instead of a green check, and there's a bunch of this in syslog:

Code:
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/riogrande: -1
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/riogrande: /var/lib/rrdcached/db/pve2-node/riogrande: illegal attempt to update using time 1670835560 when last update time is 1670837149 (minimum one second step)
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: -1
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: /var/lib/rrdcached/db/pve2-storage/riogrande/local: illegal attempt to update using time 1670835560 when last update time is 1670837289 (minimum one second step)
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: -1
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: illegal attempt to update using time 1670835560 when last update time is 1670837289 (minimum one second step)
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/riogrande: -1
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/riogrande: /var/lib/rrdcached/db/pve2-node/riogrande: illegal attempt to update using time 1670835570 when last update time is 1670837149 (minimum one second step)
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: -1
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: /var/lib/rrdcached/db/pve2-storage/riogrande/local: illegal attempt to update using time 1670835570 when last update time is 1670837289 (minimum one second step)
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: -1
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: illegal attempt to update using time 1670835570 when last update time is 1670837289 (minimum one second step)

as far as I can tell, time on all four nodes is identical. they are all using htpdate because NTP is blocked for me.
 
Last edited:
I tried copying the keyring file that does exist into the locations it seems to be looking for it. still getting this:

Code:
root@riogrande:~# ceph-volume lvm batch --osds-per-device 2 /dev/nvme0n1
--> DEPRECATION NOTICE
--> You are using the legacy automatic disk sorting behavior
--> The Pacific release will change the default to --no-auto
--> passed data devices: 1 physical, 0 LVM
--> relative data size: 0.5

Total OSDs: 2

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  data            /dev/nvme0n1                                            953.86 GB       50.00%
----------------------------------------------------------------------------------------------------
  data            /dev/nvme0n1                                            953.86 GB       50.00%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) yes
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 35ddb225-63a4-4775-8624-e23bbaac579b
 stderr: 2022-12-12T13:43:27.975-0600 7f3b51d22700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
 stderr: 2022-12-12T13:43:27.975-0600 7f3b51521700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
 stderr: [errno 1] RADOS permission error (error connecting to the cluster)
-->  RuntimeError: Unable to create a new OSD id
 
Hi,

I have same problem than you (trying to create 2 osd by nvme, as currently gui is missing this feature)

simply:

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

to generate the keyring at the correct location.
 
  • Like
Reactions: takeokun