hi - I have a new install of PVE with 4 nodes, all with Ceph installed. when I go to create OSDs, specifically two on one 2TB NVMe, I get this:
it's right that there are no keyrings that match those names. there is ceph.client.admin.keyring and ceph.mon.keyring in /etc/pve/priv, but it's not looking for either of those.
contents of ceph.conf:
ETA: it also seems to think one of my four nodes is offline, even though everything with respect to it works. it has a red X instead of a green check, and there's a bunch of this in syslog:
as far as I can tell, time on all four nodes is identical. they are all using htpdate because NTP is blocked for me.
Code:
root@riogrande:/etc/pve/priv# ceph-volume lvm batch --osds-per-device 2 /dev/nvme0n1
--> DEPRECATION NOTICE
--> You are using the legacy automatic disk sorting behavior
--> The Pacific release will change the default to --no-auto
--> passed data devices: 1 physical, 0 LVM
--> relative data size: 0.5
Total OSDs: 2
Type Path LV Size % of device
----------------------------------------------------------------------------------------------------
data /dev/nvme0n1 953.86 GB 50.00%
----------------------------------------------------------------------------------------------------
data /dev/nvme0n1 953.86 GB 50.00%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) yes
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 950219b9-c16d-4115-893d-9dae1297f064
stderr: 2022-12-12T02:42:29.623-0600 7f2d074bc700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
stderr: 2022-12-12T02:42:29.623-0600 7f2d074bc700 -1 AuthRegistry(0x7f2d00060800) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 AuthRegistry(0x7f2d00060800) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 AuthRegistry(0x7f2d000657d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
stderr: 2022-12-12T02:42:29.627-0600 7f2d0625a700 -1 AuthRegistry(0x7f2d062590d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
stderr: [errno 2] RADOS object not found (error connecting to the cluster)
--> RuntimeError: Unable to create a new OSD id
root@riogrande:/etc/pve/priv#it's right that there are no keyrings that match those names. there is ceph.client.admin.keyring and ceph.mon.keyring in /etc/pve/priv, but it's not looking for either of those.
contents of ceph.conf:
Code:
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 192.168.9.10/24
fsid = a817096f-27b2-4213-90ad-5eddeff5af87
mon_allow_pool_delete = true
mon_host = 192.168.9.10 192.168.9.13 192.168.9.11
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 2
public_network = 192.168.9.10/24
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[mon.localhost]
public_addr = 192.168.9.10
[mon.orinoco]
public_addr = 192.168.9.13
[mon.riogrande]
public_addr = 192.168.9.11ETA: it also seems to think one of my four nodes is offline, even though everything with respect to it works. it has a red X instead of a green check, and there's a bunch of this in syslog:
Code:
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/riogrande: -1
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/riogrande: /var/lib/rrdcached/db/pve2-node/riogrande: illegal attempt to update using time 1670835560 when last update time is 1670837149 (minimum one second step)
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: -1
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: /var/lib/rrdcached/db/pve2-storage/riogrande/local: illegal attempt to update using time 1670835560 when last update time is 1670837289 (minimum one second step)
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: -1
Dec 12 02:59:20 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: illegal attempt to update using time 1670835560 when last update time is 1670837289 (minimum one second step)
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/riogrande: -1
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/riogrande: /var/lib/rrdcached/db/pve2-node/riogrande: illegal attempt to update using time 1670835570 when last update time is 1670837149 (minimum one second step)
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: -1
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local: /var/lib/rrdcached/db/pve2-storage/riogrande/local: illegal attempt to update using time 1670835570 when last update time is 1670837289 (minimum one second step)
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: -1
Dec 12 02:59:30 riogrande pmxcfs[1408]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: /var/lib/rrdcached/db/pve2-storage/riogrande/local-lvm: illegal attempt to update using time 1670835570 when last update time is 1670837289 (minimum one second step)as far as I can tell, time on all four nodes is identical. they are all using htpdate because NTP is blocked for me.
Last edited: