[SOLVED] PVE6 Ceph cluster storage creation issue

[markus-r]

Hi,

to test an intended setup I created a three node cluster with a dedicated storage network for Ceph. After setting up the cluster, I installed and configured Ceph. When I set up a pool for my disk images in Ceph, I chose "Add as storage". The pool and the storage got created, but the storage stayed unavailable for all of my cluster nodes (grey question mark). When examining the storage configuration I figured out that the checkbox for "Use Proxmox VE managed hyper-converged ceph pool" was ticked. The monitor names were the server names of my monitors (noth their ip addresses). I unchecked the hyper-converged checkbox, entered the ip addresses of my monitors in the storage network and the pool name. After I clicked "OK", the storage was online on all of the nodes.
Could it be that Use Proxmox VE managed hyper-converged ceph pool has a problem if there is no name resulution possible between the cluster nodes? If so, could this be considered as a bug?

My servers do not know each other by name and since they share two corosync networks, one vm network and one storage network I do net intend to add them to each others hosts files.

Markus

 

[Alwin]

This usually works out of the box. How does your ceph.conf look like? And could you add a 'ceph config dump'?

 

[markus-r]

Hi Alwin,

for cephfs, it worked. For the block storage not.
Below you will find the requested information.

Markus



ceph.conf

[global]
         auth_client_required = none
         auth_cluster_required = none
         auth_service_required = none
         cluster_network = 192.168.23.1/24
         fsid = 8ddc47bd-98a7-49fd-8277-c485a5c43687
         mon_allow_pool_delete = true
         mon_host = 192.168.23.1 192.168.23.2 192.168.23.3
         osd_pool_default_min_size = 2
         osd_pool_default_size = 3
         public_network = 192.168.23.1/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.tvm01]
         host = tvm01
         mds_standby_for_name = pve

[mds.tvm02]
         host = tvm02
         mds_standby_for_name = pve

[mds.tvm03]
         host = tvm03
         mds standby for name = pve

ceph config dump

root@tvm01:~# ceph config dump
WHO MASK LEVEL OPTION VALUE RO
root@tvm01:~#

 

[Alwin]

Can you please also check if there are ceph keyrings under '/etc/pve/priv/'?

It might be related to this bug entry, while it is for PVE 5.x, it may need revisitng.
https://bugzilla.proxmox.com/show_bug.cgi?id=2006

 

[markus-r]

This did the trick.

I just created a pool and storage from the ceph configuration. Like before, storage was not available on the hosts.
As soon as I removed the keyring file from /etc/pve/priv/ceph, I could access the newly created storage.

Thanks for helping, Alwin!

Markus