[SOLVED] PVE6 Ceph cluster storage creation issue

Mar 1, 2018
22
1
43
Hi,

to test an intended setup I created a three node cluster with a dedicated storage network for Ceph. After setting up the cluster, I installed and configured Ceph. When I set up a pool for my disk images in Ceph, I chose "Add as storage". The pool and the storage got created, but the storage stayed unavailable for all of my cluster nodes (grey question mark). When examining the storage configuration I figured out that the checkbox for "Use Proxmox VE managed hyper-converged ceph pool" was ticked. The monitor names were the server names of my monitors (noth their ip addresses). I unchecked the hyper-converged checkbox, entered the ip addresses of my monitors in the storage network and the pool name. After I clicked "OK", the storage was online on all of the nodes.
Could it be that Use Proxmox VE managed hyper-converged ceph pool has a problem if there is no name resulution possible between the cluster nodes? If so, could this be considered as a bug?

My servers do not know each other by name and since they share two corosync networks, one vm network and one storage network I do net intend to add them to each others hosts files.

Markus
 
This usually works out of the box. How does your ceph.conf look like? And could you add a 'ceph config dump'?
 
Hi Alwin,

for cephfs, it worked. For the block storage not.
Below you will find the requested information.

Markus



ceph.conf

Code:
[global]
         auth_client_required = none
         auth_cluster_required = none
         auth_service_required = none
         cluster_network = 192.168.23.1/24
         fsid = 8ddc47bd-98a7-49fd-8277-c485a5c43687
         mon_allow_pool_delete = true
         mon_host = 192.168.23.1 192.168.23.2 192.168.23.3
         osd_pool_default_min_size = 2
         osd_pool_default_size = 3
         public_network = 192.168.23.1/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.tvm01]
         host = tvm01
         mds_standby_for_name = pve

[mds.tvm02]
         host = tvm02
         mds_standby_for_name = pve

[mds.tvm03]
         host = tvm03
         mds standby for name = pve


ceph config dump
Code:
root@tvm01:~# ceph config dump
WHO MASK LEVEL OPTION VALUE RO
root@tvm01:~#
 
Last edited:
This did the trick.

I just created a pool and storage from the ceph configuration. Like before, storage was not available on the hosts.
As soon as I removed the keyring file from /etc/pve/priv/ceph, I could access the newly created storage.

Thanks for helping, Alwin!

Markus
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!