Ignoring custom ceph config for storage

[Timothy1056]

Good day I was wondering how to get rid of this error.

Jun 28 14:37:01 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:37:12 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:37:21 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:37:31 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:37:41 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:37:51 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:38:01 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:38:11 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:38:21 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:38:31 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)! Jun 28 14:38:41 pve13 pvestatd[1495]: ignoring custom ceph config for storage 'CephData', 'monhost' is not set (assuming pveceph managed cluster)!


Currently everything is operating as normal.

 

[jsterr]

Please post your ceph.conf cat /etc/pve/ceph.conf and also the output of ceph health

 

[Timothy1056]

CAT /etc/pve/ceph.conf

cat /etc/pve/ceph.conf
[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
         cluster_network = 192.168.50.11/24
         fsid = 7e77370a-73da-4441-a059-8337ba12eff3
         mon_allow_pool_delete = true
         mon_host = 192.168.50.29 192.168.50.19 192.168.50.8 192.168.50.23
         ms_bind_ipv4 = true
         ms_bind_ipv6 = false
         osd_pool_default_min_size = 2
         osd_pool_default_size = 3
         public_network = 192.168.50.11/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mon.pve13]
         public_addr = 192.168.50.23

[mon.pve18]
         public_addr = 192.168.50.29

[mon.pve19]
         public_addr = 192.168.50.19

[mon.pve8]
         public_addr = 192.168.50.8

Ceph Health - I am aware of the one Mons being down.

ceph health
HEALTH_WARN 1/4 mons down, quorum pve18,pve8,pve13; all OSDs are running quincy or later but require_osd_release < quincy

 

[Philipp Hufnagl]

Does that config look the same on all of your nodes?

 

[Timothy1056]

Sorry for such a delay response.

I have confirmed that all the of the config files are the same in /etc/pve/ceph.conf

 

[sb-jw]

First of all, the note that you shouldn't deploy 4 Mons. Either 3, 5 or a maximum of 7. In your setup you will easily get by with 3. Larger setups with several hundred OSDs only need 5 and 7 in very special cases.

Do you only have the error message on pve13? What does your /etc/pve/storage.cfg look like? Do you have anything left under /etc/ceph?

 

[Timothy1056]

Hi @sb-jw I have reduced the Monitors to 3.

Please find attached storage.cfg

For the /etc/ceph directory there are two file in there one named ceph.conf & rbdmap

 

[sb-jw]

Do you only have the error message on pve13?

the above question and does the rbdmap file exist on all servers? You don't actually need that and you can delete it; that might solve the error.

 

[Timothy1056]

This error appears on all the servers and the rbdmap file is located on every server as well.

I have noticed that the ceph.conf file is missing on PVE7, PVE10 and PVE11

 

[sb-jw]

I have noticed that the ceph.conf file is missing on PVE7, PVE10 and PVE11

The one under /etc/ceph/ceph.conf (actually just a symlink) or /etc/pve/ceph.conf? If the last one, you should urgently check the nodes in question, because the folder should be replicated to all of them and should not differ.

 

[Timothy1056]

This will be the one located under /etc/ceph/ceph.conf

 

[Philipp Hufnagl]

ceph health
HEALTH_WARN 1/4 mons down, quorum pve18,pve8,pve13; all OSDs are running quincy or later but require_osd_release < quincy

Did you fix this? If you do not update the osd version for 2 or more releases, you can run into quite severe issues. Here is the wiki to this topic.

Also, before you do this, make sure you have a working backup (just in case).

 

[Timothy1056]

Hi @Philipp Hufnagl Unfortunately still having this problem, I will upgrade the OSD versions and update asap.

 

[Timothy1056]

I can see they are upgraded to Version 17.2.7

 

[Philipp Hufnagl]

Hello

The issue is not that the OSDs are not upgraded. The issue is that your system is still set to allow an older OSD version than you are using. CEPH guarantees only compatibility for 2 versions older than you are using. Having a bigger difference to the required release most likely will corrupt your OSDs and force you to restore from backup.

 

[Timothy1056]

When running

ceph mon dump | grep min_mon_release

I get the following output

min_mon_release 17 (quincy)

Based on the documentation you sent me in your previous message, The setting to only allow Quincy is set And the error is still present

 

[Philipp Hufnagl]

Hey

Did you add the CEPH storage as a local or as a remote storage?

As far as I can currently tell, the error you are receiving should only be possible if you add an external ceph cluster.

 

[Timothy1056]

It is a local ceph cluster created on proxmox.