PMX 8.1.4 - Ceph 18.2.1 install fail/ Non-operational

[jorel83]

Hi,

I have used proxmox since about 6-7 years with CEPH as storage, and just reinstalled the servers 4 host cluster with 8.1.4 and Ceph Reef (18.2.1) and must admit its been the most challenging part of Proxmox so far. Even fresh install it been multiple but this one is a show stopper.

Ceph has 20Gbit per host separated network from the 20G Cluster traffic network, all connectivity is operational.

Nothing with this upgrade has went smooth as we have come to expect from Proxmox and after overcoming most of it, CEPH simply refuses to operate properly.

Everything from:
rados_connect failed - No such file or directory (500)
binary not installed: /usr/bin/ceph-mon (500)
"Configuration already initialized"
got timeout (500)

I have uninstalled as per this thread, https://forum.proxmox.com/threads/removing-ceph-completely.62818/

All network part is operating normally but simply I cannot find a way forward.

Even this is on Hobby level for me, now the servers has been down for almost 5 days and it starts to be a problem.

Seems there arnt many threads about this issues so maybe better create one..

Any suggestions is very welcome.

Cheers

[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 172.16.102.102/24
fsid = d9693fd3-f095-4305-934d-ac51b97f35a0
mon_allow_pool_delete = true
mon_host = 10.254.10.13
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 10.254.10.12/24

[client]
keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.pmx4]
public_addr = 10.254.10.13










Only host 4 somehow seems to be able to start

 

[sb-jw]

just reinstalled the servers 4 host cluster with 8.1.4 and Ceph Reef (18.2.1)

Nothing with this upgrade has went smooth

Was it an upgrade or a new installation?

Is the Proxmox Cluster itself in good and functional condition?

 

[jorel83]

Was it an upgrade or a new installation?

Is the Proxmox Cluster itself in good and functional condition?

Yes the joining process was normal and gives no errors, this is a new installation with latest iso off the proxmox website.

Tried to downgrade to older Ceph but that completley fails, it implies proxmox should be downgraded to achieve Ceph 17.2

 

[alex-cm]

Hi all, any update here? This is terrible. My 8.1.2 was working great, than I tried to passthrough another nvme disk (not related to ceph) and it all went haywire. Then wiped the entire cluster (I was still in setup) did a clean install of 8.2.2 (i though hey, it is brand new and should be an improvement....) I couldn't even finish the ceph setup.

EXACT same behavior as described above. The first mgr dies after a few second from startup and, once that is gone, mon are dead in the water. Cluster is stuck. I could now even complete the osd creation.

PS. I also did the zap/destroy manual command from previous osd's disks.
PPS. I'm on enterprise "stable" repos.

Very disappointed that there is no clear docs on how to remove ceph, let alone to try and recover a broken cluster.

Considering to go back to metal for my use-case.

 

[alex-cm]

My bad, I was so sure that have reconfigured the cluster correctly that I did not check the networking. I have a non-trivial interfaces config and I something went south while reconfiguring the cluster.

Eventually it was always a networking fault that generated my issue.

As soon as I fixed that back. It all started to work flawlessy.

Tried to downgrade to older Ceph but that completley fails, it implies proxmox should be downgraded to achieve Ceph 17.2

Check your networking. It might help. Are you using separates nets fir ceph data and public?