Proxmox Totally Disconnected From Ceph cluster. Cannot reconnect.

djfreak

New Member
May 27, 2023
26
6
3
I'll try to be as brief as possible but still include necessary info.

Six Nodes on various hardware named proxmox-proxmox6

CEPH 19.2. Proxmox 8.3.2.

18 OSD's containing over 500 placement groups totalling 3TB in a standard block storage pool which has worked beautifully.

Suddenly however, not sure if it was related to the 19.2 upgrade or not, suddenly I was unable to list my disks in the proxmox GUI when I clicked in by storage pool. I got a communication error. I was also unable to delete VM's without manual intervention. Deleting VM's produced an RBD error.

I have hit the chatGPT limit 4 times now. I no longer have any access to my storage pool. Question mark in the Proxmox GUI. If you know how much chat it takes to hit that limit four times then you know how deep I am into this trying to fix it. Without success.

We (ChatGPT and I) had absolutely tremendous difficulty getting any of my previous monitor nodes (proxmox3-proxmox6) to reinitialize. Finally we figured out that ceph.conf had the incorrect keyring path which FINALLY allowed us to initialize the monitor however when we start the monitor service, it still fails. I just hit another chatGPT limit as we were in the process of dealing with this.

Please tell me there is a way of rebuilding the crush map or rebuilding whatever we need to rebuild to initiate another GUI based reconfig of the CEPH cluster without losing my data. Yes I have backups but rebuilding this cluster will still take weeks I'm really hopeful that there's something obvious I'm missing here. I have tried absolutely everything that chatGPT could possibly think of over two days including reinstalling CEPH while preserving settings which appeared to be successful.

Thank you very much in advance. My only priority is preserving those placement groups. Happy to do whatever rebuilding is required.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!