Communication error on all the proxmox nodes.

srmvel

Member
Mar 16, 2021
34
0
6
37
Hi team,

I have a 11 nodes setup.When I browse the SSD pool-->summary or VM disks,Im getting the communication error.

becuase of this,I could not create any virtual machines from GUI.I see the same error on all the nodes.

Please advise how to fix?



1617440237184.png
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
3,152
536
118
Is this a Ceph pool? If so, what is the output of ceph -s? Do you get any other communication failures in any of the Ceph GUI panels?
 

srmvel

Member
Mar 16, 2021
34
0
6
37
Yes.its ceph pool.
I cannot get any output of ceph -s. Connection aborted and from GUI nothing worked out.I see a question mark on my SSD & HDD CEPH pool.


We had a serious issues in the last weekend.This is 11 nodes cluster and dont know the cause of this issue yet.

Issues:-

1) 5 TB SSD pool became full due to recurrent cloning API request.This has been fixed by increasing SSD pool size.
2) We used 5 monitors for this cluster.Montior Quorum reported as "no" and could not bring up any of the monitor servers.
3) As soon as we clear the space,monitor root parition became full and used to do "compact" command on ceph.conf.No luck.

In short.

All monitors did not come online and space also became full recurrently as soon as we clearing up.
We tried with monitor compact command but nothing worked out.CEPH storage disapperaed and

Because of monitor and quorum unavailabity,CEPH crashed all the data got lost.

We created a 11 node cluster freshly and it started.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!