Communication error on all the proxmox nodes.

[srmvel]

Hi team,

I have a 11 nodes setup.When I browse the SSD pool-->summary or VM disks,Im getting the communication error.

becuase of this,I could not create any virtual machines from GUI.I see the same error on all the nodes.

Please advise how to fix?

 

[aaron]

Is this a Ceph pool? If so, what is the output of ceph -s? Do you get any other communication failures in any of the Ceph GUI panels?

 

[srmvel]

Yes.its ceph pool.
I cannot get any output of ceph -s. Connection aborted and from GUI nothing worked out.I see a question mark on my SSD & HDD CEPH pool.


We had a serious issues in the last weekend.This is 11 nodes cluster and dont know the cause of this issue yet.

Issues:-

1) 5 TB SSD pool became full due to recurrent cloning API request.This has been fixed by increasing SSD pool size.
2) We used 5 monitors for this cluster.Montior Quorum reported as "no" and could not bring up any of the monitor servers.
3) As soon as we clear the space,monitor root parition became full and used to do "compact" command on ceph.conf.No luck.

In short.

All monitors did not come online and space also became full recurrently as soon as we clearing up.
We tried with monitor compact command but nothing worked out.CEPH storage disapperaed and

Because of monitor and quorum unavailabity,CEPH crashed all the data got lost.

We created a 11 node cluster freshly and it started.