Communication error on all the proxmox nodes.
- Thread starter srmvel
- Start date
Yes.its ceph pool.
I cannot get any output of ceph -s. Connection aborted and from GUI nothing worked out.I see a question mark on my SSD & HDD CEPH pool.
We had a serious issues in the last weekend.This is 11 nodes cluster and dont know the cause of this issue yet.
Issues:-
1) 5 TB SSD pool became full due to recurrent cloning API request.This has been fixed by increasing SSD pool size.
2) We used 5 monitors for this cluster.Montior Quorum reported as "no" and could not bring up any of the monitor servers.
3) As soon as we clear the space,monitor root parition became full and used to do "compact" command on ceph.conf.No luck.
In short.
All monitors did not come online and space also became full recurrently as soon as we clearing up.
We tried with monitor compact command but nothing worked out.CEPH storage disapperaed and
Because of monitor and quorum unavailabity,CEPH crashed all the data got lost.
We created a 11 node cluster freshly and it started.
I cannot get any output of ceph -s. Connection aborted and from GUI nothing worked out.I see a question mark on my SSD & HDD CEPH pool.
We had a serious issues in the last weekend.This is 11 nodes cluster and dont know the cause of this issue yet.
Issues:-
1) 5 TB SSD pool became full due to recurrent cloning API request.This has been fixed by increasing SSD pool size.
2) We used 5 monitors for this cluster.Montior Quorum reported as "no" and could not bring up any of the monitor servers.
3) As soon as we clear the space,monitor root parition became full and used to do "compact" command on ceph.conf.No luck.
In short.
All monitors did not come online and space also became full recurrently as soon as we clearing up.
We tried with monitor compact command but nothing worked out.CEPH storage disapperaed and
Because of monitor and quorum unavailabity,CEPH crashed all the data got lost.
We created a 11 node cluster freshly and it started.
Last edited:
Goodnight!
I went through the same problem and after a whole day of analysis and research, I came to the conclusion that there is no solution for this problem. It's frustrating to realize that a solution that has so much potential runs into situations like this, with no solution.
I hope there is some light, even though I have daily backup of my applications, I will still have a big loss of data worked on throughout the day. Not to mention the time for restoration.
After this incident I need to rethink whether it is worth working with Ceph integrated with Proxmox.
If anyone has any light I would be forever grateful.
I went through the same problem and after a whole day of analysis and research, I came to the conclusion that there is no solution for this problem. It's frustrating to realize that a solution that has so much potential runs into situations like this, with no solution.
I hope there is some light, even though I have daily backup of my applications, I will still have a big loss of data worked on throughout the day. Not to mention the time for restoration.
After this incident I need to rethink whether it is worth working with Ceph integrated with Proxmox.
If anyone has any light I would be forever grateful.