[SOLVED] ceph is hung

[huky]

i add a big disk as OSD to ceph cluster. and after one day,, my ceph is hung.

command 'ceph -s' hung

gui 'got timeout(500)'

how can i make it work?

in th process, i found disk full by /var/lib/ceph/mon/ceph-$NODE/store.db
now the node003 is 97%, node001 and node002 are full and i stop mon on the nodes.

 

[Alwin]

now the node003 is 97%, node001 and node002 are full and i stop mon on the nodes.

Ceph will not work without the MONs. You will need to provide more space to the partition where the store.db is located.

 

[huky]

th

Ceph will not work without the MONs. You will need to provide more space to the partition where the store.db is located.

thanks, i created mon on other nodes with enough disk, after a few hours the size of mon (/var/lib/ceph/mon/ceph-$nodename/store.db/) is equal, but ceph is still hung.


I got it.
refer :
"REMOVING MONITORS FROM AN UNHEALTHY CLUSTER" in https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/

delete two node and ceph cmd worked