Help! 3-node cluster. One node down. Timeouts, ceph unavailable.

[kobaz]

I have a HA 3-node cluster with 3-replica ceph.
Just had a power outage, one of the nodes is down after a ups was flapping and the battery ran down. I get a kernel panic on boot. I think the boot drive is corrupted (but that's a different issue)

I believe the idea of ceph is we should be able to survive a 1 out of 3 node outage? Ceph is completely unavailabe.







Running ceph -s... I hit ctrl-c after 10 minutes of waiting
office-pmvm-01 {~} root# ceph -s
^CCluster connection aborted

strace is showing that it's trying to connect to 10.200.6.11 (which is down)

[pid 73196] setsockopt(14, SOL_TCP, TCP_NODELAY, [1], 4 <unfinished ...>
[pid 73197] <... setsockopt resumed>) = 0
[pid 73196] <... setsockopt resumed>) = 0
[pid 73197] connect(12, {sa_family=AF_INET, sin_port=htons(6789), sin_addr=inet_addr("10.200.6.11")}, 16 <unfinished ...>
[pid 73196] connect(14, {sa_family=AF_INET, sin_port=htons(3300), sin_addr=inet_addr("10.200.6.11")}, 16 <unfinished ...>
[pid 73197] <... connect resumed>) = -1 EINPROGRESS (Operation now in progress)
[pid 73196] <... connect resumed>) = -1 EINPROGRESS (Operation now in progress)
[pid 73197] epoll_ctl(6, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLET, {u32=12, u64=12}} <unfinished ...>
[pid 73196] epoll_ctl(3, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLET, {u32=14, u64=14}} <unfinished ...>
[pid 73197] <... epoll_ctl resumed>) = 0
[pid 73196] <... epoll_ctl resumed>) = 0
[pid 73197] connect(12, {sa_family=AF_INET, sin_port=htons(6789), sin_addr=inet_addr("10.200.6.11")}, 16 <unfinished ...>
[pid 73196] connect(14, {sa_family=AF_INET, sin_port=htons(3300), sin_addr=inet_addr("10.200.6.11")}, 16 <unfinished ...>
[pid 73197] <... connect resumed>) = -1 EALREADY (Operation already in progress)
[pid 73196] <... connect resumed>) = -1 EALREADY (Operation already in progress)
[pid 73197] epoll_ctl(6, EPOLL_CTL_MOD, 12, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=12, u64=12}} <unfinished ...>
[pid 73196] epoll_ctl(3, EPOLL_CTL_MOD, 14, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=14, u64=14}} <unfinished ...>
[pid 73197] <... epoll_ctl resumed>) = 0
[pid 73196] <... epoll_ctl resumed>) = 0
[pid 73197] epoll_wait(6, <unfinished ...>
[pid 73196] epoll_wait(3, ^C <unfinished ...>
[pid 73191] <... futex resumed>) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)

What are next steps?

Thanks!

 

[B.Otto]

Hmm, how many Ceph monitors and managers have you set up? The default Ceph GUI installation wizard only makes one of each and you should have at least 3 monitors. https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster#pve_ceph_monitors

 

[kobaz]

Hmm, how many Ceph monitors and managers have you set up? The default Ceph GUI installation wizard only makes one of each and you should have at least 3 monitors. https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster#pve_ceph_monitors

Each node has a monitor and it looks like the monitors are running on the remaining two nodes.


ceph 71638 0.0 0.1 422788 65084 ? Ssl Nov30 0:33 /usr/bin/ceph-mon -f --cluster ceph --id office-pmvm-01 --setuser ceph --setgroup ceph

But I think the monitors have failed.

[58918.643387] ceph: No mds server is up or the cluster is laggy
[58929.555584] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58932.720369] libceph: mon0 (1)10.200.6.10:6789 socket closed (con state OPEN)
[58936.531823] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58937.720776] libceph: mon0 (1)10.200.6.10:6789 socket closed (con state OPEN)
[58942.720924] libceph: mon0 (1)10.200.6.10:6789 socket closed (con state OPEN)
[58946.547876] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58953.559906] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58960.568184] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58962.721851] libceph: mon0 (1)10.200.6.10:6789 socket closed (con state OPEN)
[58966.708218] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58967.722083] libceph: mon0 (1)10.200.6.10:6789 socket closed (con state OPEN)
[58969.784215] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58972.722463] libceph: mon0 (1)10.200.6.10:6789 socket closed (con state OPEN)
[58976.564282] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58982.964544] libceph: mon1 (1)10.200.6.11:6789 socket closed (con state V1_BANNER)
[58986.228235] ceph: No mds server is up or the cluster is laggy

 

[kobaz]

If anyone runs into this... Here's what happened:

I had a 3-node ceph (OSD's on 3 nodes)
I had only two monitors. The backup-monitor was down.

I manually removed the second monitor from the ceph conf

Using using ceph commandline tools:
- Export the monmap
- Manually remove the monmap
- Inject the new monmap

##
monmaptool < foo
monmaptool foo --rm mon.office-pmvm-02
/usr/bin/ceph-mon -f --cluster ceph --id office-pmvm-01 --inject-monmap foo
##

pmvm-02 was the dead monitor
pmvm-01 is the surviving monitor

Finally... restart ceph on all nodes


Note for future reference:
In a 3-node cluster. Ensure all nodes have a monitor