I fixed it with a reboot of the switch, but it bothers me that it happened in the first place - It started after a reboot of the cluster - here's what I was seeing:
Proxceph1:
Node Sts Inc Joined Name
1 M 624 2015-02-04 14:45:46 proxceph1
2 M 988 2015-02-04 14:53:11 proxceph2
3 X 0 proxceph3
4 M 884 2015-02-04 14:50:35 proxve1
5 X 884 proxve2
proxceph2:
root@proxceph2:~# pvecm nodes
Node Sts Inc Joined Name
1 M 988 2015-02-04 14:53:11 proxceph1
2 M 872 2015-02-04 14:50:11 proxceph2
3 X 0 proxceph3
4 M 988 2015-02-04 14:53:11 proxve1
5 X 988 proxve2
proxceph3:
Node Sts Inc Joined Name
1 X 0 proxceph1
2 X 0 proxceph2
3 M 1672 2015-02-04 15:07:40 proxceph3
4 X 0 proxve1
5 X 0 proxve2
proxve2:
root@proxve2:~# pvecm nodes
Node Sts Inc Joined Name
1 X 0 proxceph1
2 X 0 proxceph2
3 X 0 proxceph3
4 X 0 proxve1
5 M 1860 2015-02-04 15:11:41 proxve2
They are all running the same version of ProxMox.
Anyone have any idea what could have happened with regard to the switch that broke the cluster that rebooting the switch fixed? I'm not even quite sure where to begin to look to see if I have something screwed up in my switch unless it blocked multicast for some reason on some ports? Thoughts?
Proxceph1:
Node Sts Inc Joined Name
1 M 624 2015-02-04 14:45:46 proxceph1
2 M 988 2015-02-04 14:53:11 proxceph2
3 X 0 proxceph3
4 M 884 2015-02-04 14:50:35 proxve1
5 X 884 proxve2
proxceph2:
root@proxceph2:~# pvecm nodes
Node Sts Inc Joined Name
1 M 988 2015-02-04 14:53:11 proxceph1
2 M 872 2015-02-04 14:50:11 proxceph2
3 X 0 proxceph3
4 M 988 2015-02-04 14:53:11 proxve1
5 X 988 proxve2
proxceph3:
Node Sts Inc Joined Name
1 X 0 proxceph1
2 X 0 proxceph2
3 M 1672 2015-02-04 15:07:40 proxceph3
4 X 0 proxve1
5 X 0 proxve2
proxve2:
root@proxve2:~# pvecm nodes
Node Sts Inc Joined Name
1 X 0 proxceph1
2 X 0 proxceph2
3 X 0 proxceph3
4 X 0 proxve1
5 M 1860 2015-02-04 15:11:41 proxve2
They are all running the same version of ProxMox.
Anyone have any idea what could have happened with regard to the switch that broke the cluster that rebooting the switch fixed? I'm not even quite sure where to begin to look to see if I have something screwed up in my switch unless it blocked multicast for some reason on some ports? Thoughts?