Caution when using Proxmox node as Ceph MON

[wahmed]

This is just an informative Posting, not really an issue. Unless somebody has a solution for it because it is really an issue.

Ceph cluster is heavily depended on Monitors(MON) to operate correctly. Minimum 3 MONs needs to be online at all times to achieve Quorum. Currently i use Proxmox VM stored locally for all MONs thus saving cash and utility bill on Physical machine. But since i have Proxmox nodes running any way, i wanted to see if i could use the Proxmox Hosts as additional MONs. Last few weeks all together i tried 5 times. Although MONs worked perfectly, all Proxmox Nodes had unwanted side effects. Mostly #pvestatd was unstable, frequent connection error although all VMs were functioning just fine. WebGUI had big trouble showing any kind of stats such as Nodes on or off, CPU/Memory Usage graphs etc. Not being able to monitor entire Proxmox cluster in moments notice is really not an option for a real life Cluster.

Just an hour ago i took off all MONs from all Proxmox nodes and within seconds everything were back to normal again. I was able to reproduce this entire tests 5 times with same results every time. I am going to document it as "Do not use Proxmox for CEPH MONs". It just confirms Proxmox should be left alone just like any other stable platform to allow it to do what it does best.

Anybody tried something similar and had same results?

 

[dietmar]

Ceph cluster is heavily depended on Monitors(MON) to operate correctly. Minimum 3 MONs needs to be online at all times to achieve Quorum.

no, only two need to be online

Currently i use Proxmox VM stored locally for all MONs thus saving cash and utility bill on Physical machine. But since i have Proxmox nodes running any way, i wanted to see if i could use the Proxmox Hosts as additional MONs. Last few weeks all together i tried 5 times. Although MONs worked perfectly, all Proxmox Nodes had unwanted side effects. Mostly #pvestatd was unstable, frequent connection error

Do you use a separate network for ceph?

 

[wahmed]

no, only two need to be online

Doesnt it require Odd numbers of MONs to form Quorum? May be i misread back when i was learning CEPH. But you are right. CEPH seems to operate just fine even with 2 MONs. I just shutdown all MONs except 2 and everything still up and running just fine.

Do you use a separate network for ceph?

Yep. Ceph Cluster is on completely separate LAN with separate switch. Each Proxmox Node has 3 NICs. NIC#1 for management, NIC#2 to physical switch #1 to talk between all VMs, NIC#3 to physical switch #2 to provide shared storage. This is the switch also all Ceph Nodes are connected.

 

[dietmar]

Doesnt it require Odd numbers of MONs to form Quorum?[/qoute]

Quorum ==> majority of MONs are online (for 3 MONs, 2 needs to be online)

Yep. Ceph Cluster is on completely separate LAN with separate switch. Each Proxmox Node has 3 NICs. NIC#1 for management, NIC#2 to physical switch #1 to talk between all VMs, NIC#3 to physical switch #2 to provide shared storage. This is the switch also all Ceph Nodes are connected.

Strange. How can that influence the other network then? What NIC is uses for corosync/cman?

 

[wahmed]

Strange. How can that influence the other network then? What NIC is uses for corosync/cman?

Now that you mentioned it, may be corosync/cman trying to communicate on all NICs and LANs. Is there a way i can force a host to use only NIC#2 for all corosync/cman? Even though there are 2 subnets right now connected to 2 separate switch, good chance that corosyn/cman will try to communicate on all available switch.

 

[dietmar]

Now that you mentioned it, may be corosync/cman trying to communicate on all NICs and LANs.

CMAN only use one network. It is choosen by a reverse lookup of hostname in /etc/hosts.