Urgent: cman problems

cesarpk

Well-Known Member
Mar 31, 2012
770
3
58
Plaese URGENT, can anyone help me?

I'll set two nodes "PVE" clustered.

I am currently setting up the first PVE node with "mdadm" RAID1, then I added network intefaces,
then by GUI I add bridges on the node, and all configuration of vmbr0 move to vmbr1 (same IP/mask/gw), then vmbr0 not have IP/mask/gw.
So far everything works fine, restart, etc without any error message.

Later I execute on the node in this order:
pvecm create MY-CLUSTER ".
cp /etc/pve/cluster.conf /etc/pve/cluster.conf.new
nano /etc/pve/cluster.conf.new
# I add: two_node="1" expected_votes="1"
In the GUI execute Activate
#The GUI don't show me message errors
reboot #finally

After reboot the node I have problems with cman:
If I run: "service cman restart", I see:

Stopping cluster:
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... two_node set but there are more than 2 nodes
cman_tool: corosync daemon didn't start Check cluster logs for details
[FAILED]

Will appreciate very much to anyone who can help me
 
Plaese URGENT, can anyone help me?

I'll set two nodes "PVE" clustered.

I am currently setting up the first PVE node with "mdadm" RAID1, then I added network intefaces,
then by GUI I add bridges on the node, and all configuration of vmbr0 move to vmbr1 (same IP/mask/gw), then vmbr0 not have IP/mask/gw.
So far everything works fine, restart, etc without any error message.

Later I execute on the node in this order:
pvecm create MY-CLUSTER ".
cp /etc/pve/cluster.conf /etc/pve/cluster.conf.new
nano /etc/pve/cluster.conf.new
# I add: two_node="1" expected_votes="1"
In the GUI execute Activate
#The GUI don't show me message errors
reboot #finally

After reboot the node I have problems with cman:
If I run: "service cman restart", I see:

Stopping cluster:
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... two_node set but there are more than 2 nodes
cman_tool: corosync daemon didn't start Check cluster logs for details
[FAILED]

Will appreciate very much to anyone who can help me
Hi,
AFAIK the cluster communication use eth0 or vmbr0. Why you want to switch vmbr0 and vmbr1?? This are only numbers...

Udo
 
Hi,
AFAIK the cluster communication use eth0 or vmbr0. Why you want to switch vmbr0 and vmbr1?? This are only numbers...

Udo

Hi udo, your kind attention is very pleasing to me.

The answer: Just to be ordained, the PC has four network interfaces. I always configure "eth0" for the WAN connection, so for me it is easier to avoid confusion linking bridges numbers (vmbr0, 1, 2, etc.) to the same numbers of networks cards. (eth0, 1.2, etc.).

Re edited: Ah!!, i forget said that for installation of PVE and for PVE cluster communication i don't have a real DNS and gateway ¿is this a problem?

But if PVE for the cluster communication use necessarily eth0 or vmbr0, I'll have to change my habit .... oh i see the many errors that i will commit in confusions or distractions (snif, snif, snif)

Questions to staff members:
1- Then I want to query the staff of PVE if they can answer me: is correct that the PVE cluster communication use only eth0 or vmbr0?
Re edited: 2- Is necessary have a real DNS and/or gateway to make PVE cluster communication?
Re edited: 3- What things do not should I change to have a successful creation of pvecm?

Best regards
Cesar
 
Last edited:
Hi,
can you post your /etc/hosts and "cat /etc/pve/cluster.conf"?

Udo

Hi great teammate Udo

I reinstalled PVE, but this problem I had in the past, then i will like the next week make the same problem for know as correct it (reinstall all isn't nice), if you can help me... can you help me to know how to correct it? :p

Best regards
Cesar
 
Hi great teammate Udo

I reinstalled PVE, but this problem I had in the past, then i will like the next week make the same problem for know as correct it (reinstall all isn't nice), if you can help me... can you help me to know how to correct it? :p

Best regards
Cesar

Please Udo or anybody can help me?

This is an appendix:
1- I installed and configurated PVE and his cluster communication without have a real DNS and gateway ¿is this a problem?
Re edited: 2- After reconfiguring PVE cluster communication and without changing settings "eth0" and "vmbr0" I get the same problem.

These are the contents of the files:
/etc/hosts:

127.0.0.1 localhost.localdomain localhost
192.168.100.253 kvm1.brain.com.py kvm1 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

/etc/pve/cluster.conf:
<?xml version="1.0"?>
<cluster name="pve-cluster" config_version="2">

<cman keyfile="/var/lib/pve-cluster/corosync.authkey" two_node="1" expected_votes="1">
</cman>

<clusternodes>
<clusternode name="kvm1" votes="1" nodeid="1"/>
</clusternodes>

</cluster>

Re edited: The four last lines of corosync.log:
Feb 09 13:17:59 corosync [QUORUM] Using quorum provider quorum_cman
Feb 09 13:17:59 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
Feb 09 13:17:59 corosync [CMAN ] the two-node option requires exactly two nodes with one vote each and expected votes of 1 (node_count=1 vote_sum=1)
Feb 09 13:17:59 corosync [CMAN ] Error reading configuration, cannot start

I would appreciate any help
Best regards
Cesar
 
Last edited:
...
/etc/pve/cluster.conf:
<?xml version="1.0"?>
<cluster name="pve-cluster" config_version="2">

<cman keyfile="/var/lib/pve-cluster/corosync.authkey" two_node="1" expected_votes="1">
</cman>

<clusternodes>
<clusternode name="kvm1" votes="1" nodeid="1"/>
</clusternodes>

</cluster>

...

Hi,
what do you expect with an cluster which has only one member?

Udo
 
Hi,
what do you expect with an cluster which has only one member?

Udo

Hi Udo

Excuse me Udo, but my cluster will have two members, please see the second line of my first post of this thread: http://forum.proxmox.com/threads/12761-Urgent-cman-problems?p=69091#post69091

After several tests I found that the command "pvecm expected 1" is the solution to "PVE" start without problems and without error mesaages.

This brings me to two news doubts:
re edit: 1- is necessary add two_node="1" expected_votes="1" to cluster.conf file?,
re edit: 2- why?, and
re edit: 3- How to reverse the command "expected vote 1" if in the future I add more nodes?

Grateful to Udo or to anyone who can dispel my doubts.

Best regards
Cesar
 
Last edited:
Hi Udo

Excuse me Udo, but my cluster will have two members, please see the second line of my first post of this thread: http://forum.proxmox.com/threads/12761-Urgent-cman-problems?p=69091#post69091

After several tests I found that the command "pvecm expected 1" is the solution to "PVE" start without problems and without error mesaages.
"pvecm expected 1" is only for a two node cluster if one node fails! With shared storage and more than one running node it's dangeous!!
This brings me to two news doubts:
re edit: 1- is necessary add two_node="1" expected_votes="1" to cluster.conf file?,
re edit: 2- why?, and
re edit: 3- How to reverse the command "expected vote 1" if in the future I add more nodes?

Grateful to Udo or to anyone who can dispel my doubts.

Best regards
Cesar
Look at the cluster.conf of an working 3node-cluster:
Code:
<?xml version="1.0"?>
<cluster name="pve-cluster" config_version="3">

  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>

  <clusternodes>
    <clusternode name="proxmox3" votes="1" nodeid="1"/>
    <clusternode name="proxmox2" votes="1" nodeid="2"/>
    <clusternode name="proxmox1" votes="1" nodeid="3"/>
  </clusternodes>

</cluster>
mean, with only one node in clusternodes you don't have an running 2node-cluster.

Udo
 
Hi Udo

Thanks for your comments and preoccupations (Besides being a great master of masters, you are a great companion)

About of your comment: "pvecm expected 1" is only for a two node cluster if one node fails! With shared storage and more than one running node it's dangeous!!"

The answer: For me no is dangerous, since i use manual fence (ie with human interaction and without fence devices).

About of my doubts:
Re edit: 1- is necessary add two_node="1" expected_votes="1" to cluster.conf file?
(consider that I run the command "pvecm expected 1"),
2- why?, and
3- How to reverse the command "expected vote 1" if in the future I add more nodes?

Can you answer them please?

Re edit (I myself answer): The answer to the question 1 is no - tested, if i add these rules cman not start.

Best regards
Cesar
 
Last edited:
Hi,
about 2: No dns nessesary, only the information in /etc/hosts. And also no default gateway, because all nodes must be in the same network. I create normaly an own cluster-network on vmbr0 and use another bridge for web-access to the gui and this bridge has also an gateway entry.

about 3: "expected vote 2" or "expected vote 3" and so on?!

Udo
 
Hi,
about 3: "expected vote 2" or "expected vote 3" and so on?!

Hi Udo
Thanks for your answers, but i don't understand very well about of expected votes.

Questions:
1- As I understand expected vote is only for the quorum, right?
2- Considering that I should not edit the cluster.conf file to modify voting rules, and i don't run the command "pvecm expected", how will behave the votes system according I add more nodes? (for example if i have 2, 3, 4, 5 and 6 PVE nodes in one cluster).will be that "pvecm" only makes the calculation of minimum necessary votes for not to stay without a quorum in any case?.

Best regards
Cesar
 
Hi Udo
Thanks for your answers, but i don't understand very well about of expected votes.

Questions:
1- As I understand expected vote is only for the quorum, right?
2- Considering that I should not edit the cluster.conf file to modify voting rules, and i don't run the command "pvecm expected", how will behave the votes system according I add more nodes? (for example if i have 2, 3, 4, 5 and 6 PVE nodes in one cluster).will be that "pvecm" only makes the calculation of minimum necessary votes for not to stay without a quorum in any case?.

Best regards
Cesar
Hi,
look with "pvecm status" - any node has votes (normaly 1), the sum of all votes (normaly node votes 1 multiplied by nodes) are the expected votes.
E.G. number of nodes equals expected votes - very easy!
Code:
pvecm status
...
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
...
The quorum has e100 explained very good in other threads.

Udo
 
Hi,
look with "pvecm status" - any node has votes (normaly 1), the sum of all votes (normaly node votes 1 multiplied by nodes) are the expected votes.
E.G. number of nodes equals expected votes - very easy!
Code:
pvecm status
...
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
...
The quorum has e100 explained very good in other threads.

Udo

Hi UDO
Thank you very much for your time and explanations.

Please let me to do a questions:
1- If the "pvecm status" command say: "Quorum: 2", then i must understand that to avoid losing quorum must have at least two PVE nodes online right?
2- If the previous question is correct, What should I change if I want to change the value of Quorum? (For example today I have 2 PVE nodes (and I want only one vote to keep quorum), and tommorow i will add 1 PVE node to my PVE Cluster of two niodes and obviously I will have a total of three nodes (and I want only two votes to keep quorum)

Best regards
Cesar
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!