Proxmox 2.2 Clustering, HA Fencing (/etc/pve/cluster.conf syntax)

webservio

Renowned Member
May 13, 2009
106
1
83
First of all my appreciation for Tom, Dietmar, and other members of Proxmox group for their hard work!
My purpose of posting this thread is more of contribution by providing some details of my research and work done so far and discussion with other engineers about Clustering and HA. Without going too much details of the information I have read all of the following:

http://pve.proxmox.com/wiki/High_Availability_Cluster
http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster
http://pve.proxmox.com/wiki/Fencinge

Although my hardware design is slightly different than Intel Modular Server I still used the instructions from:
http://pve.proxmox.com/wiki/Intel_Modular_Server_HA

My test implementation involves with the following hardware with two node member in mind:
- two Dell PE 1950 both with DRAC5
- Using a cluster ready storage using SAS5E controllers and MD3000 subsystem

The shared storage model using LVM group and clustering was very much simple however I ran into challenges configuring the HA, Quorum, and fencing that I would like to share:

- The main issue and problem is obtaining the exact and correct the syntax of /etc/pve/cluster.conf in which I ran into.
- Quorum Disk Configuration: I understand there are multiple ways to implement this: iscsi, etc. However i simply used a separate shared storage for both nodes with no problem. The part I do not still understand is the exact syntax of the information in /etc/pve/cluster.conf.
- Back to /etc/pve/cluster.conf I ran into multiple issues that I like to share:
The command:
ccs_config_validate /etc/pve/cluster.conf.new
returns no error:
Configuration validates
where:
ccs_config_validate -v -f /etc/pve/cluster.conf.new
Creating temporary file: /tmp/tmp.u6XMWVmwbl
Config interface set to:
Configuration stored in temporary file
Updating relaxng schema
Validating..
tempfile:18: parser error : Opening and ending tag mismatch: clusternodes line 11 and clusternode
</clusternode>
^
tempfile:25: parser error : expected '>'
</clusternode>
^
tempfile:25: parser error : Extra content at the end of the document
</clusternode>
^
Validation completed


I believe the main challenges would resolve if we together work on a clear documentation for /etc/pve/cluster.conf.

I would be more than happy to discuss further.

Regards,
 
the error message points you directly to your problem, check your tags!
 
Hi Tom,

Thanks very much for your post! below is partial listing of my cluster.conf.new:

<clusternodes>
<clusternode name="vpshost49" votes="1" nodeid="1"/>
<fence>
<method name="1">
<device name="vpshost49drac"/>
</method>
</fence>
</clusternode>
<clusternode name="vpshost46" votes="1" nodeid="2"/>
<fence>
<method name="1">
<device name="vpshost46drac"/>
</method>
</fence>
</clusternode>
</clusternodes>

Which generates the errors:

tempfile:18: parser error : Opening and ending tag mismatch: clusternodes line 11 and clusternode
</clusternode>
^
tempfile:25: parser error : expected '>'
</clusternode>
^
tempfile:25: parser error : Extra content at the end of the document
</clusternode>
^

The tags seem to be valid. What am I missing here please?

Regards
 
Hi,

I believe I caught the cause of the error and fixed by replacing:
<clusternode name="vpshost46" votes="1" nodeid="2"/>

to:

<clusternode name="vpshost46" votes="1" nodeid="2">


However my question about quorum disk remains:

Where exactly in cluster.conf would I define and what is the syntax?

Thanks,
 
Hi,

I believe I caught the cause of the error and fixed by replacing:
<clusternode name="vpshost46" votes="1" nodeid="2"/>

to:

<clusternode name="vpshost46" votes="1" nodeid="2">


However my question about quorum disk remains:

Where exactly in cluster.conf would I define and what is the syntax?

Thanks,


As I mentioned I did create the quorum disk on a shared storage between the two nodes using:

http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster#Create_the_Quorum_Disk

root@vpshost49:~# mkqdisk -c /dev/sdd1 -l cluster1_qdisk
mkqdisk v1349169030

Writing new quorum disk label 'cluster1_qdisk' to /dev/sdd1.
WARNING: About to destroy all data on /dev/sdd1; proceed [N/y] ? y
Warning: Initializing previously initialized partition
Initializing status block for node 1...
Initializing status block for node 2...
Initializing status block for node 3...
Initializing status block for node 4...
Initializing status block for node 5...
Initializing status block for node 6...
Initializing status block for node 7...
Initializing status block for node 8...
Initializing status block for node 9...
Initializing status block for node 10...
Initializing status block for node 11...
Initializing status block for node 12...
Initializing status block for node 13...
Initializing status block for node 14...
Initializing status block for node 15...
Initializing status block for node 16...


However not sure where I would place the information. I assume it should be in cluster.conf?
 
As I mentioned I did create the quorum disk on a shared storage between the two nodes using:

http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster#Create_the_Quorum_Disk

However not sure where I would place the information. I assume it should be in cluster.conf?
In my cluster.conf (Datacenter Tab 'HA') I have the following:
<cluster config_version="xx" name="<clustername>">
<cman .... />
<quorumd allow_kill="0" interval="3" label="proxmox1_qdisk" tko="10">
<heuristic interval="3" program="ping $GATEWAY -c1 -w1" score="1" tko="4"/>
<heuristic interval="3" program="ip addr | grep eth0 | grep -q UP" score="2" tko="3"/>
</quorumd>
<totem token="54000"/>
 
Mir,

After posting I found an example which is very similar to yours and it worked nicely.
root@vpshost49:~# clustat
Cluster Status for cluster1 @ Sun Oct 28 20:56:56 2012
Member Status: Quorate

Member Name ID Status
------ ---- ---- ------
vpshost49 1 Online, Local, rgmanager
vpshost46 2 Online, rgmanager
/dev/block/8:49 0 Offline, Quorum Disk

Service Name Owner (Last) State
------- ---- ----- ------ -----
pvevm:100 vpshost46 started
root@vpshost49:~#
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!