Proxmox 2-node cluster - fencing device

[lousek]

Hello Forum,

I am actually trying out to build up a two-node cluster with Proxmox VE 2.0 Beta (RC1).
My test setup are two HP ML110 G5 with Intel Xeon CPU, 8 GB of memory, and identical hard disk configuration.
Both have an additional network card, which are connected directly (no switch between!) together.
I would like to use this additional NICs for cluster and DRBD sync.

At this article: http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster
or this one: http://pve.proxmox.com/wiki/Fencing
there is the written about a "fencing device".
If I understand correctly, the "failed" host is powered-off by the master to prevent dual access to the same disk files.
In my configuration, I neither have a iLO nor I have a "controlable" UPS.

Is there any way to use my direct network connection between the two hosts as "fencing" device?
Or DRBD?

BR & Thanks,
lousek

 

[MicW]

I try to setup an equal installation:
- Servers, both have an additional NIC
- 2-primary DRBD synced over both Servers using this connection
- 2 identical Proxmox 2.0 installations

I set up a cluster with this both nodes. I did not set up any fencing. Manual migration of normal nodes works as expected.

But when I shut down one node, i cannot failover it's machines anymore. Even moving the Configs like 101.cfg fails because /etc/pve gets read only.

So the same question from me: how can i make it working without having a fencing device?

Regards, Michael.

 

[MicW]

I found out that I can change the "expected quorum" value for my cluster to 1 with the command "pvecm expected 1".
This means that the Cluster works if there is "1 vote". Each reachable node has a vote. So with setting the expected value to 1, the culster is considered to be intact if there is one node running (which is fine for a 2 node cluster).

So now I can manually move a node from an offline cluster to an online cluster by moving it's config (e.g. from host-h1b to host-h1a):
mv /etc/pve/nodes/host-h1b/qemu-server/*.conf /etc/pve/nodes/host-h1a/qemu-server/

Then I can start the node manually on the other node.

This is not full-ha but at least hot standby with manual failover. I'll keep trying to get the automatic failover work.

 

[dietmar]

I'll keep trying to get the automatic failover work.

You can't do automatic failover without fencing. At least it is very dangerous to do that.

 

[lousek]

Hello you two

I managed now also to have everything running (Cluster, DRBD, except HA).

But a fencing device is used to "power cycle" the failed node to make sure that the VMs are no longer running on it, right?
Can't I do something over the direct network connection? Or maybe a direct serial connection (=terminal access)?

BR,
Lukas

 

[dietmar]

Can't I do something over the direct network connection? Or maybe a direct serial connection?

A cable is a cable (and does not include a switch)

 

[MicW]

You can't do automatic failover without fencing. At least it is very dangerous to do that.

But with a 2-Node Setup which are connected by a direct cable, it should be ok, right? I cannot imagine a situation where it is really dangerous (except when someone pulls out the network cable) - but in this case drbd would detect the split-brain szenario and would switch one of the nodes to secondary so that the filesystem would be read-only.

Please correct me, when I'm wrong with that assumption.

 

[MicW]

For my sceenario I could use a script that sends a hardware reset to the other node (an API is provided by our hoster for this). Would this be valid fencing? The other node would then do a normal startup. But in case of that the other node cannot be reached, would this node then start the vms (which would cause inconsistencies)? If yes, first it would kill the other node. So both nodes would then keep killing each other?

 

[lousek]

Of course. I mean, there is no other device (e.g. a switch) which can fail. If the other host is not reachable over the direct connection, it is A) overloaded or B) really down (failed)

Btw: Is there a possibility to configure Proxmox to check the cluster over both NICs?
Actually, Proxmox Cluster has configured the nodes automatically with the "external" IP-addresses (not with them of the direct connection).
Now if the switch dies, DRBD is still syncing over direct connection, but Proxmox cluster goes into a "split-brain situation" I guess ... would be nice when it will check also over direct connection.

BR,
lousek

 

[dietmar]

Of course. I mean, there is no other device (e.g. a switch) which can fail. If the other host is not reachable over the direct connection, it is A) overloaded or B) really down (failed)

or C: cable is damaged
or D: network card on node 1 is damaged
or E: network card on node 2 is damaged
or F: PCI bus problem on node2
or G: ....

Btw: Is there a possibility to configure Proxmox to check the cluster over both NICs?
Actually, Proxmox Cluster has configured the nodes automatically with the "external" IP-addresses (not with them of the direct connection).

The suggestion is to use bonding to get redundant network access.

 

[MicW]

For testing I changed my test domain to "ha managed". I set up no fencing for testing). Then i started the domain on node B and switched node B off. Nothing hapend (no failover or so).
Did I forget someting to configure?

Is there a way to maunally force a failover (i.e. restart) the domain on node A, after node B went down?

 

[dietmar]

For testing I changed my test domain to "ha managed". I set up no fencing for testing). Then i started the domain on node B and switched node B off. Nothing hapend (no failover or so).
Did I forget someting to configure?

Yes: FENCING

 

[MicW]

i guessed that.

Now i added "manual" fencing as described in the cman manual:

<?xml version="1.0"?>
<cluster config_version="8" name="h1">
<cman keyfile="/var/lib/pve-cluster/corosync.authkey" two_node="1" expected_votes="1"/>
<fencedevices>
<fencedevice name="human" agent="fence_manual"/>
</fencedevices>
<clusternodes>
<clusternode name="host-h1a" nodeid="1" votes="1">
<fence>
<method name="single">
<device name="human" nodename="host-h1a"/>
</method>
</fence>
</clusternode>
<clusternode name="host-h1b" nodeid="2" votes="1">
<fence>
<method name="single">
<device name="human" nodename="host-h1b"/>
</method>
</fence>
</clusternode>
</clusternodes>
<rm>
<pvevm autostart="1" vmid="105"/>
</rm>
</cluster>

Then I started vm 105 on node B and rebootet node B. Nothing happend to vm 105 (it was not migrated).

Update:
seems that rmanager is not running. I got the following in dlm_controld.log:
Mar 07 11:25:00 dlm_controld dlm_join_lockspace no fence domain
Mar 07 11:25:00 dlm_controld process_uevent online@ error -1 errno 2

Update2:
I forgot to join the "fencing domain" as described in the wiki:
http://pve.proxmox.com/wiki/Fencing#Enable_fencing_on_all_nodes

Then the rgmanager must be started on all nodes. Watch /var/log/cluster/*.log for what's happening.

After I did this, all works as expected:

- Start vm105 on node B
- poweroff node B
- go to node A. Do manual fencing with command: fence_ack_manual host-h1b
- confirm with "absolutely"
The other node takes over vm 105 as expected.

This is the final solution for the original request as well. There are 2 posssibilities:
- doing manual fencing (means that one must confirm that the other node is down)
- disabling fencing completely (not recommended): creating a dummy fence script that does nothing.

Instead of doing nothing, the script could use all available channels (both NICs, serial connection, ...) to check if the other host is really down. If not, fencing fails, if yes, fencing is ok. This is still not as good as real fencing but may be ok if at least one channel is reliable.

Update 3:
In http://doc.opensuse.org/products/draft/SLE-HA/SLE-ha-guide_sd_draft/cha.ha.fencing.html many ways of fencing including fencing via software are described.

 

[Nascire]

Hello,

i´m interested in the solution of a custom fence script.
I have no possibility to use any supported fence agent at my hosting provider,
but the possibility to trigger a reboot/shutdown of the servers via webinterface-api.

May someone be able to tell me how this script should/could look like?

This is how it currently looks like, but I´m sure I´ve missed something important,
such as an exit code or some stuff:

#!/bin/sh
while getopts "a:l" opt;
do
case $opt in
a) address=$2 ;;
l) user=$2 ;;
p) pass=$2 ;;
o) action=$2 ;;
*) echo "Usage: $0 -a -l -p -o" ; exit 1 ;;
esac
echo "Issuing: curl -u $user:$pass https://robot-ws.your-server.de/reset/$address -d type=$action"
curl -u $user:$pass https://robot-ws.your-server.de/reset/$address -d type=$action
done

Any help would be greatly appreciated.

 

[MicW]

Hello Nascire,

someone wrote a fencing script that uses the webinterface of hetzner.de (a german hosting provider) to shut down the other node which may be a good entry point. I could not find it now but you'll find it with the keywords "hetzner stonith agent" somewhere in a mailing list.

Regards,
Michael.

 

[Nascire]

Hello MicW,

thank you for this suggestion - with these I found this link, which helped me few steps further:
http://www.gossamer-threads.com/lists/linuxha/dev/72740?do=post_view_threaded

With some further research I´ve also found this one, which describes how to implement a fencing agent:
https://fedorahosted.org/cluster/wiki/FenceAgentAPI

My script is finished and running, in case that everyone else needs it, either directly for hetzner, or to adapt it, here it is:

###############################################################################################################
#
# Possible parameters from STDIN described below
#
#
# action = operation (on, off, reboot, monitor, list, status)
# on - needs to be implemented -> WOL
# off - not supported, will be automatic hardware reset
# reboot - will we CTL+ALT+DEL
# monitor - not supported
# list - not supported
# status - not supported
# option = obsolet parameter - use action instead
# not parsed
# ipaddr = hostname or ip
# login = username or login name
# passwd = password
# passwd_script = for script outside of cluster config
# not implemented
# port = if port needs to be specified
# not implemented
# nodename = if agent fences by node name, choose between nodename and port - preference although is port
# not parsed
#
###############################################################################################################

# define some variables
timestamp=date
logfile=/var/log/fence_hetzner.log

# First check if we are running from command line
if [ $# -gt 0 ]
then
while getopts "a:l:" opt;
do
# parse valid arguments
case $opt in
a) address=$OPTARG ; echo $address ;;
l) user=$OPTARG ; echo $user ;;
p) pass=$OPTARG ; echo $pass ;;
o) action=$OPTARG ; echo $action ;;
# description needs to be implemented
*) echo "Usage: $0 -a -l -p -o" ; exit 1 ;;
esac
done
# ok, so we are getting parameters from fenced
else
while read LINE;
do
# split input by =, and parse arguments
param=`echo $LINE | awk -F "=" '{print $1}'`
case $param in
ipaddr) address=`echo $LINE | awk -F "=" '{print $2}'` ;;
login) user=`echo $LINE | awk -F "=" '{print $2}'` ;;
passwd) pass=`echo $LINE | awk -F "=" '{print $2}'` ;;
action) action=`echo $LINE | awk -F "=" '{print $2}'` ;;
esac
done
fi

# translate action to hetzner webservice supported
case $action in
off) action=hw ;;
reboot) action=sw ;;
esac

# write command to be issued to logfile
echo "[$date] Issuing: curl -u $user":"$pass https://robot-ws.your-server.de/reset/$address -d type=$action \n" >> $logfile

# function to parse return code from webservice
if [ $1 = 200 ]
then
echo "[$date] STATUS 200 - Reset ok" >> $logfile
return 0
else
# write some basic error message to logfile
case $1 in
400) echo "[$date] ERROR 400 - Invalid Input" >> $logfile ;;
404) echo "[$date] ERROR 404 - Server with $address not found" >> $logfile ;;
409) echo "[$date] ERROR 409 - Manual reset already active" >> $logfile ;;
500) echo "[$date] ERROR 500 - Reset failed due to internal error" >> $logfile ;;
esac
return 1
fi
}

# issue our constructed call to parse function
check_http_response $(curl --silent -o /dev/null -w '%{http_code}' -u $user:$pass https://robot-ws.your-server.de/reset/$address -d type=$action)

# exit with return code defined by function
exit $?

 

[cesarpk]

Right, it Works

Best regard
Cesar

 

[Nikhil]

Dear All, I am looking help on proxmox 4 HA ..

I wanted make HA setup on proxmox 4.
can any help me on this. will appreciate your time.

Thanks,
Nikihl