Proxmox 2.1 HA - fencing using BMC - Dell PE860s

[hotwired007]

I have an interesting issue with my HA setup, 3x Dell 860s using BMC, fencing using the BMC is tested and working. (removed the lanplus option from ipmi config and works fine )

The issue:

If i use fence_node node3, the VM image running on the machine gets migrated.

If i pull the power cable on the machine, the VM doesnt migrate although a few secs after i plug it back in, the VM is migrated to another node.

I'd expect the VM to migrate if the power cable was pulled... any ideas what i ahve got configured wrong?

 

[dietmar]

I assume your fencing device does not work without power?

 

[nate]

(removed the lanplus option from ipmi config and works fine )

i have a couple of dell 1950s i've been trying to get fencing to work with using bmc, but couldn't get it to work. What is up with removing lanplus? What happens when that is in the config? I'm wondering if that might have been my issue.

 

[hotwired007]

I assume your fencing device does not work without power?

No, its the BMC on the Dell 860.

I'm assuming that the monitoring will only work if the server has power? That seems odd as i'd expect the system to check if the server is contactable and if not then to migrate the VM to a running server... is this a bug or expected behavior?

 

[hotwired007]

i have a couple of dell 1950s i've been trying to get fencing to work with using bmc, but couldn't get it to work. What is up with removing lanplus? What happens when that is in the config? I'm wondering if that might have been my issue.

i also had to install ipmitool on all of my servers - ive not yet tested my Dell 1950s but it seems to work fine on my 860s.

my cluster.conf:

<?xml version="1.0"?>
<cluster name="ACCSYS-CLUSTER" config_version="7">

  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>

    <fencedevices>
        <fencedevice agent="fence_ipmilan" name="node1-bmc" ipaddr="xxx.xxx.xxx.xxx" login="root" passwd="password" power_wait="5"/>
        <fencedevice agent="fence_ipmilan" name="node2-bmc" ipaddr="xxx.xxx.xxx.xxx" login="root" passwd="password" power_wait="5"/>
        <fencedevice agent="fence_ipmilan" name="node3-bmc" ipaddr="xxx.xxx.xxx.xxx" login="root" passwd="password" power_wait="5"/>
    </fencedevices>
    <clusternodes>
    <clusternode name="node1" votes="1" nodeid="1">
        <fence>
            <method name="1">
                 <device name="node1-bmc"/>
            </method>
        </fence>
    </clusternode>
    <clusternode name="node2" votes="1" nodeid="2">
        <fence>
            <method name="1">
                 <device name="node2-bmc"/>
            </method>
        </fence>
    </clusternode>
    <clusternode name="node3" votes="1" nodeid="3">
        <fence>
            <method name="1">
                 <device name="node3-bmc"/>
            </method>
        </fence>
    </clusternode>
</clusternodes>
</cluster>

 

[dietmar]

is this a bug or expected behavior?

You simply need a working fencing device.

 

[hotwired007]

You simply need a working fencing device.

The BMC works fine, surely its a better assumption that if the host is unavailble for two machines in a cluster the currently running VM will need to be migrated to a node that IS contactable, when the system contacts the the fence device, if it uncontactable the cluster should assume that the machine is down and migrate the VM.

 

[dietmar]

The BMC works fine,

From what you write it does not seem to work if power is down?

surely its a better assumption that if the host is unavailble for two machines in a cluster the currently running VM will need to be migrated to a node that IS contactable, when the system contacts the the fence device, if it uncontactable the cluster should assume that the machine is down and migrate the VM.

You probable do not understand fencing. The whole purpose is to make 100% sure that a host is down. Assuming a host is down because it is uncontactable is very dangerous.

 

[hotwired007]

You probable do not understand fencing. The whole purpose is to make 100% sure that a host is down. Assuming a VM is down because it is uncontactable is very dangerous.

I assume you mean host and not VM? In my opinion if the machine is uncontactable on both the PVE IP address and the BMC IP address and the cluster cannot communicate to it and it cannot communicate with the cluster - it IS fenced and cannot do anything to the cluster data or VM storage. If i have misunderstood this can you explain how it would cause any damage?

 

[dietmar]

I assume you mean host and not VM?

yes (I just corrected that in above post).

In my opinion if the machine is uncontactable on both the PVE IP address and the BMC IP address and the cluster cannot communicate to it and it cannot communicate with the cluster - it IS fenced and cannot do anything to the cluster data or VM storage.

This assumption is simply wrong.

If i have misunderstood this can you explain how it would cause any damage?

What if communication only works in one direction (a simple iptables rule can do that)?

 

[dietmar]

If i have misunderstood this can you explain how it would cause any damage?

Also, an node can have several network interfaces, and can access shared resources through other paths (FC for example).

 

[hotwired007]

What if communication only works in one direction (a simple iptables rule can do that)?

But isnt that something that would need to be configured to break it - it not like a hardware failure could cause it.

 

[dietmar]

But isnt that something that would need to be configured to break it - it not like a hardware failure could cause it.

Hardware failure can cause similar behavior. Also switches can do strange things ;-)

 

[hotwired007]

I understand that there may be configurations where a host may have other connections - on my clusters all my hosts have two NICs with an onboard BMC/DRAC that shares the network connection - the server only communicates to the network using these two NICs, is there a way i can make my cluster assume that if the BMC is uncontactable that the VMs need migrating without buying any extra hardware?

 

[dietmar]

Again, this is not safe. So if you want HA, you need fencing.

 

[nate]

i also had to install ipmitool on all of my servers - ive not yet tested my Dell 1950s but it seems to work fine on my 860s.

Thanks for that. Can you give me any info on installing ipmitool on the proxmox ve server? Looks complicated, is it?

 

[hotwired007]

Looks complicated, is it?

all i did was aptitude install ipmitool and everything worked - no configuration required.