[SOLVED] HUGE Fencing problem with IPMI

O

offerlam

Renowned Member
Dec 30, 2012
218
0
81
Denmark
Hi all,I have 3 HP DL 165 G7 servers which im trying to use with proxmox. If we can get this working its the plan that we get atleast a community licens but for this proof of concept we still running the open source free edition.The Problem:We are trying to setup fecing and its somewhat working. If i reboot a node the VMs fail over to other nodes. The problem comes when i pull the power cord directly from a server. This does NOT result the vm in migrating to other host. They stay on the failed host and over time shows the web icon for being powered off. If i at any point put power back to the node it will start to migrate the VMs during the servers post operation.another issue:These servers has ILO100 BMC controller and you access the BMC controller by adding a IPMI IP in the BIOS. You also have to choose shared nics which means you can reach IPMI web interface on all nics in the server. Currently my ip are as followsproxmox00 10.10.99.20 - BMC IP 10.10.99.30proxmox01 10.10.99.21 - BMC IP 10.10.99.31proxmox02 10.10.99.22 - BMC IP 10.10.99.32we have been able to both ping the BMC IP and access the web untill a few days ago. All of sudden we couldn't ping OR access the web interface. further troupleshooting on this has showed we ARE ABLE to ping the BMC IP during POST. The ping and http access stops as soon as proxmox start CMAN process. By that i mean it writes Starting CMAN.... OKHere is my cluster conf:root@proxmox00:~# cat /etc/pve/cluster.conf
Code: 
  root@proxmox00:~#
When we try
Code: 
fence_node proxmox00 -vv
it fails with:
Code: 
root@proxmox00:~# fence_node proxmox01 -vvfence proxmox01 dev 0.0 agent fence_ipmilan result: error from agentagent args: nodename=proxmox01 agent=fence_ipmilan ipaddr=10.10.99.21 lanplus=1 auth=password login=admin passwd=XXXXX power_wait=5 method=cyclefence proxmox01 failed
using
Code: 
fence_ipmilan
fails with
Code: 
root@proxmox00:~# fence_ipmilan -l admin -p XXXXXXXX -P -a 10.10.99.32 -T 4 -o off -vPowering off machine @ IPMI:10.10.99.32...Spawning: '/usr/bin/ipmitool -I lanplus -H '10.10.99.32' -U 'admin' -P '[set]' -v chassis power status'...ipmilan: Failed to connect after 20 secondsFailedroot@proxmox00:~#
but the fence commands are being issued while the BMC IP isn't responding so it COULD be that it fails because there is no connection to the BMC controller in gennerel.. Still i'm confused and not sure if this is right cause my VMs do migrate when i do a reboot of the proxmox node which should indicate that fencing is working... How did we set this up...Well you got the cluster conf alleady.. All nodes are fully upgraded. Ofcause we don't get the stable updates since the servers are not licensed yetipmitool is installed on all serversredhat-cluster-pve has FENCE_JOIN set to yesfence_tool join has been run on all serversfence_tool ls shows:
Code: 
root@proxmox00:~# fence_tool lsfence domainmember count  3victim count  0victim now    0master nodeid 1wait state    nonemembers       1 2 3
ANY input would be GREATLY appriciated we have been working on this for perhaps a week now... THANKSCasperEDIT:The servers are also fully firmware updated.. if you want to give the servers a look one of them has the following serial number CZJ2120JSW
 
Last edited:
From what I see you have put the proxmox IP addresses on the fencedevice and it should be the IP address of the iLO for IPMI.
( this is also obvious in your example of trying to fence manually and you have the IP address of pve, but when you use fence_ipmilan
you use the correct IP )
However this doesn't explain the fact that you cannot ping the iLO ip addresses.. try to debug the situation using brctl show mac on the
pve bridge, arp for each IP to see if it is correctly matched. But maybe first post /etc/network/interfaces from the host so we can understand
better your IP setup.
 
thheo said:
From what I see you have put the proxmox IP addresses on the fencedevice and it should be the IP address of the iLO for IPMI.
( this is also obvious in your example of trying to fence manually and you have the IP address of pve, but when you use fence_ipmilan
you use the correct IP )
However this doesn't explain the fact that you cannot ping the iLO ip addresses.. try to debug the situation using brctl show mac on the
pve bridge, arp for each IP to see if it is correctly matched. But maybe first post /etc/network/interfaces from the host so we can understand
better your IP setup.
Click to expand...

Hi Thheo and thanks for answering so quick...

first thing first..

this is interface output on node 0 aka proxmox00 or 10.10.99.20

Code: 
root@proxmox00:~# cat /etc/network/interfaces
# network interface settings
auto lo
iface lo inet loopback


iface eth0 inet manual


iface eth1 inet manual


iface eth2 inet manual


iface eth3 inet manual


auto vmbr0
iface vmbr0 inet static
        address  10.10.99.20
        netmask  255.255.255.0
        gateway  10.10.99.1
        bridge_ports eth0 eth1 eth2 eth3
        bridge_stp off
        bridge_fd 0

as for the time being i have only one vmbr01 so here is the brctl output you asked for:

Code: 
root@proxmox00:~# brctl show mac vmbr0
bridge name     bridge id               STP enabled     interfaces
mac             can't get info No such device
vmbr0           8000.80c16e648d04       no              eth0
                                                        eth1
                                                        eth2
                                                        eth3
root@proxmox00:~#

ooops about the fencing ips in the cluster.conf.. though we have tried with both the prox ip, bmc ip and even hostnames... but i will change them to the relevant ip .30 .31 and .32 and do a test just to be sure...

that being said im really alarmed that i can see my BMC ip stops responding to ping when i get to the part in the proxmox boot op where CMAN is started.. i can't stop but think if its proxmox or debian that makes the BMC stop responding..??

Again thanks for the quick response,..

Casper

EDIT:

in regards to the network configuration. eth0-3 are a member of vmbr01 but there are only cables in eth 0 and 1 NOT 2 and 3.. its configured with LACP in prox 802.3ad.. i have NOT configred the corrosponding switch ports with LACP...

also take note that BMC has worked before by that i mean we have been able to ping bmc ip after prox startup before.. this happens out of no where that we suddenly can't ping it after prox starts CMAN
 
Last edited:
So i corrected to cluster.conf with the correct bmc ips..


cluster.conf now looks like this:

Code: 
root@proxmox00:~# cat /etc/pve/cluster.conf
<?xml version="1.0"?>
<cluster config_version="23" name="DingITCluster">
  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>
  <fencedevices>
    <fencedevice agent="fence_ipmilan" ipaddr="10.10.99.30" lanplus="1" auth="password" login="admin" name="ipmi1" passwd="XXXXXXX" power_wait="5" method="cycle" />
    <fencedevice agent="fence_ipmilan" ipaddr="10.10.99.31" lanplus="1" auth="password" login="admin" name="ipmi2" passwd="XXXXXXX" power_wait="5" method="cycle"/>
    <fencedevice agent="fence_ipmilan" ipaddr="10.10.99.32" lanplus="1" auth="password" login="admin" name="ipmi3" passwd="XXXXXXX" power_wait="5" method="cycle"/>
  </fencedevices>
  <clusternodes>
    <clusternode name="proxmox00" nodeid="1" votes="1">
      <fence>
        <method name="1">
          <device name="ipmi1"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="proxmox01" nodeid="2" votes="1">
      <fence>
        <method name="1">
          <device name="ipmi2"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="proxmox02" nodeid="3" votes="1">
      <fence>
        <method name="1">
          <device name="ipmi3"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <rm>
    <pvevm autostart="1" vmid="101"/>
    <pvevm autostart="1" vmid="102"/>
    <pvevm autostart="1" vmid="103"/>
    <pvevm autostart="1" vmid="104"/>
    <pvevm autostart="1" vmid="105"/>
    <pvevm autostart="1" vmid="107"/>
    <pvevm autostart="1" vmid="108"/>
    <pvevm autostart="1" vmid="109"/>
    <pvevm autostart="1" vmid="110"/>
    <pvevm autostart="1" vmid="111"/>
  </rm>
</cluster>
root@proxmox00:~#

Than i rebooted all servers and pinged the bmc ips of all servers... same error.. can't ping when prox starts...
than i did the fence_node along with fence_ipmilan and this is it the output

Code: 
root@proxmox00:~# fence_node proxmox02 -vv
fence proxmox02 dev 0.0 agent fence_ipmilan result: error from agent
agent args: nodename=proxmox02 agent=fence_ipmilan ipaddr=10.10.99.32 lanplus=1 auth=password login=admin passwd=XXXXXXX power_wait=5 method=cycle
fence proxmox02 failed

root@proxmox00:~# fence_ipmilan -l admin -p XXXXXXX -P -a 10.10.99.32 -T 4 -o off -v
Powering off machine @ IPMI:10.10.99.32...Spawning: '/usr/bin/ipmitool -I lanplus -H '10.10.99.32' -U 'admin' -P '[set]' -v chassis power status'...
ipmilan: Failed to connect after 20 seconds
Failed

Hmm this may be interesting.. I figured i would do a fence_tool ls just to see what was going on and i got this

Code: 
root@proxmox00:~# fence_tool ls
fence domain
member count  2
victim count  0
victim now    0
master nodeid 3
wait state    none
members       1 3

Now a simple fence_join could perhaps solve this but I think i would like to know what you guys thing before i go and do anything else...
 
Last edited:
ipmi fencing only works if the node has power, so such setup will never work if you loose power on a node.
 
Yes, ipmi for fencing will require UPS for the nodes. In general it is recommended to base fencing on external components otherwise you are fucked in case of power failure onless nodes are powered over UPS.
 
@Dietmar

not even if i used a external device as mir suggest?

@mir
wouldn't this external device not be a single point of failure against a power failure?

my problem is if I used Vmware or Hyper-v this would not be a problem... actually im not sure on hyper-v but i vmware has to have this feature... I saw proxmox as a even product with both of them when it comes to fail over and redudancy.. what do you think?

EDIT: Mir do you have some good suggestion for cheap external devices for fencing? that perhaps are also easy to setup with proxmox?

THANKS

Casper
 
offerlam said:
@mir
wouldn't this external device not be a single point of failure against a power failure?
Click to expand...
In the end there will always be a single point of failure.

offerlam said:
my problem is if I used Vmware or Hyper-v this would not be a problem... actually im not sure on hyper-v but i vmware has to have this feature... I saw proxmox as a even product with both of them when it comes to fail over and redudancy.. what do you think?

EDIT: Mir do you have some good suggestion for cheap external devices for fencing? that perhaps are also easy to setup with proxmox?
Click to expand...
As Dietmar wrote you could purchase a UPS with support for fencing by shutting down individual nodes. This give you the advantage of eliminating lost updates due to power failures as well as support fencing. Eg. one of these or similar: http://www.ebay.com/itm/APC-AP7941-..._Power_Distribution_Units&hash=item58a5561083

Another solution could be to use a managed switch but for a production environment I would suggest to purchase a UPS.
 
offerlam said:
Hi Dietmar,

would any UPS work? like i found this one: APC Smart-UPS SC 420VA (Grå) (420 VA) the only reason i would buy the ups was for the fencing functionality... its not gonna serve as a ups..

Thanks!

Casper
Click to expand...

You already have the fencing capability through iLO you shouldn't buy a UPS if you don't need one ( like for example having the power backed-up by UPS and generator in a datacenter ).
I would focus only on fencing functionality at first just try to send commands manually ( use: http://pve.proxmox.com/wiki/Fencing as a base info ). For example in my case with iDrac7 (Dell)
I had to create a different user in IPMI setup of iDrac ( I was not able to login with admin, although via web it was working ) just for fencing..
Example of ipmitool usage: ipmitool -H 192.168.5.1 -U root -P whatever chassis power cycle
But all these after you resolve your IP connectivity problems; I didn't understand the LACP you mentioned... I don't see any bonding in your network config.. maybe you are mixing bridging with
bonding?
 
Last edited:
"You already have the fencing capability through iLO you shouldn't buy a UPS if you don't need one"
How will you fence a node which have no power?
 
That's a node already "fenced". However you should read again the first post. And to reply to you in the same idea: Why would you fence a node with no-power?
Quote: "We are trying to setup fecing and its somewhat working." ( I copy-pasted along with the typo )
But then he confused HA functionality with fencing by the fact that he powered-down a node and wanted the VMs migrated ( or at least this is what I understood ).
So I was debating the fencing item.. he actually mentioned: "the only reason i would buy the ups was for the fencing functionality... its not gonna serve as a ups.."
So it's seem there is no power problem.. he was just trying different scenarios.
You'd activate HA to resume VMs as soon as possible, but you need fencing to make sure they don't overlap in case of some split-brain situation ( inside the cluster )

 
Last edited:
Coming back to the issue of HA not migrating the VMs to the other hosts, you should check in /var/log/cluster/rgmanager.log
and the other logs maybe to see hints of potential problems..
Is the shared storage configured properly?
 
You missed a crucial peace of information "Still i'm confused and not sure if this is right cause my VMs do migrate when i do a reboot of the proxmox node which should indicate that fencing is working". This indicates that HA and fencing is not working since the online nodes did not forcefully start VM's from the failing node since the were believing it was still online. When the failing node comes online again it knows it was fenced and therefore it migrated the VM's of to some other node. This is potentially a dangerous situation because what if the other nodes already had these VM's up and running?

This is why fencing must be done by some external third party which stays online when nodes are failing - this third party is the only one with the complete picture of the current status. In case this external party also goes down it will ensure the situation stays sound since this party is the only one which can bring fenced nodes back into the cluster.
 
Fencing a node is decided by the cluster system as a total through "voting". He has 3 nodes so that should do it just fine. iLO is like an external system so you can count on it
to reboot your node in such case. If iLO has no power then the node has no power, so it is fenced.. I just wanted to highlight that based on what he said, he doesn't need
another hardware to do the fencing since he already has the means to do it.
The "VMs do migrate when i do a reboot of the proxmox node which should indicate that fencing is working" part of course it's not right. This proves that the cluster does its job
and restarts the VM ( although I don't understand why it didn't do this when he forced a power-off ), but it doesn't prove that fencing works.. because as I noticed earlier the
config was wrong, plus he cannot manually use the fence tool. But it seems that he has all that is needed to make this work ( however I still don't have the shared storage
information.. but hopefully is the right one ).

 
Hi guys..

First of all a BIG BIG thank you for helping out!

this thread just exploded and i'm not really sure I got all of what Mir and Thheo was discussing...

but I can try to clear stuff up...

If i do a shutdown -r now on a node that HAS vms.. these VMs are migrated to other nodes fine - is this HA with Fencing?

If i unplug the power cords nothing happens. My Vms stays on the node and turns off - is it HA/Fencing that should move them to another node? and is that all that fencing does?

how would fencing react if i pulled all my network kables instead of power cords?

Thheo is right in that the only reason i would get a UPS was to have a external fencing device. He is also right that these servers are hosted in a rented rack in a data center so i really don't need a ups since they have alle the juicy stuff on that part.

@Mir
do you suggest the ups for production enviroments because you figure a UPS would be needed anyway? or because they make for better fencing devices? Im asking because as i said above these servers are in a rented rack in a datacenter and so i don't really need a UPS and so a PDU looks more my thing in my scenario.. also they have installed a pdu in my rack maybe they will let me use it i don't know...

how do you get this to work with a PDU? would you buy a PDU with IPMI support? or Fencing support or?

As for the question about storage:
I have two Synology DS1513+ one for backup and one for ISO/VM storage and so on. they are connected with LACP and it is working. the protocol used to connected the sharse in proxmox is NFS..

thheo good you mentioned you had to make a different user, cause i too am using the built in admin user i will set this up also...

But i think we are forgetting one important problem here... WHY does my BMC ip stop responding the second proxmox start CMAN during boot up?
cause i think i can get my fencing up and running if this didn't happen. I thought it was a misconfiguration in the config that killed it but it seems not to be.. so what is it? any ideas how to troupleshoot this?
I will check RGmanager.log to see if can cast a light on things...

Thanks to all 3 of you for your help so fare!!

Casper
 
1. Fencing means isolating a node that maybe is working on its own outside the cluster control. You want this to prevent a case that such a node is writing to the
shared storage and probably overlapping with a node that is entitled ( by the cluster ) to write in that area. So when a node is outside cluster control for
whatever reason you trigger a fencing scenario with the help of a device that can take actions like power cycle on the box and hopefully it comes back
online properly to the cluster system
2. The fact that VMs get moved (actually restarted ) to the other nodes on reboot it is called HA ( fencing is issued as a trigger because of your config and it's mandatory for the
reasons explained at 1. , but in your case it doesn't actually do anything ) and this is handled by rgmanager, because you have a <rm> section in your cluster.conf
which contains the proper settings for the VMs. You can check what's going on in /var/log/cluster/rgmanager.log
3. Of course you have to take care of the network connectivity issue you are facing before trying ipmitool to get fencing working:
Why do you bridge eth0 and eth1? Since you have STP off on the bridge, is it maybe a possibility that you are creating a loop with eth0, eth1 and an
external switch? Better explain to what eth0 is connected to, and to what eth1 is? And what exactly do you want to use them for? Also you mentioned
that you chose BMC to use all the NICs in a shared mode with the server so this one doesn't sound right with the fact that you are bridging all the interfaces, better
straighten your network setup first, it might just solve the reachability issue with the BMC.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back