fence_ipmilan oddities

GT1 · Jan 31, 2012

so, i'm testing fencing. i have a supermicro board with dedicated ipmi bmc. (bios & ipmi firmware are on the latest revision available)

pve2.0 on latest version through full-upgrade, fence-agents-pve & ipmitool installed.

powering off/on a server works with ipmitool and this command (as example):

Code:

ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P 'ipmipassword' -v chassis power off

takes around 3-4 seconds after a

Code:

ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername -P 'ipmipassword' -v chassis power status

replies with Chassis Power is off

good so far, now with the fence_ipmilan wrapper:

Code:

fence_ipmilan -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -T 4 -o off

this fails

ipmilan: Power still on

however when adding a -v:

Code:

fence_ipmilan -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -T 4 -o off -v

Powering off machine @ IPMI:192.168.190.83...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
ipmilan: Power still on
Failed

uhm ? right, it spawns ipmitool in rapid succession with the exact commandline i've wrote above and already confirmed to be working fine. to me it looks like it either ignores the -T parameter or the spawn spamming keeps resetting the command queue on the bmc which seem to only execute the last one send.

is there a way to increase the delay between the spawning of ipmitool processes (for what the -T switch was supposed to be there ? guessing here) ?

or any other suggestions ?

thanks.

dietmar · Feb 1, 2012

Does it help if you add '-A password' :

# fence_ipmilan -A password -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -o off -v

GT1 · Feb 1, 2012

no, the -A password does not have any effect.

it seems the positioning of the attributes does tho, when i place the -T 5 (for a 5 seconds delay) at first, powering up with on works:

Code:

:~#fence_ipmilan -T 5 -A password -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -o on -v
Powering on machine @ IPMI:192.168.190.83...
(1) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
(2) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power on'...
(3) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
Done

the delay between (2) & (3) is exactly 5 seconds.

let's retry that with off to power down:

Code:

:~#fence_ipmilan -T 5 -A password -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -o off -v
Powering off machine @ IPMI:192.168.190.83...
(1) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
(2) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power off'...
(3) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
(4) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power off'...
(5) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
...
(15) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power off'...
(16) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
ipmilan: Power still on
Failed

the delay between (2) & (4) is still 2 seconds, wich is the default according to the manpage. it ignores the -T 5

GT1 · Feb 1, 2012

found it:
http://git.fedorahosted.org/git/?p=fence-agents.git;a=blob;f=fence/agents/ipmilan/ipmilan.c

for the function ipmi_on, the delay is used (see line 518)
for the function ipmi_off, it's hardcoded to 2 seconds (see line 476)

going to change & compile that tomorrow and see how it goes.

GT1 · Feb 2, 2012

yup, works as expected now. yay

modified binary attached if someone else got problems with this:
View attachment fence_ipmilan.zip

spirit · Feb 3, 2012

Hi,
Could you tell me what did you have change in code?

i'll try to add your change in the proxmox package.

could you provide a cluster.conf ? So I'll add it in the wiki.

GT1 · Feb 3, 2012

spirit said:
Could you tell me what did you have change in code?

line 476: sleep(2);
replace with: sleep(ipmi->i_power_wait);

spirit said:
could you provide a cluster.conf ?

Code:

<?xml version="1.0"?>
<cluster name="space" config_version="6">
    <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
    </cman>
    <fencedevices>
        <fencedevice agent="fence_ipmilan" name="space1ipmi" lanplus="1" ipaddr="192.168.190.81" login="ipmiusername" passwd="ipmipassword" power_wait="5"/>
        <fencedevice agent="fence_ipmilan" name="space2ipmi" lanplus="1" ipaddr="192.168.190.83" login="ipmiusername" passwd="ipmipassword" power_wait="5"/>
        <fencedevice agent="fence_ipmilan" name="space3ipmi" lanplus="1" ipaddr="192.168.190.85" login="ipmiusername" passwd="ipmipassword" power_wait="5"/>
    </fencedevices>
    <clusternodes>
        <clusternode name="space1" votes="1" nodeid="1">
            <fence>
                <method name="1">
                    <device name="space1ipmi"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="space2" votes="1" nodeid="2">
            <fence>
                <method name="1">
                    <device name="space2ipmi"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="space3" votes="1" nodeid="3">
            <fence>
                <method name="1">
                    <device name="space3ipmi"/>
                </method>
            </fence>
        </clusternode>
    </clusternodes>
    <rm>
        <service autostart="1" exclusive="0" name="ha_test_ip" recovery="relocate">
            <ip address="192.168.190.127"/>
        </service>
    </rm>
</cluster>

keep in mind you'll have to replace the clustername, the fencedevices ip/name/login/password and the clusternode names to match your setup. but that goes without saying i suppose

spirit · Feb 3, 2012

Ok, thanks !

I find the original commit which add the power_wait
http://git.fedorahosted.org/git/?p=...ff;h=7d53eb8ab06a8713d2b52500da741b6170fbfc91

I'll try to contact the redhat author to get it upstream.

kprkpr · Mar 25, 2012

i have strange behaviour.

follwing works on cli
ipmitool -I lanplus -H '10.162.192.6' -U 'admin' -P 'admin' -v chassis power on
ipmitool -I lanplus -H '10.162.192.6' -U 'admin' -A 'admin' -P 'admin' -v chassis power status
fence_ipmilan -l admin -p admin -P -a 10.162.192.6 -T 4 -o off

setup 2 nodes with drbd and fencing

Code:

<?xml version="1.0"?>
<cluster config_version="31" name="master">
  <cman keyfile="/var/lib/pve-cluster/corosync.authkey" two_node="1" expected_votes="1"/>
  <fencedevices>
    <fencedevice agent="fence_ipmilan" name="masteripmi"  ipaddr="10.162.192.5" login="admin" passwd="admin" power_wait="5"/>
    <fencedevice agent="fence_ipmilan" name="slaveipmi"  ipaddr="10.162.192.6" login="admin" passwd="admin" power_wait="5"/>
  </fencedevices>
  <clusternodes>
    <clusternode name="master" nodeid="1" votes="1">
        <fence>
          <method name="1">
            <device name="masteripmi" nodename="master" action="reboot"/>
          </method>
        </fence>
      </clusternode>
    <clusternode name="slave" nodeid="2" votes="1">
        <fence>
          <method name="1">
            <device name="slaveipmi" nodename="slave" action="reboot"/>
          </method>
        </fence>
      </clusternode>
  </clusternodes>
  <rm>
    <pvevm autostart="1" vmid="100"/>
  </rm>
</cluster>

All seems ok... so I think.

Here is the strange behaviour.
A)
When im stopping RGManager on master node, bad name for setup, i know... where the VM is running. Nothing happens, the VM is still running.
If i start the rgmanager the vm is shutting down on master

B)
When im stopping RGManager on master node, where the VM is running. Nothing happens, the VM is still running.
Stopping RGManager on SLAVE too, the vm is starting on slave. in proxmox webfronted you can see the vm is switching from MASTER to SLAVE.
BUT it is still running on master.
So it is 2x on the quorum running.

C)
When i restart the machine where the vm is running. the vm is "shutting down" on master.

I think im doing something wrong :/ but dont know for sure.

seems i have solved...
aptitude update && aptitude full-upgrade
<fencedevice agent="fence_ipmilan" ipaddr="10.162.192.5" lanplus="1" login="admin" name="masteripmi" passwd="admin" power_wait="5"/>

loisl · Mar 25, 2012

Hi GT1,
with which board you're working on?
I'm testing currently X7DBN + AOC SIMLC not get to run to the ipmitool!

Code:

# ipmitool -I lanplus -A PASSWORD -H '192.168.1.151' -U 'ADMIN' -P 'ADMIN' -vv chassis power status
IPMI LAN host IPMI LAN host 192.168.1.151 port 623

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 

Get Auth Capabilities error

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 

Get Auth Capabilities error
Error issuing Get Channel Authentication Capabilies request
Error: Unable to establish IPMI v2 / RMCP+ session
Unable to get Chassis Power Status

ping intern-Host:

Code:

# ping 192.168.1.151
PING 192.168.1.151 (192.168.1.151) 56(84) bytes of data.
From 192.168.1.115 icmp_seq=2 Destination Host Unreachable
From 192.168.1.115 icmp_seq=3 Destination Host Unreachable

ping extern-Host:

Code:

$ ping 192.168.1.151
PING 192.168.1.151 (192.168.1.151): 56 data bytes
64 bytes from 192.168.1.151: icmp_seq=0 ttl=64 time=4.671 ms
64 bytes from 192.168.1.151: icmp_seq=1 ttl=64 time=3.920 ms

Does anyone have any help for me!

loisl · Mar 25, 2012

OK
I have already installed a 2nd test server, and from there it seems to work.

Code:

# ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -P 'ADMIN' -v chassis power off
Chassis Power Control: Down/Off


# ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -P 'ADMIN' -v chassis power status
Chassis Power is off


# fence_ipmilan -T 5 -A password -l ADMIN -p ADMIN -P -a 192.168.1.151 -o on -v
Powering on machine @ IPMI:192.168.1.151...Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -A 'password' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -A 'password' -P '[set]' -v chassis power on'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -A 'password' -P '[set]' -v chassis power status'...
Done

is it normal that your own server is not reachable via ipmitool on eth1??

I go into the data center and most times all at the right servers.

GT1 · Mar 25, 2012

loisl said:
Hi GT1,
with which board you're working on?

x9scl-f

hk@ · Mar 25, 2012

loisl said:
OK
I have already installed a 2nd test server, and from there it seems to work.

is it normal that your own server is not reachable via ipmitool on eth1??

I go into the data center and most times all at the right servers.

If your BMC and eg. eth0 share the same ethernetport this is quite normal, you could usually work around by using a vlan tag for the ipmi-interface or simply connect another ethernetinface to your local network (eth1) and set ipmi-IP-address to something that shares the subnet with your eth1 config (and make sure it doesn't overlap with your IP-config on eth0).

Finally as fencing usually will switch off the "other" server, this shouldn't be a big issue, you should be able to reach the other hosts BMC anyways, even if you can't talk to your local BMC via IP. In case you have to talk to your local BMC you can always do this via kcs or on some servers the BMC also offers a dedicated usb-ethernet-connection to your local system and other fun-stuff

regards
hk

spirit · Mar 28, 2012

Hi Guys,

could you try this package ?

http://odisoweb1.odiso.net/fence-agents-pve_3.1.7-2_amd64.deb

This is the coming soon next version of fence-agents-pve package (not yet release in proxmox repos).

this is correcting the power_wait bug and a log of other ipmi bugs.

Search

Search

fence_ipmilan oddities

GT1

Active Member

dietmar

Proxmox Staff Member

GT1

Active Member

GT1

Active Member

GT1

Active Member

spirit

Distinguished Member

GT1

Active Member

spirit

Distinguished Member

kprkpr

New Member

loisl

New Member

loisl

New Member

GT1

Active Member

hk@

Renowned Member

spirit

Distinguished Member