fence_ipmilan oddities

GT1

Active Member
Aug 10, 2011
48
0
26
so, i'm testing fencing. i have a supermicro board with dedicated ipmi bmc. (bios & ipmi firmware are on the latest revision available)

pve2.0 on latest version through full-upgrade, fence-agents-pve & ipmitool installed.

powering off/on a server works with ipmitool and this command (as example):
Code:
ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P 'ipmipassword' -v chassis power off
takes around 3-4 seconds after a
Code:
ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername -P 'ipmipassword' -v chassis power status
replies with Chassis Power is off

good so far, now with the fence_ipmilan wrapper:
Code:
fence_ipmilan -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -T 4 -o off
this fails :( ipmilan: Power still on

however when adding a -v:
Code:
fence_ipmilan -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -T 4 -o off -v

Powering off machine @ IPMI:192.168.190.83...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power off'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -P '[set]' -v chassis power status'...
ipmilan: Power still on
Failed

uhm ? right, it spawns ipmitool in rapid succession with the exact commandline i've wrote above and already confirmed to be working fine. to me it looks like it either ignores the -T parameter or the spawn spamming keeps resetting the command queue on the bmc which seem to only execute the last one send.

is there a way to increase the delay between the spawning of ipmitool processes (for what the -T switch was supposed to be there ? guessing here) ?

or any other suggestions ?

thanks.
 
Does it help if you add '-A password' :

# fence_ipmilan -A password -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -o off -v
 
no, the -A password does not have any effect.

it seems the positioning of the attributes does tho, when i place the -T 5 (for a 5 seconds delay) at first, powering up with on works:

Code:
:~#fence_ipmilan -T 5 -A password -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -o on -v
Powering on machine @ IPMI:192.168.190.83...
(1) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
(2) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power on'...
(3) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
Done
the delay between (2) & (3) is exactly 5 seconds.

let's retry that with off to power down:

Code:
:~#fence_ipmilan -T 5 -A password -l ipmiusername -p ipmipassword -P -a 192.168.190.83 -o off -v
Powering off machine @ IPMI:192.168.190.83...
(1) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
(2) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power off'...
(3) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
(4) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power off'...
(5) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
...
(15) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power off'...
(16) Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.190.83' -U 'ipmiusername' -A 'password' -P '[set]' -v chassis power status'...
ipmilan: Power still on
Failed
the delay between (2) & (4) is still 2 seconds, wich is the default according to the manpage. it ignores the -T 5 :(
 
Could you tell me what did you have change in code?
line 476: sleep(2);
replace with: sleep(ipmi->i_power_wait);

could you provide a cluster.conf ?
Code:
<?xml version="1.0"?>
<cluster name="space" config_version="6">
    <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
    </cman>
    <fencedevices>
        <fencedevice agent="fence_ipmilan" name="space1ipmi" lanplus="1" ipaddr="192.168.190.81" login="ipmiusername" passwd="ipmipassword" power_wait="5"/>
        <fencedevice agent="fence_ipmilan" name="space2ipmi" lanplus="1" ipaddr="192.168.190.83" login="ipmiusername" passwd="ipmipassword" power_wait="5"/>
        <fencedevice agent="fence_ipmilan" name="space3ipmi" lanplus="1" ipaddr="192.168.190.85" login="ipmiusername" passwd="ipmipassword" power_wait="5"/>
    </fencedevices>
    <clusternodes>
        <clusternode name="space1" votes="1" nodeid="1">
            <fence>
                <method name="1">
                    <device name="space1ipmi"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="space2" votes="1" nodeid="2">
            <fence>
                <method name="1">
                    <device name="space2ipmi"/>
                </method>
            </fence>
        </clusternode>
        <clusternode name="space3" votes="1" nodeid="3">
            <fence>
                <method name="1">
                    <device name="space3ipmi"/>
                </method>
            </fence>
        </clusternode>
    </clusternodes>
    <rm>
        <service autostart="1" exclusive="0" name="ha_test_ip" recovery="relocate">
            <ip address="192.168.190.127"/>
        </service>
    </rm>
</cluster>
keep in mind you'll have to replace the clustername, the fencedevices ip/name/login/password and the clusternode names to match your setup. but that goes without saying i suppose ;)
 
Last edited:
i have strange behaviour.

follwing works on cli
ipmitool -I lanplus -H '10.162.192.6' -U 'admin' -P 'admin' -v chassis power on
ipmitool -I lanplus -H '10.162.192.6' -U 'admin' -A 'admin' -P 'admin' -v chassis power status
fence_ipmilan -l admin -p admin -P -a 10.162.192.6 -T 4 -o off

setup 2 nodes with drbd and fencing
Code:
<?xml version="1.0"?>
<cluster config_version="31" name="master">
  <cman keyfile="/var/lib/pve-cluster/corosync.authkey" two_node="1" expected_votes="1"/>
  <fencedevices>
    <fencedevice agent="fence_ipmilan" name="masteripmi"  ipaddr="10.162.192.5" login="admin" passwd="admin" power_wait="5"/>
    <fencedevice agent="fence_ipmilan" name="slaveipmi"  ipaddr="10.162.192.6" login="admin" passwd="admin" power_wait="5"/>
  </fencedevices>
  <clusternodes>
    <clusternode name="master" nodeid="1" votes="1">
        <fence>
          <method name="1">
            <device name="masteripmi" nodename="master" action="reboot"/>
          </method>
        </fence>
      </clusternode>
    <clusternode name="slave" nodeid="2" votes="1">
        <fence>
          <method name="1">
            <device name="slaveipmi" nodename="slave" action="reboot"/>
          </method>
        </fence>
      </clusternode>
  </clusternodes>
  <rm>
    <pvevm autostart="1" vmid="100"/>
  </rm>
</cluster>

All seems ok... so I think.

Here is the strange behaviour.
A)
When im stopping RGManager on master node, bad name for setup, i know... where the VM is running. Nothing happens, the VM is still running.
If i start the rgmanager the vm is shutting down on master

B)
When im stopping RGManager on master node, where the VM is running. Nothing happens, the VM is still running.
Stopping RGManager on SLAVE too, the vm is starting on slave. in proxmox webfronted you can see the vm is switching from MASTER to SLAVE.
BUT it is still running on master.
So it is 2x on the quorum running.

C)
When i restart the machine where the vm is running. the vm is "shutting down" on master.

I think im doing something wrong :/ but dont know for sure.




seems i have solved...
aptitude update && aptitude full-upgrade
<fencedevice agent="fence_ipmilan" ipaddr="10.162.192.5" lanplus="1" login="admin" name="masteripmi" passwd="admin" power_wait="5"/>
 
Last edited:
Hi GT1,
with which board you're working on?
I'm testing currently X7DBN + AOC SIMLC not get to run to the ipmitool!

Code:
# ipmitool -I lanplus -A PASSWORD -H '192.168.1.151' -U 'ADMIN' -P 'ADMIN' -vv chassis power status
IPMI LAN host IPMI LAN host 192.168.1.151 port 623

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 

Get Auth Capabilities error

>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x8e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 


>> Sending IPMI command payload
>>    netfn   : 0x06
>>    command : 0x38
>>    data    : 0x0e 0x04 

Get Auth Capabilities error
Error issuing Get Channel Authentication Capabilies request
Error: Unable to establish IPMI v2 / RMCP+ session
Unable to get Chassis Power Status

ping intern-Host:
Code:
# ping 192.168.1.151
PING 192.168.1.151 (192.168.1.151) 56(84) bytes of data.
From 192.168.1.115 icmp_seq=2 Destination Host Unreachable
From 192.168.1.115 icmp_seq=3 Destination Host Unreachable

ping extern-Host:
Code:
$ ping 192.168.1.151
PING 192.168.1.151 (192.168.1.151): 56 data bytes
64 bytes from 192.168.1.151: icmp_seq=0 ttl=64 time=4.671 ms
64 bytes from 192.168.1.151: icmp_seq=1 ttl=64 time=3.920 ms

Does anyone have any help for me!
 
OK
I have already installed a 2nd test server, and from there it seems to work.

Code:
# ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -P 'ADMIN' -v chassis power off
Chassis Power Control: Down/Off


# ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -P 'ADMIN' -v chassis power status
Chassis Power is off


# fence_ipmilan -T 5 -A password -l ADMIN -p ADMIN -P -a 192.168.1.151 -o on -v
Powering on machine @ IPMI:192.168.1.151...Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -A 'password' -P '[set]' -v chassis power status'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -A 'password' -P '[set]' -v chassis power on'...
Spawning: '/usr/bin/ipmitool -I lanplus -H '192.168.1.151' -U 'ADMIN' -A 'password' -P '[set]' -v chassis power status'...
Done

is it normal that your own server is not reachable via ipmitool on eth1??

I go into the data center and most times all at the right servers.
 
Last edited:
OK
I have already installed a 2nd test server, and from there it seems to work.

is it normal that your own server is not reachable via ipmitool on eth1??

I go into the data center and most times all at the right servers.


If your BMC and eg. eth0 share the same ethernetport this is quite normal, you could usually work around by using a vlan tag for the ipmi-interface or simply connect another ethernetinface to your local network (eth1) and set ipmi-IP-address to something that shares the subnet with your eth1 config (and make sure it doesn't overlap with your IP-config on eth0).

Finally as fencing usually will switch off the "other" server, this shouldn't be a big issue, you should be able to reach the other hosts BMC anyways, even if you can't talk to your local BMC via IP. In case you have to talk to your local BMC you can always do this via kcs or on some servers the BMC also offers a dedicated usb-ethernet-connection to your local system and other fun-stuff :)

regards
hk
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!