vgs and lvs hang without output

[tux]

I have big problem to work with clvm. I cannot run clvmd correct on 2 different testing clusters (enterprise and comodity hardware). lvcreate dont work and vgs and lvs hang. The clvm service is runnning but it hang on restart (At Deactivating VG)

root@xxxxx:~# /etc/init.d/clvm restart
Deactivating VG ::

root@xxxx:~# /etc/init.d/cman restart
Stopping cluster: 
   Leaving fence domain... found dlm lockspace /sys/kernel/dlm/clvmd
fence_tool: cannot leave due to active systems
[FAILED]

It is impossible to create snapshots. This is a very important feature - i hope i can deactivate clvm and work for local storage without it. Any ideas? At first i think it is the firewall ... or may the switch. Where i can see the logs for this service?

 

[dietmar]

When you run a cluster you need to make sure your you have working fencing devices (already configured?).

Also, try to never loose quorum - else you get such hangs.

 

[tux]

I don't configure any fenced devices. An the cluster status is ok.

root@xxxx:~# pvecm n
Node  Sts   Inc   Joined               Name
   1   M  85732   2011-10-10 20:53:38  xxx
   2   M  85736   2011-10-10 20:53:38  yyy

root@xxx:~# clustat
Cluster Status for datacenter-1-de @ Sat Oct 15 13:11:10 2011
Member Status: Quorate

 Member Name                             ID   Status
 ------ ----                             ---- ------
 xxx                                 1 Online, Local
 yyy                                 2 Online

root@xxx:~# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster name="datacenter-1-de" config_version="4">

  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>

  <clusternodes>

  <clusternode name="yyy" votes="1" nodeid="2"/><clusternode name="xxx" votes="1" nodeid="1"/></clusternodes>

</cluster>

What about the logs? I cannot find any thing about this problem in the logs.

BTW. The logs a full with this useless info (every 10 seconds generated- is it really needed - may be you do it a otherlog then syslog)

Oct 15 07:48:35 xxx pvestatd[1889]: status update finished (0.015 seconds)
Oct 15 07:48:45 xxx pvestatd[1889]: start status update

The logs in /var/log/cluster/ are useless for this problem.

 

[tom]

if you have a two node cluster you easily loose quorum. if you lost quorum, basically nothing works, the pmxcfs gets readonly (/etc/pve/...). so at least 3 nodes are nice. or use a quorum disk or configure cman as a two node cluster.

for the cluster status, see 'pvecm -s'

if you run service on top of cman, you cannot restart cman. in your case its clvm. stop clvm and you can restart cman. clustering is not that easy to understand but it make sense to try to understand the basics.

 

[tux]

I check out the firewall and network again - looks like communication problem.

 

[dietmar]

Also check if fencing is in action:

# fence_tool ls

if so, you need to manually acknowledge (because you have not configured fencing devices).

 

[tux]

this firewall rules dont work with clvm - i hope somebody working on a iptables rule for proxmox 2.0 and this is usefull. if i start this in rc.local i dont work - if i start later over terminal it works fine.

/sbin/iptables -A INPUT -i lo -j ACCEPT
    /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    /sbin/iptables -A OUTPUT -o lo -j ACCEPT
    /sbin/iptables -A OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT    

    /sbin/iptables -A INPUT -m addrtype --dst-type MULTICAST -j ACCEPT 

    /sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT
    /sbin/iptables -A INPUT -p tcp --dport 83 -j ACCEPT
    /sbin/iptables -A INPUT -p tcp --dport 80 -j ACCEPT
    /sbin/iptables -A INPUT -p tcp --dport 443 -j ACCEPT
    /sbin/iptables -A INPUT -p tcp --dport 8006 -j ACCEPT
    /sbin/iptables -A INPUT -p udp --dport 5400:5450 -j ACCEPT
    /sbin/iptables -A INPUT -p tcp --dport 5900:6100 -j ACCEPT
    
    /sbin/iptables -I INPUT -p tcp --sport 41966:41969 -j ACCEPT
    /sbin/iptables -I INPUT -p tcp --sport 50006:50009 -j ACCEPT
    /sbin/iptables -I INPUT -p tcp --sport 21064 -j ACCEPT

    /sbin/iptables -I OUTPUT -p tcp --dport 41966:41969 -j ACCEPT
    /sbin/iptables -I OUTPUT -p tcp --dport 50006:50009 -j ACCEPT
    /sbin/iptables -I OUTPUT -p tcp --dport 21064 -j ACCEPT 
    
    /sbin/iptables -I INPUT -p udp --sport 50007 -j ACCEPT
    /sbin/iptables -I INPUT -p udp --sport 5405 -j ACCEPT

    /sbin/iptables -I OUTPUT -p udp --dport 5405 -j ACCEPT
    /sbin/iptables -I OUTPUT -p udp --dport 50007 -j ACCEPT

 

[tux]

Sorry but at my test i see if some node is down - vgs and lvs just hang ... this is not good. If one server in cluster crash, it is impossible to create snapshot or create dumps on all servers in the cluster. Don't think this a good solution.

BTW. clvm works now without firewall.

 

[tom]

Sorry but at my test i see if some node is down - vgs and lvs just hang ... this is not good. If one server in cluster crash, it is impossible to create snapshot or create dumps on all servers in the cluster. Don't think this a good solution.

BTW. clvm works now without firewall.

without configuration a two node cluster works like you described (or better, does not work). you need quorum in all cases.

 

[tux]

without configuration a two node cluster works like you described (or better, does not work). you need quorum in all cases.

Do you mean I just need more server? May be 3 or 5? (Please say yes. Need a reason to buy more hardware )

 

[dietmar]

Do you mean I just need more server? May be 3 or 5?

2 Node cluster work well when you have shared storage, so that you can use a quorum disk (qdisk).

Else you need more than 2 nodes (if you want to tolerate one node failure).