[SOLVED] Execute/run command on all cluster nodes

[Morten]

Hi all,

To whoever wants to run a command on all cluster nodes, here is a way...

On one node create the file /bin/cssh with the following content:

#!/bin/bash
if [ ! -n "$1" ]; then echo "Usage: $0 [command]"; else
for n in `xmllint --xpath //clusternode/@name /etc/pve/cluster.conf | sed 's/name="\|"//g'`; do
echo --- $n -----------------------------------------------------
ssh -T $n "$@"
done
fi

and make it executable with chmod +x /bin/cssh.
Now you should be able to run a command sequentially on all cluster nodes like

cssh "pveversion -v | grep kernel"
or
cssh df -h | egrep "(pve-|---)".
​

Copy cssh to all other cluster nodes with cssh scp nodeX:/bin/cssh /bin/.
During this process try to eliminate all time consuming authorization questions or key mismatches:
If you get prompted for password when scp'ing to node7, for instance, run ssh-copy-id node7.
If you get a key/fingerprint mismatche that you can explain, note the offending key file and number and then remove the key with something like sed -i 3d .ssh/known_hosts

Questions or improvements are more than welcome.

 

[mlanner]

Looks like a great solution. Personally I've been using Salt (saltstack.org) for similar tasks. That said, because Salt is a full remote execution environment (based on ZeroMQ) it can also do a lot of other stuff. Salt was just accepted into Debian too.

 

[Morten]

Good things comes to those who wait - and to those who creates the good things themselves
The Salt Stack seems like a very cool solution, but I can't get it to run so I can't test it. I downloaded the .deb file and this is what I got...

root@pizza:~# dpkg -i salt_0.9.5.pre-v0.9.4-10-g8182e48-1_all.deb 
Selecting previously deselected package salt.
(Reading database ... 30432 files and directories currently installed.)
Unpacking salt (from salt_0.9.5.pre-v0.9.4-10-g8182e48-1_all.deb) ...
dpkg: dependency problems prevent configuration of salt:
 salt depends on python-setuptools; however:
  Package python-setuptools is not installed.
 salt depends on python-yaml; however:
  Package python-yaml is not installed.
 salt depends on python-crypto; however:
  Package python-crypto is not installed.
 salt depends on python-m2crypto; however:
  Package python-m2crypto is not installed.
 salt depends on python-zmq (>= 2.1.9); however:
  Package python-zmq is not installed.
 salt depends on libzmq1 (>= 2.1.9); however:
  Package libzmq1 is not installed.
 salt depends on libzmq-dev (>= 2.1.9); however:
  Package libzmq-dev is not installed.
 salt depends on python-jinja2; however:
  Package python-jinja2 is not installed.
dpkg: error processing salt (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db ...
Errors were encountered while processing:
 salt
root@pizza:~# apt-get install python-setuptools python-yaml python-crypto python-m2crypto python-zmq libzmq1 libzmq-dev python-jinja2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package python-zmq is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libzmq1 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libzmq-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python-zmq' has no installation candidate
E: Package 'libzmq1' has no installation candidate
E: Package 'libzmq-dev' has no installation candidate

Talk about anticlimax

 

[mlanner]

Yeah, I think you need to install ZeroMQ first and maybe some of the Python modules. Take a look here:

http://salt.readthedocs.org/en/v0.9.8/topics/installation/debian.html

Also, it's not obvious from the page above, but Salt was just accepted into Debian, so there should be an up-to-date package in the Debian repos. I'll try to find the link to it and provide it here.

 

[mlanner]

Found it! Here you go:

http://packages.debian.org/search?searchon=names&keywords=Salt

 

[e100]

Lots of great info in this thread, hopefully someone finds this post useful too.

i've been using Chef ( http://www.opscode.com/chef/ ) to automate configuration/management of our physical and virtual servers.
With Chef you can use a search to run a command on all the nodes that are returned from the search results.
We name all of our Proxmox servers vmX.domain.com, so that will be our Chef search "fqdn:vm*.domain.com"
If I want to run the "uptime" command on all of our Proxmox nodes I can do this:

knife ssh "fqdn:vm*.domain.com" "uptime"

The results of the command look like this:

vm1.domain.com   23:08:04 up 34 days,  8:48,  1 user,  load average: 0.07, 0.08, 0.02
vm4.domain.com   23:08:04 up 27 days,  7:05,  1 user,  load average: 0.05, 0.04, 0.00
vm9.domain.com   23:08:04 up 36 days,  9:59,  1 user,  load average: 0.01, 0.02, 0.00
vm7.domain.com   23:08:04 up 34 days,  7:38,  1 user,  load average: 0.05, 0.07, 0.08
vm8.domain.com   23:08:04 up 34 days,  8:00,  1 user,  load average: 0.05, 0.01, 0.00
vm2.domain.com   23:08:04 up 34 days,  8:34,  1 user,  load average: 0.02, 0.03, 0.00
vm14.domain.com  23:08:04 up 32 days,  8:29,  1 user,  load average: 0.00, 0.00, 0.00
vm6.domain.com   23:08:04 up 27 days,  6:58,  1 user,  load average: 0.18, 0.21, 0.18
vm10.domain.com  23:08:04 up 36 days, 10:11,  1 user,  load average: 0.00, 0.01, 0.00
vm11.domain.com  23:08:04 up 36 days,  5:30,  1 user,  load average: 0.00, 0.00, 0.00
vm12.domain.com  23:08:04 up 36 days,  5:19,  1 user,  load average: 0.01, 0.06, 0.01
vm3.domain.com   23:08:04 up 27 days,  6:47,  1 user,  load average: 0.00, 0.02, 0.01
vm5.domain.com   23:08:04 up 27 days,  6:38,  1 user,  load average: 0.06, 0.13, 0.11
vm13.domain.com  23:08:04 up 32 days,  8:29,  1 user,  load average: 0.09, 0.27, 0.17

Note: I have previously added my public SSH key to the proxmox servers so I can log in from my workstation using my SSH key.

This is just one cool thing Chef can do, if you manage lots of servers I encourage you to take a look at Chef. http://www.opscode.com/chef/

 

[mlanner]

Yeah, I've read good things about Chef. I found Chef to have a somewhat higher learning curve than Salt. But that might just be me. For others interested, and to include "all" devops tools, Puppet will do most, if not all, that Chef and Salt does too.