proxmox-ve active/passive cluster

bread-baker

Member
Mar 6, 2010
432
0
16
hello
We've setup a pair of proxmox-ve servers into a high availability system using drbd and heartbeat. This uses drbd in Primary/Secondary mode.

We are running 2 open-vz and 2 lvm containers. We still may have some changes to make, but have done a lot of fail over tests and things are working well. We can unplug the primary server , and within a few minutes the second server turns into primary and brings up our 4 vm's.
We use Supermicro servers with 3-ware raid-10 .

see this as I got a lot of info from: http://wiki.openvz.org/HA_cluster_with_DRBD_and_Heartbeat


/etc/drbd.conf
Code:
global { usage-count yes; }
common { syncer { rate 100M; } }

resource r2 {
 protocol C;
        handlers {
                     # "pri-on-incon-degr": This handler is called if the node is primary,
                     # degraded and if the local copy of the data is inconsistent.
                 pri-on-incon-degr "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f ";
                 outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
        }

        startup {
                wfc-timeout 0;
                degr-wfc-timeout 30;
        }

        disk   {
               on-io-error   detach;
               fencing resource-only;
        }

        net {
                cram-hmac-alg sha1;
                shared-secret "my-secret";
                after-sb-0pri discard-zero-changes;
                after-sb-1pri discard-secondary;
                after-sb-2pri disconnect;
        }

        on proxmox1 {
                    device /dev/drbd2;
                    disk /dev/vg1/data;
                    address 10.0.7.19:7790;
                    meta-disk internal;
        }
        on proxmox2 {
                    device /dev/drbd2;
                    disk /dev/vg1/data;
                    address 10.0.7.16:7790;
                    meta-disk internal;
        }
}

# init.d/drbd reload
Install heartbeat:
Code:
aptitude install heartbeat
/etc/heartbeat/ha.cf
Code:
# /etc/ha.d/ha.cf
# /etc/init.d/heartbeat reload  # for change to take
# 2010-03-08 added. see wiki. we had a bug and 80GB/sec of coredumbs filled the disk
coredumps false
use_logd on
baud 19200
# Heartbeat cluster members
node proxmox1
node proxmox2
# Heartbeat communication timing
keepalive 1
warntime 10
deadtime 30
initdead 60
# Heartbeat communication paths
udpport 694
ucast eth1 10.0.7.19
ucast eth1 10.0.7.16
ucast eth0 10.100.100.19
ucast eth0 10.100.100.16
# 2010-04-05 commented out as we do not have cable connected:
#serial /dev/ttyS0
# Don't fail back automatically
auto_failback off
# Monitoring of network connection to default gateway
ping 10.100.100.2
respawn hacluster /usr/lib64/heartbeat/ipfail
# /etc/init.d/heartbeat reload  # for change to take
/etc/ha.d/haresources
Code:
#!/bin/bash
# /etc/ha.d/  haresources

proxmox1 fbc6 \
         drbddisk::r2  \
         Filesystem::/dev/drbd2::/data::ext3 \
         fbc6 \
         10.100.100.6 \
         apache2 \
         pvedaemon \
         vz      \
         qm-fbc \
         cron \
         MailTo::put-your-address-here

# /etc/init.d/heartbeat reload  # for change to take

# notes:
#       cron added 2010-04-03 .  pve cron scripts can only be run on Primary.
/etc/ha.d/resource.d/fbc6
Code:
#!/bin/bash
# /etc/ha.d/resource.d/fbc6
#
# I call this 2x from haresources  as heartbeat and vz have
#  some kind of confusion with mounts .. see wiki

/fbc/bin/update-rc-fbc6  # rmv init.d scripts controlled by heartbeat.

# testing
##echo "testing "| mail -s "$0 $HOSTNAME /etc/ha.d/resource.d/fbc6"  rob


# I think the /home and /bkup umounts are caused by drbd and heartbeat not getting along with pve/vz

umount /home
mount  /home

#  if /bkup used in any vz's then un comment:
#umount /bkup
#mount /bkup

exit 0
/etc/ha.d/resource.d/qm-fbc
Code:
#!/bin/bash
# /etc/ha.d/resource.d/qm-fbc

# stop KVM containers from here, else it takes 180 secs
# using  'qemu-server' in haresources

CMD="$1"

#qm $CMD 105

#qm $CMD 106


case "$CMD" in
        start)
           /etc/init.d/qemu-server start ;
            ;;
        stop)
           qm stop 105 ;
           qm stop 106 ;
           /etc/init.d/qemu-server stop ;
            ;;
esac

exit 0
run this now and from cron hourly:
/fbc/bin/update-rc-fbc6
Code:
#!/bin/bash

/fbc/bin/update-rc-fbc19  # you  may want this in /usr/local/bin .

# *********************************
# this is used  by
#     /etc/ha.d/resource.d/fbc19
# later in a cronscript like
#
# this is needed as deb updates sometimes change init.d starts
#
# ********************************
if [ "${EUID}" -ne 0 ]; then
   echo "$0: must be root."
   exit 1
fi

update-rc.d -f pvedaemon  remove
update-rc.d -f apache2  remove
/etc changes
on both nodes:
Code:
umount /var/lib/vz
mv /var/lib/vz /var/lib/vz.orig

vi /etc/fstab  # comment out /var/lib/vz mount:

#   this is on drbd mounted by heartbeat :
#/dev/pve/data /var/lib/vz.orig ext3 defaults 0 1

# only on Primary node:
mount  /dev/pve/data /var/lib/vz.orig
prepare folders
Code:
#                   on both nodes:
####### /etc
mv /etc/vz /etc/vz.orig
ln -s /data/etc/vz /etc/vz

mv /etc/pve /etc/pve.orig
ln -s /data/etc/pve /etc/pve

mv /etc/qemu-server /etc/qemu-server.orig
ln -s /data/etc/qemu-server /etc/qemu-server

########  /var/lib
mv /var/lib/vzquota /var/lib/vzquota.orig
ln -s /data/var/lib/vzquota /var/lib/vzquota

mv /var/lib/vzctl /var/lib/vzctl.orig
ln -s /data/var/lib/vzctl /var/lib/vzctl

mv /var/lib/pve-manager  /var/lib/pve-manager.orig
ln -s /data/var/lib/pve-manager  /var/lib/pve-manager
copy etc and lib
on Primary do this:
Code:
mount /dev/drbd2  /data
mkdir -p /data/var/lib/vz   

# /etc
rsync -a /etc/vz.orig/ /data/etc/vz/
rsync -a /etc/pve.orig/ /data/etc/pve/
raync -a /etc/qemu-server.orig/   /data/etc/qemu-server/

# /var/lib
rsync -a /var/lib/vz.orig/          /data/var/lib/vz/ # may take awhile
rsync -a /var/lib/vzquota.orig/     /data/var/lib/vzquota/
rsync -a /var/lib/vzctl.orig/       /data/var/lib/vzctl/
rsync -a /var/lib/pve-manager.orig/ /data/var/lib/vz/pve-manager/

umount /data
authkeys /etc/ha.d/authkeys
Code:
auth 1
1 sha1 PutYourSuperSecretKeyHere
Finally, you can now start heartbeat on both nodes:
/etc/init.d/heartbeat start

there are some issues with heartbeat filesystem and open-vz . If you have VMID.mount doing mounts inside containers, those actual mount may try to be unmounted when heartbeat stops. This happens with nfs and local mounts. check /var/log/daemon.log and look for something like this:
Code:
Apr  3 14:30:24 proxmox2 ResourceManager[14751]: [16812]: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd1 /var/lib/vz ext3 stop
Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16854]: INFO: Running stop for /dev/drbd1 on /var/lib/vz
Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16864]: INFO: Trying to unmount /var/lib/vz
Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16866]: INFO: unmounted /var/lib/vz/root/102/home successfully
Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16867]: INFO: Trying to unmount /var/lib/vz
Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16869]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGTERM
Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16871]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
Apr  3 14:30:25 proxmox2 Filesystem[16824]: [16874]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGTERM
Apr  3 14:30:25 proxmox2 Filesystem[16824]: [16876]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
Apr  3 14:30:26 proxmox2 Filesystem[16824]: [16879]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGTERM
Apr  3 14:30:26 proxmox2 Filesystem[16824]: [16881]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
Apr  3 14:30:27 proxmox2 Filesystem[16824]: [16884]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGKILL
Apr  3 14:30:27 proxmox2 Filesystem[16824]: [16886]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
Apr  3 14:30:28 proxmox2 Filesystem[16824]: [16889]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGKILL
Apr  3 14:30:28 proxmox2 Filesystem[16824]: [16891]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
Apr  3 14:30:29 proxmox2 Filesystem[16824]: [16894]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGKILL
Apr  3 14:30:29 proxmox2 Filesystem[16824]: [16896]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
Apr  3 14:30:30 proxmox2 ntpd[16694]: synchronized to 128.113.28.67, stratum 2
Apr  3 14:30:30 proxmox2 ntpd[16694]: kernel time sync status change 0001
Apr  3 14:30:30 proxmox2 Filesystem[16824]: [16898]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc, giving up!
Apr  3 14:30:30 proxmox2 Filesystem[16824]: [16899]: INFO: Trying to unmount /var/lib/vz
Apr  3 14:30:30 proxmox2 Filesystem[16824]: [16902]: INFO: unmounted /var/lib/vz/root/102/bkup successfully
In our case /home is an nfs mount, so in haresources we do umount / mount to get around the problem.

If you've got questions ask. We've used Debian , drbd and heartbeat for a while. and I can answer questions about our setup.
see http://www.drbd.org/users-guide/ for great info on drbd .
 
Last edited:
here is info on drbd setup.

this goes after "/etc/drbd.conf" above.
I could not insert it there as the post was too large.

setup system group, i'll have to check my notes on why we needed this:
Code:
addgroup --system  haclient
  chgrp haclient /sbin/drbdsetup
  chmod o-x /sbin/drbdsetup
  chmod u+s /sbin/drbdsetup

  chgrp haclient /sbin/drbdmeta
  chmod o-x /sbin/drbdmeta
  chmod u+s /sbin/drbdmeta
start drbd :
/etc/init.d/drbd start

Initializes the meta data storage. see man drbdadm . on both nodes:
Code:
drbdadm create-md r2
# on Primary :
drbdadm -- --overwrite-data-of-peer primary r2

# check progress of init r2 :

watch cat /proc/drbd

# on primary:
mkfs.ext3 /dev/drbd2
 
Last edited:
there are still some things to be done to improve the setup. these I know of, and I'm sure that there are more. please point out things I do not know .

- deb updates probably can only be done on the primary. to fix this we'll need to setup and test drbdlinks. we use drbdlinks on our old main server
 
If I start moving the folders from PVE, my cluster doesn't work anymore and I won't be able to access proxmox through the webinterface, because Proxmox is asking for the files in /etc/pve (which is now /etc/pve.orig).

I'm trying to do this step by step, but there are a few unclear things in your setup. For example you use alot of different IP addresses. And some IP addresses are unclear for what you use those.

But I could be misunderstanding things.
 
which i/p addresses?
the drbd.conf one is for a 'crossover' cable . there is a nic on each computer with a crossover cabe conneting both of them .
this is our /etc/network/interfaces file , you may not need vmbr1 . you do need a dedicated nic for drbd:
Code:
auto vmbr0
iface vmbr0 inet static
        address 10.100.100.19
        netmask 255.255.255.0
        gateway 10.100.100.2
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

# for DRBD / Heartbeat
auto eth1
iface eth1 inet static
        address  10.0.7.19
        netmask  255.255.240.0

# added 2010-04-01 for phone network. needed for faxing and probably more.
auto vmbr1
iface vmbr1 inet static
        address 192.168.1.19
        netmask 255.255.255.0
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0
I run pve only on the active cluster node. with active/passive setup , only the active one will have the configuration available.

we use the heartbeat address of '10.100.100.6' to connect to the web manager pve page.
 
Last edited:
Also, I added a script to haresources to start/stop vz . it is /etc/resources.d/vz-fbc . we did this because after a vz container with bind mounts is stopped, the system still sees the mounts. this may be a bug or something we messed up.



so our haresources is now:
Code:
# /etc/ha.d/  haresources
proxmox1 \
         drbddisk::r1  \
         Filesystem::/dev/drbd1::/data2::ext3 \
         drbddisk::r2  \
         Filesystem::/dev/drbd2::/data::ext3 \
         drbdlinks \
         10.100.100.6 \
         apache2 \
         dhcp3-server \
         pvedaemon \
         vz-fbc      \
         qm-fbc \
         killnfsd \
         nfs-kernel-server \
         cron \
         fbc6 \
         MailTo::fbcadmin@fantinibakery.com

/etc/ha.d/resource.d/vz-fbc
Code:
#!/bin/bash                                                                                                      
# /etc/ha.d/resource.d/vz-fbc                                                                                    

CMD="$1"

case "$CMD" in
        start)
           mount -a ;
           /etc/init.d/vz start ;
            ;;
        stop)
          # do umounts here as  they get left mounted when container stops
           umount /data/var/lib/vz/root/101/u   ;
           umount /data/var/lib/vz/root/101/fbc ;
           umount /data/var/lib/vz/root/101/home ;

           umount /data/var/lib/vz/root/102/home ;
           umount /data/var/lib/vz/root/102/fbc  ;
           umount /data/var/lib/vz/root/114/home ;
           umount /data/var/lib/vz/root/114/fbc  ;


           vzctl  stop 101 ;
           vzctl  stop 102 ;
           vzctl  stop 114 ;
           /etc/init.d/vz  stop ;
           # some real mounts can get unmounted.  try to handle here
           mount -a ;
            ;;
esac
exit 0
 
I run pve only on the active cluster node. with active/passive setup , only the active one will have the configuration available.

we use the heartbeat address of '10.100.100.6' to connect to the web manager pve page.

How do you let Proxmox know to use 10.100.100.6 for connection to the web interface? I only see this ip address in your haresources. This is enough for proxmox?

And something that is unclear to me:


  1. What is fbc?
  2. Did you already have the vm's running or did you make them after the configurations?
 
1- 'fbc' is our company's name. you can name the scripts 'local-vz' . that would be clearer.

2- I would have all the services stopped before working on this.
 
Sorry I meant something different with Question 2:
Did you already have the vm's created on your disk before the configuration or did you create them with proxmox web interface after you configured your Active/Passive cluster??

The reason I ask this because I want to know if it is still possible to use the webgui to add containers without any problems.
 
I already had vm's created.

We use the web gui at address the heartbeat address. Your heartbeat address needs to be part of your main network. our network is 10.100.100.0 .
 
Do you use a seperate hard disk for your drbd synchronisation, or is it on the same hard disk as your proxmox?

Right now I have in each server 2 hard disks. 1 hard disk has proxmox on it and the other I want to store VM's. This 2nd hard disk is synchronized by drbd. Am I right to say that this won't work with your setup?
 
If I make my drbd synced harddisk into LVM group it doesn't allow me to make openvz containers on it.

And I only have DRBD running so far. I'm still trying to figure out the other bits of your setup.
 
I have read that open-vz requires an ext3 file system.

you need to have drbd setup correctly before using heartbeat .
try doing the following... check 'man drbdadm' ..

on the primary node,
1- assuming your device is drbd0
2- that the resource is named r0 [check the 'resource' in /etc/drbd.conf ]
Code:
  mkfs.ext3  /dev/drbd0  
  mkdir /mnt/drbd0
  mount /dev/drbd0 /mnt/drbd0
  ls  /etc > /mnt/drbd0/test-file-delete-me
  umount /mnt/drbd0
  drbdadm secondary r0
on the other node
Code:
drbdadm primary r0
mkdir /mnt/drbd0
mount /dev/drbd0 /mnt/drbd0  
ls /mnt/drbd0
 
Hello,

I'm working on the set up of a HA configuration for virtualization with proxmox, drbd and heartbeat.

I'm actually testing and trying to set up the two servers but i run into a lot of troubles ...

Actually the biggest one is how can i get heartbeat and proxmox to work together ?

When i'm installing proxmox above heartbeat, he uninstall it.

Can someone help me in this matter ?

For drbd, i could set it up perfectly, i can also easily use proxmox. But heartbeat is giving me real headache right now ...

I would really appreciate any help you can provide.

Thanks a lot for reading !
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!