proxmox-ve active/passive cluster

Discussion in 'Proxmox VE 1.x: Installation and configuration' started by bread-baker, Apr 8, 2010.

  1. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    hello
    We've setup a pair of proxmox-ve servers into a high availability system using drbd and heartbeat. This uses drbd in Primary/Secondary mode.

    We are running 2 open-vz and 2 lvm containers. We still may have some changes to make, but have done a lot of fail over tests and things are working well. We can unplug the primary server , and within a few minutes the second server turns into primary and brings up our 4 vm's.
    We use Supermicro servers with 3-ware raid-10 .

    see this as I got a lot of info from: http://wiki.openvz.org/HA_cluster_with_DRBD_and_Heartbeat


    /etc/drbd.conf
    Code:
    global { usage-count yes; }
    common { syncer { rate 100M; } }
    
    resource r2 {
     protocol C;
            handlers {
                         # "pri-on-incon-degr": This handler is called if the node is primary,
                         # degraded and if the local copy of the data is inconsistent.
                     pri-on-incon-degr "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f ";
                     outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
            }
    
            startup {
                    wfc-timeout 0;
                    degr-wfc-timeout 30;
            }
    
            disk   {
                   on-io-error   detach;
                   fencing resource-only;
            }
    
            net {
                    cram-hmac-alg sha1;
                    shared-secret "my-secret";
                    after-sb-0pri discard-zero-changes;
                    after-sb-1pri discard-secondary;
                    after-sb-2pri disconnect;
            }
    
            on proxmox1 {
                        device /dev/drbd2;
                        disk /dev/vg1/data;
                        address 10.0.7.19:7790;
                        meta-disk internal;
            }
            on proxmox2 {
                        device /dev/drbd2;
                        disk /dev/vg1/data;
                        address 10.0.7.16:7790;
                        meta-disk internal;
            }
    }
    
    # init.d/drbd reload
    
    Install heartbeat:
    Code:
    aptitude install heartbeat
    
    /etc/heartbeat/ha.cf
    Code:
    # /etc/ha.d/ha.cf
    # /etc/init.d/heartbeat reload  # for change to take
    # 2010-03-08 added. see wiki. we had a bug and 80GB/sec of coredumbs filled the disk
    coredumps false
    use_logd on
    baud 19200
    # Heartbeat cluster members
    node proxmox1
    node proxmox2
    # Heartbeat communication timing
    keepalive 1
    warntime 10
    deadtime 30
    initdead 60
    # Heartbeat communication paths
    udpport 694
    ucast eth1 10.0.7.19
    ucast eth1 10.0.7.16
    ucast eth0 10.100.100.19
    ucast eth0 10.100.100.16
    # 2010-04-05 commented out as we do not have cable connected:
    #serial /dev/ttyS0
    # Don't fail back automatically
    auto_failback off
    # Monitoring of network connection to default gateway
    ping 10.100.100.2
    respawn hacluster /usr/lib64/heartbeat/ipfail
    # /etc/init.d/heartbeat reload  # for change to take
    
    /etc/ha.d/haresources
    Code:
    #!/bin/bash
    # /etc/ha.d/  haresources
    
    proxmox1 fbc6 \
             drbddisk::r2  \
             Filesystem::/dev/drbd2::/data::ext3 \
             fbc6 \
             10.100.100.6 \
             apache2 \
             pvedaemon \
             vz      \
             qm-fbc \
             cron \
             MailTo::put-your-address-here
    
    # /etc/init.d/heartbeat reload  # for change to take
    
    # notes:
    #       cron added 2010-04-03 .  pve cron scripts can only be run on Primary.
    
    /etc/ha.d/resource.d/fbc6
    Code:
    #!/bin/bash
    # /etc/ha.d/resource.d/fbc6
    #
    # I call this 2x from haresources  as heartbeat and vz have
    #  some kind of confusion with mounts .. see wiki
    
    /fbc/bin/update-rc-fbc6  # rmv init.d scripts controlled by heartbeat.
    
    # testing
    ##echo "testing "| mail -s "$0 $HOSTNAME /etc/ha.d/resource.d/fbc6"  rob
    
    
    # I think the /home and /bkup umounts are caused by drbd and heartbeat not getting along with pve/vz
    
    umount /home
    mount  /home
    
    #  if /bkup used in any vz's then un comment:
    #umount /bkup
    #mount /bkup
    
    exit 0
    
    /etc/ha.d/resource.d/qm-fbc
    Code:
    #!/bin/bash
    # /etc/ha.d/resource.d/qm-fbc
    
    # stop KVM containers from here, else it takes 180 secs
    # using  'qemu-server' in haresources
    
    CMD="$1"
    
    #qm $CMD 105
    
    #qm $CMD 106
    
    
    case "$CMD" in
            start)
               /etc/init.d/qemu-server start ;
                ;;
            stop)
               qm stop 105 ;
               qm stop 106 ;
               /etc/init.d/qemu-server stop ;
                ;;
    esac
    
    exit 0
    
    run this now and from cron hourly:
    /fbc/bin/update-rc-fbc6
    Code:
    #!/bin/bash
    
    /fbc/bin/update-rc-fbc19  # you  may want this in /usr/local/bin .
    
    # *********************************
    # this is used  by
    #     /etc/ha.d/resource.d/fbc19
    # later in a cronscript like
    #
    # this is needed as deb updates sometimes change init.d starts
    #
    # ********************************
    if [ "${EUID}" -ne 0 ]; then
       echo "$0: must be root."
       exit 1
    fi
    
    update-rc.d -f pvedaemon  remove
    update-rc.d -f apache2  remove
    
    /etc changes
    on both nodes:
    Code:
    umount /var/lib/vz
    mv /var/lib/vz /var/lib/vz.orig
    
    vi /etc/fstab  # comment out /var/lib/vz mount:
    
    #   this is on drbd mounted by heartbeat :
    #/dev/pve/data /var/lib/vz.orig ext3 defaults 0 1
    
    # only on Primary node:
    mount  /dev/pve/data /var/lib/vz.orig
    
    
    prepare folders
    Code:
    #                   on both nodes:
    ####### /etc
    mv /etc/vz /etc/vz.orig
    ln -s /data/etc/vz /etc/vz
    
    mv /etc/pve /etc/pve.orig
    ln -s /data/etc/pve /etc/pve
    
    mv /etc/qemu-server /etc/qemu-server.orig
    ln -s /data/etc/qemu-server /etc/qemu-server
    
    ########  /var/lib
    mv /var/lib/vzquota /var/lib/vzquota.orig
    ln -s /data/var/lib/vzquota /var/lib/vzquota
    
    mv /var/lib/vzctl /var/lib/vzctl.orig
    ln -s /data/var/lib/vzctl /var/lib/vzctl
    
    mv /var/lib/pve-manager  /var/lib/pve-manager.orig
    ln -s /data/var/lib/pve-manager  /var/lib/pve-manager
    
    
    copy etc and lib
    on Primary do this:
    Code:
    mount /dev/drbd2  /data
    mkdir -p /data/var/lib/vz   
    
    # /etc
    rsync -a /etc/vz.orig/ /data/etc/vz/
    rsync -a /etc/pve.orig/ /data/etc/pve/
    raync -a /etc/qemu-server.orig/   /data/etc/qemu-server/
    
    # /var/lib
    rsync -a /var/lib/vz.orig/          /data/var/lib/vz/ # may take awhile
    rsync -a /var/lib/vzquota.orig/     /data/var/lib/vzquota/
    rsync -a /var/lib/vzctl.orig/       /data/var/lib/vzctl/
    rsync -a /var/lib/pve-manager.orig/ /data/var/lib/vz/pve-manager/
    
    umount /data
    
    authkeys /etc/ha.d/authkeys
    Code:
    auth 1
    1 sha1 PutYourSuperSecretKeyHere
    
    Finally, you can now start heartbeat on both nodes:
    /etc/init.d/heartbeat start

    there are some issues with heartbeat filesystem and open-vz . If you have VMID.mount doing mounts inside containers, those actual mount may try to be unmounted when heartbeat stops. This happens with nfs and local mounts. check /var/log/daemon.log and look for something like this:
    Code:
    Apr  3 14:30:24 proxmox2 ResourceManager[14751]: [16812]: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd1 /var/lib/vz ext3 stop
    Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16854]: INFO: Running stop for /dev/drbd1 on /var/lib/vz
    Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16864]: INFO: Trying to unmount /var/lib/vz
    Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16866]: INFO: unmounted /var/lib/vz/root/102/home successfully
    Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16867]: INFO: Trying to unmount /var/lib/vz
    Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16869]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGTERM
    Apr  3 14:30:24 proxmox2 Filesystem[16824]: [16871]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
    Apr  3 14:30:25 proxmox2 Filesystem[16824]: [16874]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGTERM
    Apr  3 14:30:25 proxmox2 Filesystem[16824]: [16876]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
    Apr  3 14:30:26 proxmox2 Filesystem[16824]: [16879]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGTERM
    Apr  3 14:30:26 proxmox2 Filesystem[16824]: [16881]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
    Apr  3 14:30:27 proxmox2 Filesystem[16824]: [16884]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGKILL
    Apr  3 14:30:27 proxmox2 Filesystem[16824]: [16886]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
    Apr  3 14:30:28 proxmox2 Filesystem[16824]: [16889]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGKILL
    Apr  3 14:30:28 proxmox2 Filesystem[16824]: [16891]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
    Apr  3 14:30:29 proxmox2 Filesystem[16824]: [16894]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc; trying cleanup with SIGKILL
    Apr  3 14:30:29 proxmox2 Filesystem[16824]: [16896]: INFO: No processes on /var/lib/vz/root/102/fbc were signalled
    Apr  3 14:30:30 proxmox2 ntpd[16694]: synchronized to 128.113.28.67, stratum 2
    Apr  3 14:30:30 proxmox2 ntpd[16694]: kernel time sync status change 0001
    Apr  3 14:30:30 proxmox2 Filesystem[16824]: [16898]: ERROR: Couldn't unmount /var/lib/vz/root/102/fbc, giving up!
    Apr  3 14:30:30 proxmox2 Filesystem[16824]: [16899]: INFO: Trying to unmount /var/lib/vz
    Apr  3 14:30:30 proxmox2 Filesystem[16824]: [16902]: INFO: unmounted /var/lib/vz/root/102/bkup successfully
    
    In our case /home is an nfs mount, so in haresources we do umount / mount to get around the problem.

    If you've got questions ask. We've used Debian , drbd and heartbeat for a while. and I can answer questions about our setup.
    see http://www.drbd.org/users-guide/ for great info on drbd .
     
    #1 bread-baker, Apr 8, 2010
    Last edited: Apr 8, 2010
  2. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    here is info on drbd setup.

    this goes after "/etc/drbd.conf" above.
    I could not insert it there as the post was too large.

    setup system group, i'll have to check my notes on why we needed this:
    Code:
    addgroup --system  haclient
      chgrp haclient /sbin/drbdsetup
      chmod o-x /sbin/drbdsetup
      chmod u+s /sbin/drbdsetup
    
      chgrp haclient /sbin/drbdmeta
      chmod o-x /sbin/drbdmeta
      chmod u+s /sbin/drbdmeta
    
    start drbd :
    /etc/init.d/drbd start

    Initializes the meta data storage. see man drbdadm . on both nodes:
    Code:
    drbdadm create-md r2
    
    # on Primary :
    drbdadm -- --overwrite-data-of-peer primary r2

    # check progress of init r2 :

    watch cat /proc/drbd

    # on primary:
    mkfs.ext3 /dev/drbd2
     
    #2 bread-baker, Apr 8, 2010
    Last edited: Apr 8, 2010
  3. remx

    remx New Member

    Joined:
    Mar 31, 2010
    Messages:
    14
    Likes Received:
    0
    thanks for this very good tutorial :)
     
  4. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    there are still some things to be done to improve the setup. these I know of, and I'm sure that there are more. please point out things I do not know .

    - deb updates probably can only be done on the primary. to fix this we'll need to setup and test drbdlinks. we use drbdlinks on our old main server
     
  5. Randy

    Randy New Member

    Joined:
    Apr 9, 2010
    Messages:
    7
    Likes Received:
    0
    If I start moving the folders from PVE, my cluster doesn't work anymore and I won't be able to access proxmox through the webinterface, because Proxmox is asking for the files in /etc/pve (which is now /etc/pve.orig).

    I'm trying to do this step by step, but there are a few unclear things in your setup. For example you use alot of different IP addresses. And some IP addresses are unclear for what you use those.

    But I could be misunderstanding things.
     
  6. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    which i/p addresses?
    the drbd.conf one is for a 'crossover' cable . there is a nic on each computer with a crossover cabe conneting both of them .
    this is our /etc/network/interfaces file , you may not need vmbr1 . you do need a dedicated nic for drbd:
    Code:
    auto vmbr0
    iface vmbr0 inet static
            address 10.100.100.19
            netmask 255.255.255.0
            gateway 10.100.100.2
            bridge_ports eth0
            bridge_stp off
            bridge_fd 0
    
    # for DRBD / Heartbeat
    auto eth1
    iface eth1 inet static
            address  10.0.7.19
            netmask  255.255.240.0
    
    # added 2010-04-01 for phone network. needed for faxing and probably more.
    auto vmbr1
    iface vmbr1 inet static
            address 192.168.1.19
            netmask 255.255.255.0
            bridge_ports eth2
            bridge_stp off
            bridge_fd 0
    
    
    I run pve only on the active cluster node. with active/passive setup , only the active one will have the configuration available.

    we use the heartbeat address of '10.100.100.6' to connect to the web manager pve page.
     
    #6 bread-baker, Apr 15, 2010
    Last edited: Apr 15, 2010
  7. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    Also, I added a script to haresources to start/stop vz . it is /etc/resources.d/vz-fbc . we did this because after a vz container with bind mounts is stopped, the system still sees the mounts. this may be a bug or something we messed up.



    so our haresources is now:
    Code:
    # /etc/ha.d/  haresources
    proxmox1 \
             drbddisk::r1  \
             Filesystem::/dev/drbd1::/data2::ext3 \
             drbddisk::r2  \
             Filesystem::/dev/drbd2::/data::ext3 \
             drbdlinks \
             10.100.100.6 \
             apache2 \
             dhcp3-server \
             pvedaemon \
             vz-fbc      \
             qm-fbc \
             killnfsd \
             nfs-kernel-server \
             cron \
             fbc6 \
             MailTo::fbcadmin@fantinibakery.com
    
    /etc/ha.d/resource.d/vz-fbc
    Code:
    #!/bin/bash                                                                                                      
    # /etc/ha.d/resource.d/vz-fbc                                                                                    
    
    CMD="$1"
    
    case "$CMD" in
            start)
               mount -a ;
               /etc/init.d/vz start ;
                ;;
            stop)
              # do umounts here as  they get left mounted when container stops
               umount /data/var/lib/vz/root/101/u   ;
               umount /data/var/lib/vz/root/101/fbc ;
               umount /data/var/lib/vz/root/101/home ;
    
               umount /data/var/lib/vz/root/102/home ;
               umount /data/var/lib/vz/root/102/fbc  ;
               umount /data/var/lib/vz/root/114/home ;
               umount /data/var/lib/vz/root/114/fbc  ;
    
    
               vzctl  stop 101 ;
               vzctl  stop 102 ;
               vzctl  stop 114 ;
               /etc/init.d/vz  stop ;
               # some real mounts can get unmounted.  try to handle here
               mount -a ;
                ;;
    esac
    exit 0
    
     
  8. Randy

    Randy New Member

    Joined:
    Apr 9, 2010
    Messages:
    7
    Likes Received:
    0
    How do you let Proxmox know to use 10.100.100.6 for connection to the web interface? I only see this ip address in your haresources. This is enough for proxmox?

    And something that is unclear to me:


    1. What is fbc?
    2. Did you already have the vm's running or did you make them after the configurations?
     
  9. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    1- 'fbc' is our company's name. you can name the scripts 'local-vz' . that would be clearer.

    2- I would have all the services stopped before working on this.
     
  10. Randy

    Randy New Member

    Joined:
    Apr 9, 2010
    Messages:
    7
    Likes Received:
    0
    Sorry I meant something different with Question 2:
    Did you already have the vm's created on your disk before the configuration or did you create them with proxmox web interface after you configured your Active/Passive cluster??

    The reason I ask this because I want to know if it is still possible to use the webgui to add containers without any problems.
     
  11. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    I already had vm's created.

    We use the web gui at address the heartbeat address. Your heartbeat address needs to be part of your main network. our network is 10.100.100.0 .
     
  12. Randy

    Randy New Member

    Joined:
    Apr 9, 2010
    Messages:
    7
    Likes Received:
    0
    Do you use a seperate hard disk for your drbd synchronisation, or is it on the same hard disk as your proxmox?

    Right now I have in each server 2 hard disks. 1 hard disk has proxmox on it and the other I want to store VM's. This 2nd hard disk is synchronized by drbd. Am I right to say that this won't work with your setup?
     
  13. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    it can work.

    have you used drbd and heartbeat ?
     
  14. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    you can use a disk or partition.
     
  15. Randy

    Randy New Member

    Joined:
    Apr 9, 2010
    Messages:
    7
    Likes Received:
    0
    If I make my drbd synced harddisk into LVM group it doesn't allow me to make openvz containers on it.

    And I only have DRBD running so far. I'm still trying to figure out the other bits of your setup.
     
  16. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    I have read that open-vz requires an ext3 file system.

    you need to have drbd setup correctly before using heartbeat .
    try doing the following... check 'man drbdadm' ..

    on the primary node,
    1- assuming your device is drbd0
    2- that the resource is named r0 [check the 'resource' in /etc/drbd.conf ]
    Code:
      mkfs.ext3  /dev/drbd0  
      mkdir /mnt/drbd0
      mount /dev/drbd0 /mnt/drbd0
      ls  /etc > /mnt/drbd0/test-file-delete-me
      umount /mnt/drbd0
      drbdadm secondary r0
    
    on the other node
    Code:
    drbdadm primary r0
    mkdir /mnt/drbd0
    mount /dev/drbd0 /mnt/drbd0  
    ls /mnt/drbd0
    
     
  17. Grimblee

    Grimblee New Member

    Joined:
    Mar 19, 2013
    Messages:
    1
    Likes Received:
    0
    Hello,

    I'm working on the set up of a HA configuration for virtualization with proxmox, drbd and heartbeat.

    I'm actually testing and trying to set up the two servers but i run into a lot of troubles ...

    Actually the biggest one is how can i get heartbeat and proxmox to work together ?

    When i'm installing proxmox above heartbeat, he uninstall it.

    Can someone help me in this matter ?

    For drbd, i could set it up perfectly, i can also easily use proxmox. But heartbeat is giving me real headache right now ...

    I would really appreciate any help you can provide.

    Thanks a lot for reading !
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice