It sounds like you've already got the server up and running with partitions and guest machines. I don't know if your disks are set up as
LVM partitions or how you were planning to store the
DRBD Metadata. Try to think of DRBD like a normal RAID controller - it might be *possible* to take an existing disk and convert it into an array, but it's much more complicated. If none of that made any sense, I'd recommend starting fresh with DRBD in mind.
The most important thing I've learned about DRBD is to test it once you've got it set up. The DRBD daemon and tools are reliable and stable, but they are *NOT* foolproof. DRBD is one of the best examples for "did what you told it to, not what you meant for it to do." Make sure you know what is supposed to happen when:
The ethernet cable gets unplugged
One of the servers reboots for no good reason
Both of the servers reboot for no good reason
Both servers reboot, and only one comes back up (!!)
Split-brain has been detected (try to generate this problem a few times)
The setup for your system is almost identical to mine. The
How-To section of the Wiki has a good
description for a single disk, you'll basically go through it twice. My server has four disks: /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd
/dev/sda1 is for Proxmox
/dev/sdb1 backs DRBD-R0 -- scratch/testing
/dev/sdc1 backs DRBD-R1 -- guests for PVE1
/dev/sdd1 backs DRBD-R2 -- guests for PVE2
I've got three DRBD resources, so I'm using three network interfaces. (This isn't required, you could do it with one.)
from /etc/network/interfaces:
Code:
auto eth1
iface eth1 inet static
address 10.1.1.1
netmask 255.255.255.0
auto eth4
iface eth4 inet static
address 10.1.2.1
netmask 255.255.255.0
auto eth5
iface eth5 inet static
address 10.1.3.1
netmask 255.255.255.0
The addresses above need to be different on the other server. (ex: 10.1.1.2)
Partitioning your disks is the same as the How-To, just do it for each device.
The drbd.conf file is very smilar, just longer. You'll have two resources instead of one.
My drbd.conf file:
Code:
include "drbd.d/global_common.conf";
resource r0 {
protocol C;
net {
shared-secret "enter_secret1_here";
}
on pve1 {
device /dev/drbd0;
disk /dev/sdb1;
address 10.1.1.1:7788;
meta-disk internal;
}
on pve2 {
device /dev/drbd0;
disk /dev/sdb1;
address 10.1.1.2:7788;
meta-disk internal;
}
}
resource r1 {
protocol C;
net {
shared-secret "enter_secret2_here";
}
on pve1 {
device /dev/drbd1;
disk /dev/sdc1;
address 10.1.2.1:7788;
meta-disk internal;
}
on pve2 {
device /dev/drbd1;
disk /dev/sdc1;
address 10.1.2.2:7788;
meta-disk internal;
}
}
resource r2 {
protocol C;
net {
shared-secret "enter_secret3_here";
}
on pve1 {
device /dev/drbd2;
disk /dev/sdd1;
address 10.1.3.1:7788;
meta-disk internal;
}
on pve2 {
device /dev/drbd2;
disk /dev/sdd1;
address 10.1.3.2:7788;
meta-disk internal;
}
}
When you're running two or more DRBD volumes that will behave in the same way, you can move these settings to the global configuration file.
global_common.conf
Code:
global {
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh";
local-io-error "/usr/lib/drbd/notify-io-error.sh";
split-brain "/usr/lib/drbd/notify-split-brain.sh root";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb;
degr-wfc-timeout 60;
become-primary-on both;
}
disk {
# on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
# no-disk-drain no-md-flushes max-bio-bvecs
}
net {
# snd‐buf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers
# max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret
# after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork
after-sb-0pri discard-zero-changes;
after-sb-1pri consensus;
after-sb-2pri disconnect;
allow-two-primaries;
}
syncer {
# rate after al-extents use-rle cpu-mask verify-alg csums-alg
rate 100M;
csums-alg md5;
verify-alg md5;
}
}
Pay close attention to what you put into the
handlers{} section of the configuration file!! The Wiki version leaves it out - it's reasonably safe for your data, but it doesn't report any problems. I'm running a reduced set of handlers, they're mostly warnings and no real action.
The
syncer{} section is worth considering - by default, DRBD doesn't check to make sure that the data on your two servers is actually the same. It's possible to have it perform consistency checks all the time,
but it causes problems with the Primary/Primary setup required for Proxmox.
After starting the drbd service, you'll create your resources:
drbdadm create-md r0
drbdadm create-md r1
drbdadm up r0
drbdadm up r1
drbdadm -- --overwrite-data-of-peer primary r0
drbdadm -- --overwrite-data-of-peer primary r1
Here's my "lvm exclusion" for multiple disks:
Code:
filter = [ "r|/dev/sdb1|", "r|/dev/sdc1|", "r|/dev/sdd1|", "r|/dev/disk/|", "r|/dev/block/|", "a/.*/" ]
You'll need to create two physical volumes, and two logical volume groups
pvcreate /dev/drbd0
pvcreate /dev/drbd1
vgcreate drbdvg0 /dev/drbd0
vgcreate drbdvg1 /dev/drbd1
Finally, add both volume groups to the storage system with the Proxmox web interface.
Additional Resources:
DRBD Manual http://www.drbd.org/users-guide/
DRBD Mailing List Archive http://www.gossamer-threads.com/lists/drbd/users/
Proxmox is a great system and it's moving fast, but it's not as big as some of the other players in the virtualization market. If you can't find the help you need for a specific problem, try searching for a similar problem on the Xen platform. At the end of the day DRBD is "just a disk" to be used by other software.
