lvm - drbd - lvm

[Frazze]

Hi guys,

I'd like to get some opinions on how to best combine lvm and drbd.

lvm -> drbd

or

drbd -> lvm

or

lvm -> drbd -> lvm

What are the advantages of these combinations?
From my understanding:

lvm -> drbd : provides me an easy way to resize the underlying lvm partition without risk.

drbd -> lvm : gives me more flexibility on every single VMs LV

lvm -> drbd -> lvm : gives me the best of both worlds at the cost of extra complexity. I can resize the underlying partitions and also individuality easy manage every VMs LV.


What setup are you running and why?
Should I be concerned by bad performance if I go for the nested lvm -> drbd -> lvm concept?

 

[cesarpk]

Hi guys,

I'd like to get some opinions on how to best combine lvm and drbd.

lvm -> drbd
or
drbd -> lvm
or
lvm -> drbd -> lvm

What are the advantages of these combinations?
From my understanding:

lvm -> drbd : provides me an easy way to resize the underlying lvm partition without risk.
drbd -> lvm : gives me more flexibility on every single VMs LV
lvm -> drbd -> lvm : gives me the best of both worlds at the cost of extra complexity. I can resize the underlying partitions and also individuality easy manage every VMs LV.

What setup are you running and why?
Should I be concerned by bad performance if I go for the nested lvm -> drbd -> lvm concept?

My suggestion is have LVM on top of DRBD, because:

1- LVM can Resize
2- DRBD can Resize:
shell # man drdbadm
resize
Causes DRBD to re-examine all sizing constraints, and resize the resource's device accordingly. For example,
if you increased the size of your backing storage devices (on both nodes, of course), then DRBD will adopt the
new size after you called this command on one of your nodes. Since new storage space must be synchronised this
command only works if there is at least one primary node present.

3- PVE can resize any format of virtual disks

Best regards
Cesar

 

[Frazze]

Okay thanks for the answer.

But why are some people and also the drbd manual suggesting the lvm -> drbd -> lvm setup? Is there any advantage we don't see?
What about lvm snapshots?

 

[screenie]

i'm using drbd -> lvm because my if i have to resize the underlying device several times i did a bad job on planning/sizing the hardware;
if i need to expand the underlying device on production servers that means changing all the harddisks to bigger ones and we are using 1u servers with 8 disks in hw-raid6;
it would be possible to swap disk for disk with rebuilding the raid and increase it afterwards but i wouldn't do that at all - i would simply move off all the vm's, re-install the whole system from scratch on the new disks and move the vm's back on;

 

[cesarpk]

Okay thanks for the answer.

But why are some people and also the drbd manual suggesting the lvm -> drbd -> lvm setup? Is there any advantage we don't see?
What about lvm snapshots?

The real question are two:
A) For best performance must i have more layers and give more work to my processor? ( lvm + drbd + lvm = 3 layers ). So, the answer is "NO"

Only for remember the layers:
1- LVM partition
2- DRBD on LVM partition
3- PV (physical volume on DRBD)
4- VG (Virtual Group on PV)
- This no is a layer: Add the VG to the Proxmox VE storage list via web interface
5- LV (Logical Volume on VG) : The Virtual disk(s) of KVM (added via PVE web interface)

Then, in this configuration i only have the minimum necessary layers

B) Benefits: What extra benefits i will have using 3 layers ( lvm + drbd + lvm)?. I don't see extra benefits

And about of lvm snapshots, since PVE 2.3, lvm snapshots is only useful for CTs and not for VMs, and since I don't use CTs, i can't talk much about of this.

 

[screenie]

i would say lvm -> drbd -> lvm make sense if you have a large storage server where you can expand the capacity easy by adding as much disks as you need;
here you can resize the first lvm to the new size, then grow the drbd device and on the second lvm on top of the drbd you would have multiple vg's/lvol's for different purposes like nfs/iscsi mounts, smb shares or whatever and you have the flexibility to snapshot each vg whenever and how often you want without snapshotting everything at once;

if you are building a (dedicated) large replicated storage system - yes, lvm->drbd->lvm is the way to go
on a computing server which are mostly 1u servers for high densityi don't see the benefit - as many people have there data stored on separate storage systems, like the replicated storage system i mentioned before;

 

[cesarpk]

i would say lvm -> drbd -> lvm make sense if you have a large storage server where you can expand the capacity easy by adding as much disks as you need;
here you can resize the first lvm to the new size, then grow the drbd device and on the second lvm on top of the drbd you would have multiple vg's/lvol's for different purposes like nfs/iscsi mounts, smb shares or whatever and you have the flexibility to snapshot each vg whenever and how often you want without snapshotting everything at once;

if you are building a (dedicated) large replicated storage system - yes, lvm->drbd->lvm is the way to go
on a computing server which are mostly 1u servers for high densityi don't see the benefit - as many people have there data stored on separate storage systems, like the replicated storage system i mentioned before;

Many roads lead to Rome !!!

If I have all space of a LVM parition with the PV and VG into DRBD, i can easily resize the LV of two manners:
1- On the PVE GUI, or
2- Manually by CLI,
Then my virtual disks of my VMs will have the size desired.

But if you purpose is to have several volumes in a disk for use with other stuff that isn't VMs or CTs, then you can do it in three layers, but you will sacrifice performance and will have more work manual.

One question: Had you tested the difference of performance using boniee++ with the same Hardware and Software installed but with only one VM running?

Best regards
Cesar