2 Node Cluster with single DRBD dual-primary - why is there no data corruption

swick

Renowned Member
Apr 19, 2015
4
1
68
Hi,

I am aware of all the risks and possible edge-cases
running a Proxmox 2 Node Cluster with a single dual-primary DRBD device.

In production we actually use two dual-primary DRBD devices
on the nodes (as recommended in the Wiki page).

So, this is just for understanding WHY and HOW it actually works
with a single dual-primary DRBD device and multiple VMs distributed
over the 2 nodes without corrupting the data on the DRBD device.

As already mentioned it is described in the Wiki
as a workable solution (but not recommended)
and I did my own tests to verify it:


  • Created a standard 2 node Proxmox Cluster
  • Both nodes shared a single drbd0
  • Created LVM on top of drbd0
  • Started four GNU/Linux Installations in parallel (via EPXE)
    - 1 Fedora
    - 1 Ubuntu
    - 1 Arch
    - 1 Debian
  • Did many Live Migrations of the VMs during their installation
  • Nothing broke and they finished installing successfully.


So,


Why and how is the data on the DRBD device not corrupted
when written to concurrently from 2 nodes?

Is DRBD actively syncing in both directions
when a VM on Node 1 and a VM on Node 2
are writing to the DRBD device (into their own LV) at the same time?

When Node 2 creates a new LV for a VM disk
and thus changes the LVM metadata,
how does Node 1 know the metadata has changed?

Is there some notification on Proxmox level
to refresh the underlying LVM information?

I read in the forum that Proxmox uses its own locking
for LVM and cLVM is not needed.

Under which circumstances does data
on the DRBD device get corrupted?

Not using the Proxmox CLI commands or Web UI
but using LVM tools directly
and manipulating the LVM metadata?

Thanks in advance for clarifying some of the questions.
 
Last edited:
...
So, this is just for understanding WHY and HOW it actually works
with a single dual-primary DRBD device and multiple VMs distributed
over the 2 nodes without corrupting the data on the DRBD device.
...

Hi,
this work, because pve take care that one LV is only active from one node at the time.
And DRBD sync the content between both nodes.

The only reason why you should use two drbd-devices is to avoid the trouble if you have an split-brain condition. In this case you must overwrite on drbd-side and this means you can't have active LVs on this side...

Udo
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!