dual-primary drbd storage connected to two-node-pve-cluster over iscsi

[ioo]

Hi!

I have run dual-primary drbd successfully with two node proxmox cluster, each pve node being also drbd node (and using local storage). And /dev/drbd0 block device is used as lvm pv device, lvm needs to be initialized at one node since block device content appears or rahter is same and both nodes. And this storage is used from pve webgui as shared-lvm flavor of storage. And eventually virtual computer's virtual disk is used from this shared-lvm resource, inside virtual computer there is for example another layer of lvm and on top of that is ext4 filesystem. Done that way, virtual computer is online moveable etc. No pacemaker needed or anything, works beautifully.

I understand it is kind of classic approach mentioned serveral times in forum.proxmox.com also, for example, https://forum.proxmox.com/threads/2...rimary-why-is-there-no-data-corruption.21883/ I imagine it to myself drbd setup being kind of software-defined-ha-storage.

And also I have run successfully two node proxmox cluster what is taking storage from over the network of third computer serving as iscsi target. iscsi resource is used as shared-lvm and as a result it is possible to have virtual computer online migration.

But i would like to ask about possible variation of these setups or actually combination of above two. I wish drbd nodes being separate from pve nodes. There would be for pve-01 node respective drbd-01 node, and for pve-02 respective drbd-02 node. drbd nodes form dual-primary setup and usable block devices at both computers are /dev/drbd0. And from drbd-01 it is connected over iscsi to pve-01 where is appears as say /dev/sdb device; and from drbd-02 it is connected to pve-02 similarily.

My expectation would be that next i create at pve-01 lvm on /dev/sdb and it will be present also at pve-02. And i could use it similarily to before mentioned so to say classic drbd-lvm approarch. Virtual computer is online movable etc.

I would be very thankful if you could comment on this, maybe suggest something to take into account, or just advice against it with little reasoning.

PS What i intend not to do and what ambition i do not have is dual-primary-iscsi-multipath like described at https://fghaas.wordpress.com/2011/11/29/dual-primary-drbd-iscsi-and-multipath-dont-do-that/ I understand drbd + iscsi are not suited for this.


Best regards,

Imre

 

[ioo]

Hi again!

Actually i am thinking of kind-of-one-pve-cluster what has two so to say inner islands i.e. two sets of nodes residing at very well connected different physical locations (say at one site its set having 6 nodes, and the other 5 nodes). And nodes of one island are connected over iscsi to drbd node residing at the same location; and nodes of other island are connected similarily to their drbd node. I believe this setup is kind of multisite solution and could resist some events (site management event, network interruption event etc). Maybe storage replication solution included into proxmox is better suited for similar ambition but at the moment i am rather contemplating about this drbd+iscsi+lvm.

Network-wise i imagine it like this that each site has its own ip layer networking especially considering end user traffic. Nodes' networks at different islands are different and independent from each other etc except one (or two) vlan-ip-subnet what is for cluster corosync etc traffic - this is stretched vlan (over ipsec or some lower layer means). iscsi traffic happens over some non-stretched i.e. over some routed traffic links between sites. I imagine big win being able to operate virtual machines online movement between two sites from ordinary pve webgui. And considering end user traffic it needs some dns help or ospf help so clients follow virtual machine to the other location. And should interruption between sites happen then at least next happens pve cluster split, 6 nodes site will continue ordinarily having quorum. 5 nodes remain unmanaged but virtual machines continue running. also happens drbd split. After network coming back it needs some cure.

I am mostly concerned not about network or controlling pve cluster situation but my main concern at the moment is this dbrb+scsi+lvm part of storage.


Imre