Proxmox ZFS + DRBD - zvol per VM?

[bravo]

I am trying to setup a two node Proxmox cluster with ZFS and DRBD. I read a couple of comments in the forum about configuring one zvol per VM. Is that a must? If so, how does it work for LXC?

Also, is there an official guide/wiki for setting up ZFS + DRBD on Proxmox?

thanks

 

[LnxBil]

ZFS is not a cluster aware file system, so you cannot open it (safely) on both nodes. This is crucial and has a lot of implications. Best is to use ZFS on each server and do asynchronous replication (if you want to use ZFS) or use LVM on DRBD.

 

[bravo]

Thanks LnxBil. I was thinking of using zfs beneath DRBD block device. What kind of implications are there in this config? Can you please give more details?

 

[bravo]

Another question. Is LXC supported on this setup with DRBD + ZFS?

 

[LnxBil]

Why do you want to use ZFS beneath DRBD? You will only use it as block storage and the real disk is a much better block storage for another storage tier on top of it.

You will have the following system for LXC:

LXC on ext4 on LV on VG on PV on DRBD (sync) on ZVOL on disk

If you would use asynchronous replication with ZFS you will have this:

ZFS on disk (sync to) ZFS on disk

 

[bravo]

Thanks LnxBil. I am setting up a primary-primary sync on a two node cluster so that VMs can be live migrated or at least shutdown on one and started on the other. Storage sync will be on a dedicated 10GbE network.

Is ZFS async replication reliable for immediate DR? Most replication strategies I found were based on scripts that you run continuously or at frequent intervals using rsync or similar. That is the reason I was trying to do an always active primary-primary replication system like DRBD or GlusterFS.

Do you have a recommendation for a reliable almost-realtime async replication script/program?

 

[LnxBil]

If you want to use ZFS on both nodes, use the Proxmox VE ZFS sync (https://pve.proxmox.com/wiki/PVE-zsync), otherwise use "real" synchronous block writes with DRBD or GlusterFS, but without ZFS.

 

[bravo]

Thanks for answering my persistent queries. I certainly understand the you are recommending against using ZFS underneath a clustering/replication engine. Can you please indicate the issues I might run into with this setup? I have not seen this warning anywhere else in my research, and people have reported success with it. I would like to know more about the implications before I run into serious troubles.

Thanks again.

 

[robhost]

Do you have a recommendation for a reliable almost-realtime async replication script/program?

You could test inotify based tools like lsyncd (https://github.com/axkibe/lsyncd).
But this does not work with millions of files very well.

 

[LnxBil]

Why do you want to use DRBD on top of ZFS? Is not a question why not, it's a question why you want to do that. Why has it to be ZFS? Of course you can do that, you can almost do anything you want on Linux.

 

[bravo]

ZFS is for protection against data-corruption, compression and block level de-dup. DRBD or GlusterFS is for primary-primary replication to the second node for migration (live or power-down) of VMs and containers. DRBD is supposed to be better than GlusterFS for large files.

 

[LnxBil]

That's all correct, but you'll also have a heck of a slow I/O path. DRBD itself it not that fast, but with ZFS on top (with compression and deduplication) it'll be very slow. Of course you can do that, but be aware of the write penalty (also the write amplification if you do not optimize everything for the correct block size).

(Only a side node: a HA setup is also not possible, you need 3 nodes for that)

 

[bravo]

Thanks LnxBil. I understand the ZFS and DRBD overhead. I am testing it to make sure that it meets my requirements. Any other implications you can think of other than I/O overhead?

 

[hansm]

Thanks LnxBil. I understand the ZFS and DRBD overhead. I am testing it to make sure that it meets my requirements. Any other implications you can think of other than I/O overhead?

You're overcomplicating things. While it's possible doesn't mean it's good. You haven't mentioned your requirements and available hardware (storage configuration), so it's very hard to mention an alternative but what you're doing might not be the best way to go. More layers in the stack increases the likelyhood something fails. Try to follow the KISS principle

 

[bravo]

Thanks LnxBil for the answers. The requirement is to setup a two nodes Proxmox cluster with live/quick migration of VMs for DR. Each node has dual E5-2630v3, 128GB RAM, 4x800GB Intel S3510 SSD and Intel X540 10GbE (dual port). Storage traffic is through a dedicated direct connection between nodes.