Native ZoL on proxmox nodes and live migration.

[acidrop]

hello and happy new year!

I have two nodes with latest proxmox installed, without HA enabled.
As vm storage I currently use ZoL(zfs on linux) on both nodes.
Each vm reside on separate ZVOL which is replicated with DRBD on both nodes.Thus DRBD sits on top of each zvol and each vm is a separate DRBD resource.It seems to work well.I have also enabled auto-snapshot on zvols on each node.
I would like to ask if this kind of setup is ok to use in production or it is better recommended to have each vm in separate zfs dataset instead,replicated with glusterfs for example for live migration?
In any case I would like to have the option to make zfs snapshots on each node so I can rollback each vm separately in case of failure.

thanks for any thoughts.

 

[wahmed]

I dont have much knowledge on DRBD and ZOL. But given your situation at this moment, 2 nodes Proxmox+Ceph or Proxmox+Gluster seems to be better idea in my opinion. I would prefer Ceph probably because i already have working knowledge. Do you have plan to grow your Proxmox cluster? If yes, i think Proxmox+Ceph is a good choice for a good start.

 

[acidrop]

Hello Wasim,

I have tested both Ceph and GlusterFS.Ceph on two node cluster does not perform well and it is not recommended anyway.For 3+ nodes setup with a lot OSDs, Ceph will be my first choice.I don't believe that this cluster will grow at least in the near future, so I would prefer to use a technology that fits better to this kind of setup like DRBD for example.
GlusterFS performed better than Ceph on two nodes, but in tests that I made , I experienced VM file corruption issues which scared me a bit.Another factor is the layers of complexity and overhead.On Gluster there is Disk -> ZFS -> Dataset -> VM_File (raw,qcow2 etc) -> GlusterFS. On DRBD there is Disk -> ZFS -> ZVOL -> DRBD -> LVM.
I'm using the latest option (DRBD), but I would like to know if somebody else has tried such setup in production.As I see Proxmox currently supports ZFS only as an external storage via iSCSI.Is native ZoL also a supported scenario?

 

[mir]

ZFS on the nodes is not supported. IMHO your setup is to complex for me to use in production since to many things can break. Apart from that using ZFS over DRBD and LVM is really not a god thing and is something to avoid. ZFS works best, an in my opinion only properly, if ZFS have full control over the disks.

 

[acidrop]

Apart from that using ZFS over DRBD and LVM is really not a god thing and is something to avoid. ZFS works best, an in my opinion only properly, if ZFS have full control over the disks.

Actually I have done the opposite.ZFS has control over disks and DRBD sits on top of each ZVOL.So, each VM lives on separate DRBD resource -> ZVOL.Each ZVOL is snapshotted automatically by using zfs-autosnapshot.
If something goes wrong I have to bring DRBD resource down.Do ZFS rollback on ZVOL's previous snapshot and after that i bring DRBD again back again and initiate full sync from "good" side to the opposite site.
I agree though that this setup brings enough complexity but it seems to work good till now.