GlusterFS on ZFS Storage Model

[FuriousGeorge]

There are no official docs, and I'm not sure if I'm understanding how this should be set up.

Let's say I have two disks that are to be my shared volumes, I'm pretty sure I want to start of by:

• Creating zpool tank0 on these two volumes
• Creating a zfs volume for gluster fs in that tank: tank0/glusterfs
• Repeating the process on the second server
• Setting up a glusterfs share between the two servers at tank0/glusterfs

This is where it get's confusing for me. Normally, at this point, I'd add my ZFS volumes, but then I won't be able to live migrate. What I'm gathering is that I want to add the glusterfs share at that point instead, and then create (I think) a child pool within that for my ZFS storage. So I'd:

• Create ZFS volume pool0
• I add pool0 as a ZFS share in PM gui, now i have tank0/glusterfs/pool0
• Create VMs there, e.g.: tank0/glusterfs/pool0/subvol-112-disk-1

At this point, will I be able to live migrate the VMs, or will it still tell me they use local disks?

If instead of the above I just create the VMs directly on the Glusterfs volume, then won't it just take a regular fs inside the tank0/glusterfs volume, and store the disk there as image files, no ZFS shares?

It will take probably ten hours for me to backup and move VMs, and I would appreciate any suggestions / pointers in the meanwhile.

Thanks.

 

[LnxBil]

You cannot mount your secondary pool on both servers, ZFS is not a cluster filesystem and will prohibit it. You have to store the files inside your glusterfs and use e.g QCOW2. Besides having two separate ZFS pools, I cannot see any advantages of this. Both pools are self-sufficient and not a "actual" copy.

If you want ZFS HA, please have a look at the article series of the BSD Magazine on dual Controller ZFS.

https://bsdmag.org/download/reusing_openbsd/

I read it but haven't tried it. It is not production ready and more of a proof of concept.

 

[FuriousGeorge]

Besides having two separate ZFS pools, I cannot see any advantages of this. Both pools are self-sufficient and not a "actual" copy.

I wanted to set up HA locally, but I also wanted to try to use PVE-zsync with my remote DPS.

You cannot mount your secondary pool on both servers, ZFS is not a cluster filesystem and will prohibit it. You have to store the files inside your glusterfs and use e.g QCOW2.

Can you elaborate as to why it won't work? I see documentation like here and anecdotes like this one from PM forums, and I wonder what they are doing differently than what I want.

 

[LnxBil]

I wanted to set up HA locally, but I also wanted to try to use PVE-zsync with my remote DPS.

What do you mean by HA locally?

Can you elaborate as to why it won't work? I see documentation like here and anecdotes like this one from PM forums, and I wonder what they are doing differently than what I want.

GlusterFS provides a clustered filesystem. What you do inside is available to every node, so that is what you want. Yet you cannot create ZFS on top of that because ZFS is not a cluster filesystem:

tank0/glusterfs/pool0 will be still local.

 

[FuriousGeorge]

What do you mean by HA locally?

I just said "locally" to distinguish from pve-zsync, which would involve a remote server.

GlusterFS provides a clustered filesystem. What you do inside is available to every node, so that is what you want. Yet you cannot create ZFS on top of that because ZFS is not a cluster filesystem:

tank0/glusterfs/pool0 will be still local.

I see what you mean. It seems like I could do the inverse: create glusterfs on top of ZFS.

I have the node set up, and I'll hopefully get a chance to experiment later today. I'm wondering if I could have a local ZFS for my VMs, and a glusterfs share of it for "manualy HA". I'm not looking to automagically roll it over if a node goes down. Being able to do it manually would be enough. I'm assuming it won't work to set up or use from the GUI.

 

[hybrid512]

Personnaly I would rather go Ceph instead of GlusterFS (even based upon ZFS) ... it features everything you need (live migration, snapshoting, thin provisionning, high availability, ...) and with ProxMox, it is rather easy to setup and well integrated.
I would also say that Ceph gives far better performances that GlusterFS for VMs disks storage (at least with XFS as underlying FS ... didn't tried Gluster on top of ZFS).

 

[LnxBil]

ZFS is in general slower on the same hardware if used only as file or block storage, so GlusterFS will not be faster on ZFS as it is on XFS.

 

[FuriousGeorge]

Thanks for the reply guys. I was actually looking into cephs earlier. I have minimal experience with gluster, but none with cephs, so I may learn something as an added bonus.

I may end up ditching ZFS all together on the test server. I run some VMs with GPU passthrough, and ZFS seems to cause the GPU driver to crash when multiple users are logged in.

I ran outta time for this week, so I'll see about it on Monday.