Does Replication + Mirroring makes sense?

[Gustavo Neves]

Hi all! Thanks for this amazing project.
I have been "studying" Proxmox for one or two months and I quite like it.

I have been wondering about storage replication and disk mirroring for a setup.
Does it make sense to have mirrored disks AND do storage replication on top of it.

For example, on a 3-node cluster where all nodes replicate to one another, you would end up having 6 copies of everything.

I am also not so sure on how to chain the replications on a 3-node cluster. Would replicate 1>2 and 1>3? Or 1 > 2 > 3? Maybe it depends?

For example, what would be safer (ignore the 3 node-cluster problem for this, consider them to be with other nodes on a bigger cluster) >
1 host with 2 disks mirrored OR 2 hosts with 1 disk each replicated ?

I guess that ZFS is not gonna be able to self-heal silent corruption if there is no mirroring, right? So if the corruption happened on the origin of the replication I would have to sync it back from the target replication node manually? And if the corruption happened on the target of the replication, just let the next replication job fix it?

Also, if we consider to be fine to have no mirroring locally (raid1) because we are replicating on one (or two) other disks, would it also be fine to have these local disks stripped (raid0)?

This a simple setup that will run just a few VMs. We will have two gigabit networks (one internal do the cluster and one to the main network). Would it make sense to think about GlusterFS instead of replication on this setup? The third node will probably be a bit weaker on the hardware side, but will be there anyways.

Havin HA would be cool, but not really a must. It is fine if we have to manually migrate or start a VM, as long as that doesn't take too long and we have no corruption or lost data.

Thanks in advance and sorry if this was posted somewhere else. I have been trying to find this info for some time and could not really find it anywhere.

Cheers!
Gus

 

[dietmar]

I guess it would be much easier to use ceph (distributed storage) instead of above setup ...

 

[Gustavo Neves]

I have read that Ceph requires 10gbit for being usable. It also adds in complexity. Or am I wrong?

 

[guletz]

I guess that ZFS is not gonna be able to self-heal silent corruption if there is no mirroring, right?

ZFS cand do seal-heal if it has redudundancy(aka mirror/raidzX)

So if the corruption happened on the origin of the replication I would have to sync it back from the target replication node manually

zfs replication is not succeded when the surce have some corruption. After the corruption is solved on source, the next replication task will be OK, without any manual intervention.

Also, if we consider to be fine to have no mirroring locally (raid1) because we are replicating on one (or two) other disks, would it also be fine to have these local disks stripped (raid0)?

Any kind of remote replication is not instant(synchron replication), so in such a case you could see some small data lose! So it is up to you if this scenario is acceptable in your case.

Would it make sense to think about GlusterFS instead of replication on this setup?

GlusterFS it do not have a very good performance(IOPs as example), and sometimes you can have some problem with it(for example when you need to resyncronise your gluster bricks).

I run like you a 3 node PMX cluster(separate Gbit network for storage/replication, each node with zfs mirror). In my case I use zfs send-receive(pve-zsync) from node 1,2 -> node 3.

 

[Gustavo Neves]

Thank you very much for the response.

For now we have just a single server running, was deployed today.

It has a single 500GB SSD using ZFS for the root disk which cointains a Windows7 VM with a postgres database (I know right!? I wish I could run that on linux but it can't be helped). The same machine has two 1TB disks mirrored in a ZFS pool. The idea is to replicate or zsync the SSD to the mirrored disks (at least for now). This windows server runs some accounting software, nothing too heavy or too big. Should also be fine to lose a few minutes of data (would basically need to redo, retype, reimport data, but nothing that would be lost to thin air).

I am still thinking about the other 2 nodes and their disk configuration. I will probably have everything mirrored anyways, even though it really seems a bit overkill to replicate mirrors on mirrors. I may need a better third node if I want to make it the dedicated backup machine as well (like you do). However, having VMs replicated from 1>2 and 2>1 would make easier/faster to migrate in case of emergency or maintenance.

The other nodes will receive mostly linux VMs/CTs which will replace stuff that currently runs on metal or are yet to be implemented.

So far the experience with Proxmox has been quite good.
Few problems here and there, but nothing that could not be researched and fixed.

I am still a newbie though.
Quite a lot of reading done already, glad there is some good material around here.
I confess it can be a bit overwhelming at times. So many options, so many decisions you can't change later without a lot of pain.

Funny fact is that I was going to use xcp-ng and I was reading on it for more than a month and then for some odd reason I decided to try Proxmox first instead and then all else is history. Maybe it is because Debian feels like home to me I haven't even tried xcp-ng in the end.

Using ZFS send / zsync is going to be awesome. Currently we make big tarballs every night (such a waste). We also currently need to put some machines in maintenance mode to do backups.

I am still not sure about my original question though.
Does it really make sense to have mirrored pools replicated to mirrored pools? (2 + 2 + 2 ... copies)

Cheers!
Gus