HA Replication via ZFS Datasets?

[sdet00]

Howdy folks. I just wanted to quickly check that my thinking is correct before I take the plunge with trying to get this setup. At the moment I have 3 Proxmox 8 nodes in a cluster, and I wish to setup VMs and storage that will replicate between Node 1 & 2. Node 3 is just for quorum / monitoring.

Node 1:

Xeon E5 v3 6-core
32GB ECC RAM
2x256GB SATA SSDs in ZFS Mirror (Boot)
3x2TB NVMe SSDs in ZFS RAIDz1 (Production Data - 4TB Usable, 1.5TB used)
3x8TB SATA HDDs in ZFS RAIDz1 (Archive Data - 16TB Usable, 2.5TB used)
2x 1GBe NICs

Node 2:

Xeon E5 v4 4-core
32GB ECC RAM
2x256GB SATA SSDs in ZFS Mirror (Boot)
3x18TB SATA HDDS in ZFS RAIDz1 (Replication Destination for Production + Archive Data, as well as Proxmox Backup Server Data - 36TB Usable, not setup yet)
2x 1GBe NICs

Node 3:

Intel Core i5
16GB RAM
1x 256GB SATA SSD


My question is as follows - is it possible to setup replication between storage in Node 1 and Node 2, despite the fact I have a very different storage configuration? If so, is it still possible to use any leftover storage for an extra set of backups? I believe it is technically possible, but I am unsure how to achieve this - it appears that Proxmox only replicates at an entire zpool level. Any suggestions would be appreciated.

 

[LnxBil]

it appears that Proxmox only replicates at an entire zpool level.

On the zfs level only, yes. It doesn't care if there are multiple pools, but it has to be ZFS, because the heavy lifting of the replication is done on ZFS level.

 

[sdet00]

On the zfs level only, yes. It doesn't care if there are multiple pools, but it has to be ZFS, because the heavy lifting of the replication is done on ZFS level.

I'm afraid I am a bit confused by this response. ZFS is a filesystem, I understand that. What I am trying to determine is whether it is possible to have a zpool with multiple datasets, where some datasets are replicated to other hosts with different storage configurations, and some are not, and have that functioning within Proxmox, ideally with High Availability. I would also be happy with a setup where it requires manually starting / stopping VMs in the event of a failure, but I would prefer something automated.

For examples:

/n1-zpool1/production
/n1-zpool2/archive

Replicated to:

/n2-zpool1/production
/n2-zpool1/archive

With the additional dataset for a secondary backup server:

/n2-zpool1/backups

Happy to explore alternatives, this system isn't in production yet, but it will be soon.

 

[LnxBil]

ZFS is a filesystem, I understand that.

Sorry for the confusion. I just meant to say that you require ZFS on both sides, which you have, so that's fine.

AFAIK, the name of the zpool on both sides have to be identical in order to work properly. At least this was the case in the past and I did not find any news on this.

 

[sdet00]

Well I don't think it's possible to have two zpools on the same system that are the same name, unless I am mistaken? In which case I may have to concede the ability to have HA for archive data? A plan I don't really like but will run with if I have no other choice

 

[LnxBil]

Well I don't think it's possible to have two zpools on the same system that are the same name, unless I am mistaken?

Yes, of course, yet on different nodes the names can be the same.

I personally go with the original intention being ZFS to have only ONE pool with all the storage and the best performance.

In which case I may have to concede the ability to have HA for archive data? A plan I don't really like but will run with if I have no other choice

There will never be HA as in synchronous replication as with e.g. CEPH, you only have asynchronous replication. If you're fine with just that, setup send/receive between the nodes, yet you have to set it up manually and so regular snapshots in order to be able to send/receive.

 

[sdet00]

Yes, of course, yet on different nodes the names can be the same.

I personally go with the original intention being ZFS to have only ONE pool with all the storage and the best performance.


There will never be HA as in synchronous replication as with e.g. CEPH, you only have asynchronous replication. If you're fine with just that, setup send/receive between the nodes, yet you have to set it up manually and so regular snapshots in order to be able to send/receive.

As much as I'd like to just use one pool with two separate zvols, this doesn't make sense when you have SSD only and HDD only storage tiers. You would cripple the performance of the SSDs by throwing HDDs into the same zpool. Unless there is a feature I don't know about that lets you specify which zvol gets used for certain types of data.

I've decided to have one VM for production data, replicated, and another VM for archive data, not replicated, but easily restorable via Proxmox Backup Server. Archive data isn't as mission critical so it can be down for a day if there ever was a major failure. We'll see how it pans out.

 

[LnxBil]

As much as I'd like to just use one pool with two separate zvols, this doesn't make sense when you have SSD only and HDD only storage tiers. You would cripple the performance of the SSDs by throwing HDDs into the same zpool. Unless there is a feature I don't know about that lets you specify which zvol gets used for certain types of data.

Then look at special devices, which is a very good way to combine those two.

I've decided to have one VM for production data, replicated, and another VM for archive data, not replicated, but easily restorable via Proxmox Backup Server. Archive data isn't as mission critical so it can be down for a day if there ever was a major failure. We'll see how it pans out.

I thought you're talking about two pools, not two VMs. VMs are replicable with internal tools and should not be problem replication-wise speaking.