Hello all. I have done a lot of looking, but haven't found exactly what I am looking for. I have been using Proxmox standalone for a number of years, but have never tried clustering. I have a site that I am wanting to install Proxmox on 3 different machines, but have failover between them, without shared storage (different locations on campus). I have read a lot about using ZFS for this, but I keep going down rabbit holes on what settings should be, and have read things about using mirrored vdevs instead of just the zpools, but I am not sure excatly what that looks like to set it up properly. I have plenty of drives and everything is on HBA controllers as well. Again, this is a learning experience for me, and just appreciate any help that can be provided!
Proxmox and ZFS Storage
- Thread starter fish2222
- Start date
First things first: you will loose data on failover in this scenario because you have to asynchronously replicate the data. If you can live with this contrainst, then go with it.
you mean mirrored device vdevs vs. single device vdevs, you always have a zpool, which has at least one vdev. Mirroring is similar to RAID1, so you can have one disk that fails and still have the data. If you have more than one vdev, you have a stripe of vdevs, so similar to RAID0. If you have 4 disks, you can have two mirrored vdevs and that's like RAID10.
Mirroring is not required, but will help in case of a disk failure, which has the statistically highest probability to fail.
Since the VM's that will be running at the Phone PBX, the HVAC control server, and Quickbooks. They all can survive data loss, which I understood when I started this project.
On the mirrored device vdevs, essentially what I read is that you should set up mirrored drives instead of a Raid 5 (raidz) with multiple disks for your pools, correct? I have 8 2TB drives, 2 that are already mirrored for Proxmox itself, and the other 6 are for the VM's.
On the mirrored device vdevs, essentially what I read is that you should set up mirrored drives instead of a Raid 5 (raidz) with multiple disks for your pools, correct? I have 8 2TB drives, 2 that are already mirrored for Proxmox itself, and the other 6 are for the VM's.
Yes, N-way (stripes of) mirrors typically perform better for VMs than raidZ1/2/3 because they give more IOPS. However, if you already have redundancy by running multiple nodes, redundancy inside each node might be less important. Proxmox itself only needs 8GB according to the installer, so unless you want to store other stuff there like ISOs, templates and/or VMs, 2TB sounds like overkill.
As mentioned before, setup Proxmox's ZFS replication to keeps the various nodes relatively up to date for redundancy. This is independent of the ZFS configuration for each Proxmox node, except that the nodes should use the same sizes and (storage) names for the ZFS pools that need to be replicated.
As mentioned before, setup Proxmox's ZFS replication to keeps the various nodes relatively up to date for redundancy. This is independent of the ZFS configuration for each Proxmox node, except that the nodes should use the same sizes and (storage) names for the ZFS pools that need to be replicated.
Generally it is recommended to go with mirrors and not zRaid 5. (If you can swallow the disk size loss and total disks as mirrors are basicly raid 1.)
This is due to both the complexity it brings to users who are new or less experienced to ZFS and zRaid 5 will also eat non enterprise (and to a lesser extent enterprise) disks for lunch.
This is due to zRaid 5 adding a lot of write amplification to the disks (even when configured correctly) and also even more writes for the check data.
And even though mirrors do not offer any write speed advantages, it does offer read advantages. (Up to 2x as it can read two disks in parallel but your milage will vary depending on use case and the data itself.)
This is due to both the complexity it brings to users who are new or less experienced to ZFS and zRaid 5 will also eat non enterprise (and to a lesser extent enterprise) disks for lunch.
This is due to zRaid 5 adding a lot of write amplification to the disks (even when configured correctly) and also even more writes for the check data.
And even though mirrors do not offer any write speed advantages, it does offer read advantages. (Up to 2x as it can read two disks in parallel but your milage will vary depending on use case and the data itself.)
Last edited: