Use Cases for 3-Way ZFS Mirror for VM/LXC Storage vs. 2-Way Mirror?

[SInisterPisces]

Hello,

I'm in the process of building up a new Proxmox server in my small home office, and this motherboard has 3 NVME slots. My boot disks are an enterprise SATA SSD pool, so I'm free to use the NVME however I want for data storage.

Previously, I've had good success with storing VM and LXC containers on a two-NVME mirror. Would it be worth it to build a 3-way NVME mirror for storage, since I've got the extra NVME slot? I don't think the enhanced read performance would really help me; 2x NVME read speed is already fast enough.

I thought about putting all three into a RaidZ1 for VM storage, but even with PCIe 4.0x4 NVME speeds, I'm wary of using a Z1 for VM/CT storage.

I'm also planning to virtualize an instance of PBS. If I go with a 2 NVME mirror, I think I'll use the third slot to give PBS a real NVME drive to boot from.

 

[leesteken]

I use 3-way mirrors for my backups (PBS). If it were a 2-way mirror then my data would be at risk of losing everything as soon as one side of the mirror fails.
I use a 2-way mirror for everything else to prevent interruption of service when one side fails (but I do regular backups to prevent data loss).
In my opinion that is the difference between two and more mirror configurations: two devices is for continuous operation, more than two devices/copies is for data safety.

 

[Johannes S]

IIRC a three-way mirror has also more IOPS than two-way

 

[leesteken]

IIRC a three-way mirror has also more IOPS than two-way

Indeed! For PBS I combined 2 HDD with 1 SSD in a 3-way mirror (even though people recommend against mixing such different drives) and there was a huge improvement in the read IOPS (especially when looking at the contents and garbage collection). But in general a 3-way mirror has 50% more read IOPS than a 2-way mirror.

 

[SInisterPisces]

IIRC a three-way mirror has also more IOPS than two-way

Thanks for mentioning this! I was wondering about it.

That's just for reads, though, right? I can't imagine how adding another disk would result in faster writes.
EDIT: My eyes glitched. I missed the last part of @leesteken 's message. 50 percent better read IOPS is a huge deal.
That does does like a great setup for PBS.

I still need to decide if I'm going all in on shared, networked storage for VM and LXC on this PVE node. If I am, then I can use the three NVME slots in the box for a three-way PBS pool.

 

[dylang]

Indeed! For PBS I combined 2 HDD with 1 SSD in a 3-way mirror (even though people recommend against mixing such different drives) and there was a huge improvement in the read IOPS (especially when looking at the contents and garbage collection). But in general a 3-way mirror has 50% more read IOPS than a 2-way mirror.

The wild thing here is that someone recommended not using a hybridized approach. If you have 3 drives, you have a quorum that enables you to know if files are corrupt, as well as having 2 copies if 1 drive fails. The fact that they have different performance doesn't cause it to fail or otherwise not function. That's what RAID is. That's why hybrid HDD/SSDs are also a product; you gain the benefits of both -- lower cost with HDD spindles for the data, and higher performance with flash for the active-set of data reads. That's the same thing (on a 1-drive level), as your 3-way RAID with 2x HDD spindles and 1xSSD (except better, because 3x1TB (so 3TB of the same 1TB) has the entire 1TB at the speed of SSD for read, plus you have 3x redundancy of data (1-2 drives can fail, and you're still safe).

Using multiple disks is a logical solution that enables better performance. The drives store data, and the mechanism doesn't matter *because that is the point of such an abstraction*; there is an interface which is standard which hides the details. For example, I can use a file path -- it doesn't matter if the path refers to a file on an SSD, HDD, or even a tape drive. The OS works at the level of a filesystem and file path (its VFS layer), and the drives are below that; I just use the file path, and I don't have to think about cylinders/heads/sectors, etc -- that's what the VFS layer of the kernel is for.

Those people didn't understand the technology they were working with. Bad recommendation

( https://en.wikipedia.org/wiki/RAID ;; "Each scheme, or RAID level, provides a different balance among the key goals: reliability, availability, performance, and capacity.")

 

[SInisterPisces]

I was really sick in November, and then Christmas and New Year happened, so I'm still putting this server together. After this great thread, I know I should do the 3-way mirror just to get the best performance, but at this point that means I'd have to buy at least 2 NVMEs, and … that doesn't feel like a fun thing to do right now, with prices being so bonkers.

I'm toying with the idea of learning how to do multipath NFS or ZFS over iSCSI so I can get 20 Gbps (2x10 GbE) to my NAS for network storage, but in terms of performance and simplicity I'd still prefer mirrors.

(My preferred Sabrent Rocket 4.0 NVMEs in a 2x mirror get about 3.3 Gbps read according to hdparm, so the network storage option would be about ⅓ slower on read, theoretically.)

 

[LnxBil]

A similar wording than @leesteken argument, which sold it to me:
A threeway-mirror buys you time to get to the action of changing the disk, which can be difference between getting waked up at night due to a disk failure and just replacing it in the morning - depending on your HA requirements.

 

[SInisterPisces]

Now I just need to find three 2 TB NVME that fell off the back of a truck or something. Even the open-box stuff I'm finding has spiked in price.

 

[alexskysilk]

The wild thing here is that someone recommended not using a hybridized approach. If you have 3 drives, you have a quorum that enables you to know if files are corrupt, as well as having 2 copies if 1 drive fails.

This isnt valid for zfs. ZFS will simply replace any read with a failed checksum and replace it on the affected vdev.

That's why hybrid HDD/SSDs are also a product; you gain the benefits of both -- lower cost with HDD spindles for the data, and higher performance with flash for the active-set of data reads.

EXCEPT this didnt actually work, which is why you dont see these anymore.

and the mechanism doesn't matter *because that is the point of such an abstraction*

abstraction doesnt change the underlying device performance, latency, or lifecycle characteristics. You can make a RAID volume on thumbdrives and it will still be slow as molasses. Your statement is only true if you dont care about what the storage actually does.

I use 3-way mirrors for my backups (PBS). If it were a 2-way mirror then my data would be at risk of losing everything as soon as one side of the mirror fails.

If survivability is paramount, mirrors arent your go to anyway. that's what parity raid is for (eg, raidz2/z3.) also, I'm surprised no one mentioned that RAID # backup; raid survivability is applicable to your primary data. your backups should follow 3-2-1 methodology- dont waste too much effort on individual media fault tolerance.

Indeed! For PBS I combined 2 HDD with 1 SSD in a 3-way mirror (even though people recommend against mixing such different drives) and there was a huge improvement in the read IOPS (especially when looking at the contents and garbage collection). But in general a 3-way mirror has 50% more read IOPS than a 2-way mirror.

People recommend against this because it makes the performance unpredictable. I understand that this is ok for your usecase but its worth noting. BTW, now that zfs 2.4 is shipping, I would redeploy the SSD as a special device (yes, you should add one more.)

EDIT: My eyes glitched. I missed the last part of @leesteken 's message. 50 percent better read IOPS is a huge deal.

the NUMBERS matter. if you're going from 100 IOPs to 150, I doubt you'll notice anything. changing your drives to SSD and going to 4000IOPs would be a lot more noticeable. and note that in context of conversation, we're discussing read iops- restoration from PBS is largely a data stream and iops dont come into play very much.

 

[leesteken]

If survivability is paramount, mirrors arent your go to anyway. that's what parity raid is for (eg, raidz2/z3.)

Probably, but in my case it's only 1TB of backups and this gave me both a read improvement and a third copy. Eventually it is via multiple locations, as you also pointed out.

also, I'm surprised no one mentioned that RAID # backup; raid survivability is applicable to your primary data. your backups should follow 3-2-1 methodology- dont waste too much effort on individual media fault tolerance.

My backups are on two different media (due to the mix of SSD and HDD on two near systems), and also in three different locations: local, near and half-way across the world (but I did not mention that). I fully appreciate the 3-2-1 backup principle, which is mentioned often on this forum, but you're right in that it was not mentioned here. Good that you pointed it out.

People recommend against this because it makes the performance unpredictable. I understand that this is ok for your usecase but its worth noting. BTW, now that zfs 2.4 is shipping, I would redeploy the SSD as a special device (yes, you should add one more.)

Probably, but this killed two concerns with just one SSD. It's indeed a fair trade-off for my particular use-case.
I agree that three stones for one bird is a better strategy than hoping to get two bird with one stone (similar proverbs exits in different languages). I would be more diligent in a professional situation but this is a "lucky" coincidence for my home setup.