Assumption: you use at least four identical devices for that. Mirrors, RaidZ, RaidZ2 are possible - theoretically.
Technically correct answer: yes, it works. But the right answers is: no, do not do that! The recommendation is very clear: use “striped mirrors”. This results in something similar to a classic Raid10.
(1) RaidZ1 (and Z2 too) gives you the IOPS of a single device, completely independent of the actual number of physical devices. For the “four devices, mirrored” approach this will double --> giving twice as many Operations per Second. For a large-file fileserver this may be not so important, but for multiple VMs running on it concurrently as high IOPS as possible are crucial!
(2) It is a waste of space because of padding blocks: Dunuin has described that problem several times, an extreme example for RaidZ3 : https://forum.proxmox.com/threads/zfs-vs-single-disk-configuration-recomendation.138161/post-616199 “A 8 disk raidz3 pool would require that you increase the block size from 8K (75% capacity loss) to 64K (43% capacity loss) or even 256K (38% capacity loss)“
There seem to be some counter arguments against “only mirrors”:
(3) Resiliency: "I will use RaidZ2 with six drives to allow two to fail. Mirrors are less secure, right?"
Yes. In a single RaidZ2-vdev any two devices may fail without data loss. In a normal mirror only one device may fail.
BUT: there are triple mirrors! These are being so rarely discussed that I need to mention them here explicitly. Let us compare that RaidZ2 with six devices:
(3a) the RaidZ2 will give us the performance of a single drive and the usable capacity of four drives. Two drives may fail.
(3b) the two vdev with triple mirrors gives us the IOPS of two drives for writing data + six fold read performance! Any two of each vdev may fail! (So up to four drive may die - but only in a specific selection.)
(4) Capacity: the only downside of (3) is that the capacity shrinks down to two drives.
Recommendation: for VM storage use a mirrored vdev approach. For important data use RaidZ2 or RaidZ3.
In any case note that “Raid” of any flavor and/or having snapshots does not count as a backup. Never!
See also:
Technically correct answer: yes, it works. But the right answers is: no, do not do that! The recommendation is very clear: use “striped mirrors”. This results in something similar to a classic Raid10.
(1) RaidZ1 (and Z2 too) gives you the IOPS of a single device, completely independent of the actual number of physical devices. For the “four devices, mirrored” approach this will double --> giving twice as many Operations per Second. For a large-file fileserver this may be not so important, but for multiple VMs running on it concurrently as high IOPS as possible are crucial!
(2) It is a waste of space because of padding blocks: Dunuin has described that problem several times, an extreme example for RaidZ3 : https://forum.proxmox.com/threads/zfs-vs-single-disk-configuration-recomendation.138161/post-616199 “A 8 disk raidz3 pool would require that you increase the block size from 8K (75% capacity loss) to 64K (43% capacity loss) or even 256K (38% capacity loss)“
There seem to be some counter arguments against “only mirrors”:
(3) Resiliency: "I will use RaidZ2 with six drives to allow two to fail. Mirrors are less secure, right?"
Yes. In a single RaidZ2-vdev any two devices may fail without data loss. In a normal mirror only one device may fail.
BUT: there are triple mirrors! These are being so rarely discussed that I need to mention them here explicitly. Let us compare that RaidZ2 with six devices:
(3a) the RaidZ2 will give us the performance of a single drive and the usable capacity of four drives. Two drives may fail.
(3b) the two vdev with triple mirrors gives us the IOPS of two drives for writing data + six fold read performance! Any two of each vdev may fail! (So up to four drive may die - but only in a specific selection.)
(4) Capacity: the only downside of (3) is that the capacity shrinks down to two drives.
Recommendation: for VM storage use a mirrored vdev approach. For important data use RaidZ2 or RaidZ3.
In any case note that “Raid” of any flavor and/or having snapshots does not count as a backup. Never!
See also:
- https://pve.proxmox.com/wiki/ZFS_on_Linux
- https://forum.proxmox.com/search/7994094/?q=FabU&c[title_only]=1&c[users]=UdoB&o=date - some more of this series of posts may be added over time...
