Migrating from mdadm to zfs despite using different disk sizes?

[jokergermany]

I am Proxmox newbie and using a complex mdadm + lvm(raid) szenario.
I read that I can use mdadm with proxmox but it's not supported. ZFS is recommended.
The Problem is i don't have disk with the same size:
18TB, 10TB , 2x4 TB
At the moment i have several 2TB slice mdadm raids with different "classes"
VG RaidSpare - 2 mdadm Raid 5 with each 2TB on every disk. One of the disks are spares.
VG Raid - 2 mdadm Raid 1 with each 2TB on 10TB and 18TB disk
VG nonRaid - All the other free diskspace

"To make it a little bit more complex" really important data has a LVM Mirror between RaidSpare and Raid.
Every other is just normal LVMs.
As filesystem i am using ext4,

Is ZFS for my usecase a good idea?
I read that it's better for ZFS using the whole disk but i read aswell that ZFS can't handle different disk size. (wasted space)
If i switch to ZFS it looks like i should do a similar split like with mdadm.

1. Am I correct?
2. Would i have Advantages using zfs instead of mdadm?

As explanation why the splits:
Therefore i can add new "parts" when i switch a disk with a bigger one.

 

[UdoB]

ZFS can't handle different disk size. (wasted space)

Well, you can put different sized disks into one pool. "Striped". This is NOT recommended because it removes some very important features!

A mirror (and that's what we mostly use in PVE context) needs two (or more) same sized disks.

You can force it to use different sizes like your 18+10 disks, with the effect that only the smaller size will effectively be usable. This may be helpful if you want to build something now and buy a better fitting disk, of the size of the larger one, later. Then you can replace the smaller one and have full capacity. This mechanism works without difficult maneuvers.

18TB, 10TB , 2x4 TB

Everything HDD? Everything SSD?

In PVE context you could build a striped mirror with effectively 2*10 + 2*4 = 28 TB, mirrored --> nominell 14. Minus overhead ~ 1TB = 13 TB. It is recommended to not fill this more than 80%. (Maybe 90 for some workloads, but not for VMs.)

If it is 18+10 HDD + 2*4 SSD I would recommend to install on HDD (mirrored) and later add those SSDs as a "Special Device", mirrored - for Metadata and for "Small Blocks".

Would i have Advantages using zfs instead of mdadm?

Sure! LVM does NOT: transparent compression, (technically) cheap snapshots, replication, guarantee integrity and some more things. From: https://forum.proxmox.com/threads/f...y-a-few-disks-should-i-use-zfs-at-all.160037/

Have fun

 

[jokergermany]

Everything HDD? Everything SSD?

Everything HDD
I want to prevent striping.

Thats the reason why i think about using the a similar model like now but instead of using mdadm i would use zfs

 

[UdoB]

I want to prevent striping.

A ZFS pools consists of one or more vdevs. Vdevs implement the known topologies: single drive/mirror/RaidZx. When you have more than one vdev the combination in a pool is called "striped". (Nearly) every ZFS pool I have has more than one vdev --> the pool is striped.

IOPS scale linear with the number of vdev. Especially when you have rotating rust you want to have as many vdevs as possible - all striped!

Edit: think of it being similar "Raid10".

 

[jokergermany]

with mdadm i used 4 devices with Raid 5. One of them was a spare. (because Raid 6 is very resource intensive )
1. Is a RaudZ2 (which should be Raid 6) resource intensiv?
2. Is a RaidZ1 (which should be Raid 5) with a spare possible?

zpool add -f spare <device>

 

[UdoB]

Is a RaudZ2 (which should be Raid 6) resource intensiv?

What do you mean by "resource intensiv"? CPU load? Slow transfer speed? Not many IOPS? Power consumption?

For HDD the limiting factor is always the mechanical behavior of the disks, never the CPU. At least not for the last few decades. If I am offered RaidZ1 plus spare I would always opt for RaidZ2. This way the redundancy level is raised.

A spare would just shorten the delay for an attempt to repair/replace the faulty disk. Depending on the kind of failure the RaidZ1 has zero redundancy during the repair/replace process. The risk for an additional read error is (relatively) high as all the data has to be read with zero errors to rebuild the single parity. This chance can be eliminated by utilizing RaidZ2.

The IOPS of RaidZ1/Z2/Z3 (with or w/o spare) are always nearly identical to a single drive. The CPU load is completely neglectable.

I am not sure how this works for superfast NVMe though. Having a million IOPS instead of 200 may shift the limiting resource.

 

[jokergermany]

Thanks.
Think about making a raid on top of a raid, because it safed me in the past because i made a configuration mistake...
A RAID 1(RAID 6(RAID 1)).
There are two things which I think of:

1. Using ZFS on top of ZFS
2. Using an LVM Mirror on top of ZFS.

What do you think?

 

[UdoB]

1. Using ZFS on top of ZFS
2. Using an LVM Mirror on top of ZFS.

Yes, technically everything is possible as "everthing is a file". You can stack filesystems on top of block devices as usual and then repeat this step as often as you want to. (Or an LVM physical volume on top of a ZVOL.) The complexity multiplies on each step. And so does the difficulty to repair something when (not: if) something fails...

What do you think?

You can test it. The fact that this is possible is one of the great aspects of Linux! For an experiment or to write a blog article; for a productive setup *I* would never do that.

(( I would stick with ZFS, of course. If there is no redundancy possible I would make sure that I run Backups with a higher frequency than usual - depending on the "Plan B" to restore everything from scratch. ))

Have fun!