ZFS RAID

[khataev]

Hi all!

I want to understand on high level how to arrange mirroring RAID using ZFS in proxmox and how to do it step by step. So according to documentation:

RAID1

Also called “mirroring”. Data is written identically to all disks. This mode requires at least 2 disks with the same size. The resulting capacity is that of a single disk.

1. I will be adding disks incrementally, so at first only one disk would be available and no RAID, I want to setup Proxmox and my machines. Then, I plan to add second disk and create RAID and mirror my data on second disk. Would it work like this?
2. If yes, If I understand correctly, since proxmox is using some space itself (it is creating a separate volume). Based on my small experience I thee that it creates LVM Thin 100gb volume and volume with type Directory (I have experience with default LVM setup, not ZFS yet). In case of RAID1, when it is created is it going to mirror both volumes (according to common sense it should)?
3. Can anyone describe how to organize everything step by step? At least in high strokes

Thank you in advance!

 

[Kingneutron]

> I will be adding disks incrementally, so at first only one disk would be available and no RAID, I want to setup Proxmox and my machines. Then, I plan to add second disk and create RAID and mirror my data on second disk

I have a script that illustrates how to do this. Start with 1-disk pool, attach mirror, then add another 2-disk mirror for additional free space + I/O speed.

https://github.com/kneutron/ansitest/blob/master/ZFS/zfsdemo-zredpool-probox-1-drive-to-4-raid10.sh

It will be easier if you separate Proxmox root / install from "data" ZFS pool. In fact I would recommend installing PMVE on Ext4 + LVM for easier maint + recovery if you need to replace the boot disk.

 

[khataev]

It will be easier if you separate Proxmox root / install from "data" ZFS pool. In fact I would recommend installing PMVE on Ext4 + LVM for easier maint + recovery if you need to replace the boot disk.

Thanks for the reply! Question regarding proxmox root install separation - does it mean I would need 3 disks at least? 1 for proxmox itself and 2 for data RAID? If yes, then it is not an option for me - my mini pc server supports only 2 internal drives... Do you think it would be still possible to have RAID?

Another question: ZFS usage is not an ultimate goal for me. I was considering it only because I've read it allows software RAID setup and proxmox supports it. But at the same time it uses a lot more memory. So question - Is it possible to build similar solution using LVM?

 

[Kingneutron]

Just a suggestion, but you could go with something like a Samsung T7 (fast and reliable) usb3 for PMVE boot disk, and use the 2 internal drives for a ZFS mirror. You can limit ZFS RAM usage - do a search on ' limit zfs arc size '. Very handy if your host only has ~8-16GB of RAM.

Adding an inexpensive L2ARC device could also help - I've been using PNY 32GB usb3 thumbdrives for the last few months and none have died yet. Easy enough to replace if they do.

 

[Dunuin]

1. I will be adding disks incrementally, so at first only one disk would be available and no RAID, I want to setup Proxmox and my machines. Then, I plan to add second disk and create RAID and mirror my data on second disk. Would it work like this?

Yes but you will have to do that manually via CLI and you will have to install PVE using ZFS as a singel disk raid0. Not using LVM.

2. If yes, If I understand correctly, since proxmox is using some space itself (it is creating a separate volume). Based on my small experience I thee that it creates LVM Thin 100gb volume and volume with type Directory (I have experience with default LVM setup, not ZFS yet). In case of RAID1, when it is created is it going to mirror both volumes (according to common sense it should)?

A LVM installation can't be turned into a ZFS mirror later.

3. Can anyone describe how to organize everything step by step? At least in high strokes

- Install PVE using ZFS raid0
- do something similar to paragraph "Changing a failed bootable device" but instead of a "zpool replace" you use a "zpool attach": https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration

And don't cheap out on storage. Enterprise/Datacenter SSDs with power-loss protection are highly recommended when using ZFS. And avoid everything that uses QLC NAND flash or shingled magnetic recording (SMR).