Proxmox and storage option for a HomeLab

[l0rdraiden]

Before posting this I have been reading a lot about proxmox and zfs, please keep in mind that I'm not an expert on linux, I'm an enthusiastic home user with IT knowledge.

I plan to setup a system based on an apollo lake board J5005, 8GB RAM, 1SSD (128GB) and probably 2 or 3 identical HDD of 4TB. It won't have HW RAID

The setup I have in mind is
VM1 PFsense or Sophos XG Firewall
VM2 NextCloud
VM3 OpenMediaVault (not sure since I have NextCloud), rtorrent and other small apps.

I'm not an expert so I don't want to over-complicate things

I plan to have VM images, log, ZFS cache (if I use it) on the SSD.
My idea is to use the HDD as a file storage an backup and consume it with Syncthing, NextCloud, Samba, Duplicati, etc. So redundancy is important but I could also copy everything again to a 6TB external drive (enough space for all my data), and purchase only 1 HDD of 4TB
¿So what is the best thing I can do with the 1, 2 or 3 HDD drives amusing I get similar drives?

It make sense to install OpenMediaVault in a VM to create and Manage ZFS storage? will the VM have access to the RAW disk or it will go trough the Promox layer impacting performance?
Should I create ZFS directly at promox level? RAIDZ1 or 2

How hard is to recover a ZFS if 1 HDD dies? I'm afraid about this in case is hard, since with HW Raid 1 is quite easy.

This thread is exposing basically a similar questions, I have read it but still I don't get all the answers.
https://forum.proxmox.com/threads/basic-setup-for-simple-home-server-where-to-store-the-data.31700/
He end up using the option of creating ZFS at proxmox level and then share it with VM using NFS but it looks like to me like and bad way to do it, why use a virtual network to share it? Proxmox can not expose directly the ZFS storage to the VM?

 

[guletz]

Hi,

Many questions, without a good structure...
Anyway, your RAM is very small for your goals. I would go with 2x4 TB for zfs-mirror only(aka raid1). And I will install proxmox on ssd only.
I can guess you will not need open-media-vault (you could run the same services directly on proxmox if you really need them)
You can use your external disk for long term backup(for yours VMs)
It is simple to recover a zfs mirror if one disk is broken (zpool replace .... )
zfs cache on ssd is out of question because your RAM is low (zfs without cache use less RAM )

 

[l0rdraiden]

zfs cache on ssd is out of question because your RAM is low (zfs without cache use less RAM )

Why? usually cache systems helps to reduce the RAM usage since files would be stored in the SSD temporary and not into RAM, right?

 

[guletz]

Why? usually cache systems helps to reduce the RAM usage since files would be stored in the SSD temporary and not into RAM, right?

Wrong! Any disk cache system, will use more RAM, but the goal for a such sistem is to reduce the IO disk and not reduce RAM usage. And think - so you put some file/folders on a dedicated cache disk, but you need to store at least some info about what do you have store already(at least metadata) on disk cache. In case of zfs cache, it is store the metadata about any block who is on zfs cache => you need more RAM if you use zfs cache.

 

[sodden]

Okay, so here's my personal opinion about storage setup for home use.

Keep it simple!

ZFS is your friend here and any kind of pass-through just makes it more error prone.
I've gone through multiple types of storage setups for my home lab and my current setup (Proxmox +ZFS) is by far the simplest approach and hasn't failed me yet.
The only thing that might be a bit of a hassle is keeping your users in sync if you need to share storage access between VMs. But you'll have that problem with any setup. I've been using freeipa to address this issue but that might be a bit complicated for someone who is just starting out.


So anyway, here is my suggestion for you.

2x4TB ZFS mirror on Proxmox (like guletz suggested)
1 dedicated zfs dataset for VM storage (I think that one is obvious)
X amount of datasets for everything you need to really share (use zfs set sharenfs/sharesmb)
Backups to external drive

If you're having concerns about NFS performance between VM's or Host and VM that shouldn't be an issue. If you're using the VirtIO network driver you'll have 10Gbe interfaces and that will probably be more than enough. I did an iperf test a while back and got about 6Gb/s average throughput on large files and 3.5Gb/s on small files. Your bottleneck will most likely be the storage itself.

I haven't had any failures yet (fingers crossed ;D) but in theory everything is pretty simple.
1. Proxmox dies -> fresh install, zfs import, restore VMs if necessary and you're good.
2. HDD dies -> zfs replace
3. VM dies -> Restore from Backup (or snapshot if you're using ZFSautosnapshot script which I highly recommend)

Simple right?