Best practice for homelab storage

[iouej]

Hello !

I've been reading a lot of posts but I still have questions and I hope for some inputs for my situation.
My aim is to have a server with proxmox on it which I would use as a media server (lxc with docker and multiple container app like plex, radarr, sonarr etc), vpn server (lxc with openvpn installed), videa game streaming (lxc with parsec installed), nextcloud server (lxc with nextcloud installed) etc...
Regarding the hardware, I have a single node with 8 HDDs and 1 SSD directly connected to the host.
I would like to find a good balanced storage solution between performance, upgradability and data security.

At the moment (after my readings and what I elaborate from them) my plan is to:
1-Use ZFS. I will install proxmox on the SSD, make place to install the container also and use half of it as a cache drive for the ZFS pool. The ZFS pool will include the 8HDDs in raidz2.
2-Create the differrent OS container for the different use. And use NFS from to host to mount my ZFS pool to the containers to share storage between them.
3-Enjoy until I read something cool and above my skill on reddit that I NEED to test on my homelab hahaha

My questions are:
1- Is my use of ZFS seems correct? Is it easy to increase the storage quantity in my set up? I've read that I can increase the pool size by swapping the drives one by one and resilvering it. But after will I have to mount all the drives again? Would the storage be immediatly accessible or I will have to tweak things in proxmox to get access?

2-What is the best way to share storage between lxc or VM when all the storage is directly connected to the host? I've read things about NFS but also iSCSI but I'm not very familiar with what is the best solution?

3- Do you have any recommendations or thoughts than come to your mind after that loooonnnnnggg read?

Bravo if you've made your way until there!
Thank you very much for you help !!!

 

[frenchfries]

Hey

When you say lxc with docker what do you mean? docker running under an lxc container? they aren't the same thing.

What is the OS formatted as? If it's ZFS then proxmox only supports using the full disk, ie. you wont be able to use part of it to speed up your pool. I highly advise ZFS for the SSD. Snapshots are easy and it just works well.
I'm not aware of your storage workload but I doubt you'd get any real improvement from caching tbh.

Answers to your questions
1. Once all disks have been replaced then ZFS will automatically report the new max size if you follow a few steps https://www.dan.me.uk/blog/2012/11/14/increase-capacity-of-freebsd-zfs-array-by-replacing-disks/ is an article about it https://madaboutbrighton.net/articles/2016/increase-zfs-pool-by-adding-larger-disks is another one

2. This really depends on your VMs i suppose. My VMs simply use SMB but i'm not really looking for high performance

3. I do something naughty around here. I run docker on the bare metal host ie. on proxmox directly and gain all benefits of ZFS including snapshots for my datastores. This is definately not recommended but I also use MDADM and XFS for my bulk storage, I like MDs ability to add a disk at a time, so my setup is kind of a frankenstein.

Basically I use proxmox as a standard debian host but with a nice GUI for VM management on top and the ability to have ZFS on boot without heaps of pain at the start..
Fileserving and docker are directly on the host and everything else is in VMs

 

[udo]

... docker are directly on the host and everything else is in VMs

Hi,
please don't do this!!
Due security reasons, you should run docker inside an VM.

Udo

 

[iouej]

Hello!

Thank you for yours replies!

When you say lxc with docker what do you mean? docker running under an lxc container?

I mean running docker inside a LXC of debian for example. From what I've read it seems to be the standard way to do it as Udo seems to suggest.

What is the OS formatted as?

As ZFS, thank you for the notice about not be able to split the drive.

I'm not aware of your storage workload but I doubt you'd get any real improvement from caching tbh.

I have seen people saying that it could improve a bit the browsing for a media library or for transcoding.

Does having the fileserver on the host a bad thing, it is better to have it on the host ? What are you performance with SMB for your VM ?
I had an other idea but it is either brillant or totally useless, what about having an application in each LXC, it would like using LXC as application virtualization platform like docker instead of OS virtualization. How bad is this idea ?

 

[frenchfries]

Hi,
please don't do this!!
Due security reasons, you should run docker inside an VM.

Udo

I run it with a less privileged account not as root but the main reason is that I want to leverage ZFS for my bind mounts. I haven't investigated doing it from within a VM then iscsi to a volume but i think that's a little ugly.

 

[frenchfries]

Hello!

Thank you for yours replies!


I mean running docker inside a LXC of debian for example. From what I've read it seems to be the standard way to do it as Udo seems to suggest.


As ZFS, thank you for the notice about not be able to split the drive.


I have seen people saying that it could improve a bit the browsing for a media library or for transcoding.

Does having the fileserver on the host a bad thing, it is better to have it on the host ? What are you performance with SMB for your VM ?
I had an other idea but it is either brillant or totally useless, what about having an application in each LXC, it would like using LXC as application virtualization platform like docker instead of OS virtualization. How bad is this idea ?

Udo is suggesting running it in a VM not a container which is what LXC is. AFAIK it's difficult to get docker to run properly inside of an unprivileged container.

First, you haven't actually said what sort of caching you're going for, read or write.
Browsing/searching a media library isn't a stressful task and is very random so you would hardly get any benefit from read (L2ARC) caching, and transcoding puts very little load on storage so write caching (SLOG) is pointless here.

I would advise you to keep it simple rather than add in stuff like caching and read up on the difference between containers (lxc) and virtual machines (kvm)

File serving on the host is fine I suppose but not really what proxmox is about. Really everything is meant to be done in a container or a vm and you can backup their images making the host install almost expendable. This is by design as it sets them up for easier clustering setup.

I don't run it this way but it's how it's meant to be run. I can't imagine I would get a lot of assistance for issues relating to my odd config.

Also, proxmox has a habit of writing a LOT to your bootdrive so I highly advise an enterprise SSD.