How to configure homelab storage

[Weyla]

Hello!

I have been running my homelab under proxmox for about half a year, I am very pleased with the experience, but since my storage options and knowledge is limited I set it up wrong and I would like to fix it now.
I have 5 drives currently attached to my server:

-500 GB Samsung EVO 870​

This is my boot drive and it stores all the VM and LXC instances.​

​

-6TB WD Purple​

This is for my CCTV cameras, it is formatted to ext4 and passed to a frigate lxc as a directory.​

​

-2TB WD RED NAS​

I use this drive for my media server stuff, downloading linux ISO's and watching it in Plex.​

​

-2TB Seagate barracuda​

Since I dont have anything in RAID I thought I would go for some cheap backup option, I use this to store snapshots.​

​

-250GB Samsung EVO 840​

Currently not used.​

Now I have some spare money that I could use for buying some extra drives and configure it properly, but since I am a dummy I would like some opinions about my ideas, before I go buy anything.

What I imagined:
-Get another 500GB SSD and setup Proxmox in a mirrored ZFS, use this as boot and store the VM and LXC instances.
-Setup the WD Purple as single disk ZFS and pass it to the frigate LXC
-Setup the WD red(almost 5 years old) with the Barracuda in a mirrored ZFS and use it as NextCloud storage. I would like to store photos and basic stuff in it, It wouldnt be heavily used, so I guess the barracuda should not be overloaded.
-Get 2x4TB Seagate Ironwolf and set it up as mirrored ZFS and pass it to a VM where I run my media stuff in docker. I could setup an SMB share in this VM if I would want to access it somewhere else.
-Use the 250GB SSD to install proxmox on it and practice migrating the VM before doing the whole thing to avoid data loss.

I didnt look into it yet, but I think I could also configure backups of the Proxmos instance and the Nextcloud data to an external cold storage service.

What do you guys think about this?

 

[krikey]

If you're spending money on new and additional drives anyway, why not overhaul the large storage space using larger drives and perhaps something like RAID5 or ZFS RAIDZ rather than having single and separate drives in some cases which have no fault tolerance.

Perhaps 2 x SSD/NVMe drives mirrored for your Proxmox core and then the remaining space with as big drives as you can afford/fit. Ideally get the same make/model drives if RAIDing them (or at least that's what we used to be told, things may have changed with ZFS).

At least with one large lump of RAIDed storage you can protect all the LXCs to a degree.

As for backups, re-use the 6TB single drive as a cold storage device perhaps that you backup your data to? If you run PBS https://www.proxmox.com/en/proxmox-backup-server/overview as a VM, you can leverage its de-duplication functions to reduce the total needed space for backups too.

 

[Johannes S]

Please note, that there are several reports here, that ZFS RAIDZ isn't good at providing performance. For performance and ZFS it's best to setup the devices as mirrors. For improving performance on HDDs it might be a good idea, to setup two SSD partitions as a special device to improve the performance. Please note, that in case of a broken special device the whole ZFS mirror is broken and can not be recovered.
It should also be noted, that ZFS and consumer ssds don't play nice together since ZFS tends to bring a higher wearout to them.
Generally I would suggest to look for some used enterprise ssds with powerloss protection and setup something like this:
- One ZFS mirror on two SSDs for Operating system and vms
- Another ZFS mirror for application data

If you don't want to spend money on larger ssds a possible route would be something like this:
- One ZFS mirror on two NVMEs or SSDs for the operating system
- One ZFS mirror on two large HDDs for vms, application data etc pp, together with two mirrored NVMEs or SSDs as special device.

This can varied of course: You might buy larger SSDs and create a partition setup where one partition is for the operating system plus vms and the remaining parition the special device. Or alternatively two SSDs/NVMEs more so the VM and containers get their own dedicated ZFS mirror.

However I know that in a homelab budget is a concern so I understand you want to start with your existing hardware.
Some things to think about:
- Is the CCTV data important? Then ZFS bit rot protection is a good reason to change the file system. Otherwise I wouldn't do it since the features of ZFS come with a performance penalty.
- ZFS mirror on your HDDs: This should work, again the performance impacts of ZFS should be taken in consideration. Maybe a special device (see above) would help with performance here.
- Now we come to the point of backup:
-- Backup is pointless without restore, so you should think how to do it and test it on a regular base.
-- One backup is nearly as bad as no backup: What happens if this backup is lost? For this reason, the 3-2-1 backup strategy shold be used: You have at least three copies of your data (production data + two copies), stored in two different medias and at least one offsite. With other words: You should have one local backup (e.G. on an external usb disk) and one offsite (e.G. Cloud storage).
-- For application data something like a S3-Storageprovider or hetzners storagebox are propably the best option. I use something like this with restic to backup the data I can't bear to lose ( from importants documents to treasured vacation photos) or don't want to re setup.
-- For a backup of the VMs the cloud option isn't so great. The normal backup functionality of proxmox will work good enough but will always create a full snapshot so you will get a lot of data, for which you will need the storage and you will have to transfer it to your cloud storage. Both together will lead to long backup times and propably won't help with the costs. For this reason I would suggest to look for a setup of the Proxmox Backup Server. With it you will get deduplication (reducing the backup sizes) and other nice features (ransomware protection etc pp). You would need to setup two instances: One in your local network (ideally on it's own host, e.g. an old mini pc doesn't need to be the biggest workhouse, it should have SSDs though otherwise performance will suck) and one offsite. The offsite instance could be another mini pc at some friend or parents house or a cheap cloud vserver. An alternative would be to use Tuxis offering for a cloud PBS instance ( https://www.tuxis.nl/en/proxmox-backup-server/ ), they don't sell it to private customers though. But if their free tier (which can be used by private customers) is enough for you that might be an option and will save you the hassle of administrating the vserver.

Hope this helps, Johannes.

 

[Weyla]

Thanks for the information!
Now my budget is really tight and my current idea is way over the limit, but I have an idea what I may be able to do.

-The 6TB drive is for CCTV, it generates 8-9gb of traffic an hour for 3 cameras and I plan to get 2 more in the future. The price per value is really high for this drive since I live in safe area and I also have an alarm system. I dont want to spend more on it, the data is not valuable enough to make it redundant at this price tag and I also dont want to put more wear on NAS drives with continously writing on it. I guess I dont need this drive as ZFS.

-I think I could affor to get 3x4TB Ironwolfs and run them in raid 5, that would give me 8TB of data which is waaay more than I actually need, but its good to have. I could use this pool for media stuff and nextcloud aswell.

-Still go for the 500 GB SSD and setup the system in raid 1.

For the bakcup options I still need to experiment and test how it works and what can I do with it, but if it can do what I think it can then I could get a cheap mini PC, add the 2x2TB HDD in raid 1 and the 250GB SSD in it as a boot drive setup a backup server and place it in my workplace where electricity is basically free and it has static IP. 2 TB should be way more than enough to store everything that is actually important. This way I would have 1 copy in the server in raid 5, 1 copy off site in raid 1 and I could do occasional saves on my desktop PC.

 

[krikey]

If you're going to host your backup server in another location (which @Johannes S mentions as a good step to 3,2,1 backups), ensure that you're using an encrypted tunnel to do so. For example a VPN or over an SSH tunnel.

On the backup server end you may also wish to consider a firewall too, limiting access to specific IPs?

 

[Weyla]

If you're going to host your backup server in another location (which @Johannes S mentions as a good step to 3,2,1 backups), ensure that you're using an encrypted tunnel to do so. For example a VPN or over an SSH tunnel.

On the backup server end you may also wish to consider a firewall too, limiting access to specific IPs?

That is a great tips, thanks!
I actually have wireguard setup for the cameras and reaching my services without opening extra ports, so that could make things simpler.