New Setup Advice - Medium Sized Setup (so much to consider)

Unboxed6512

New Member
Sep 23, 2023
1
0
1
Starting to build a new system, first time using Proxmox. I have a lot of experience with linux/zfs/TrueNas/Unraid and am comfortable with the cli and programming in general. This is going to be a single instance home server, primarily for NAS (via smb and nfs) with some experimentation in various VMs (i.e. HAOS) as needed. No gaming or other high CPU/GPU requirements.

I think my primary goal is to ensure I have plenty of "sanity" around the config, the disk setup and ensuring I can backup the config to do a re-build if needed. I want good disk redundancy (i.e. disk failure protection) and I will backup my critical data and configs offsite with something like duplicacy in docker. Initially, I was planning on doing everything on bare metal with Debian 12 (ZFS 1+0, ansible to manage the OS/Docker/NFS/SMB config) as I don't mind cli operations and devops tools at all. Now, the lure of Proxmox has me second guessing that design.

I was thinking of running proxmox like this:
  • zfs mirrored boot hard drives, space for ISOs, etc.
  • zfs mirrored nvme drives used to support VM boot disks
  • proxmox managed zfs 1+0 pool of all of the other disks (roughly 12) attached to LSI HBAs (the hardware would not be passed into any VM)
  • run a VM with Debian 12 using LVMs that mount boot disks from the proxmox managed nvme pool and the rest of space needed allocated to the vm from the proxmox larger disk pool
  • I should also mention there are 2xSSDs on that HBA, but not sure what to do with them in this setup at the moment.

Pros:
  • All NFS/SMB/Docker admin and content backups will be run from the Debian VM. I'm comfortable with that as I can back most things up from a container in that VM
  • All ZFS management will be from Proxmox which seems to have a decent UI but is backed by a good cli
  • The Debian LVMs can be easily expanded as I add more (or different sized) disks in proxmox
  • I don't have to stumble around with a lot of NFS mounts if I used TrueNAS (since it is so crappy with docker, i'd have to run docker in a different vm)
  • snapshots, headless control, general VM goodness

Cons:
  • More complicated than just running Debian on bare metal
  • That complication might make it more difficult to reproduce the proxmox system config. Things I might typically use tools like ansible to do (maybe it's just my unfamiliarity tho)
  • It feels a little I'm relying too much on Proxmox for disk management and I don't know how I would backup the config offsite (i.e. I use duplicacy in docker right now, but how will I get to the important proxmox config data? I don't want to set up a proxmox backup server rn.

Sorry, it's a lot to unpack, but I guess I would feel comfortable with some advice on:
  • whether this sounds crazy or reasonably sane. Maybe it would help to hear your stories and experience with disk or config failure made recovery easy because of good initial designs and setup practices. I'm just coming off of some Unraid shenanigans that will likely make me never leave zfs again :)
  • if all goes bad from a VM OS perspective, whether the proxmox zfs disks would still contain data that I could simply mount on another system that supports ZFS and be mounted and read. The bare metal debian option feels like it could reproduced easier than proxmox (again, could just be my inexperience)
  • What to do with the SSDs? I think it might be a bad idea to use them for zfs cache, but not sure what other good use they may have the moment.

Thanks for the feedback and though I have read a bunch of the docs, other good references welcome.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!