A Few Storage Questions from a PVE Newbie

[pr0927]

Hi all, I've just finished building a Proxmox server, and am very excited to begin using it and learning everything along the way. Before I say anything else, due to the absurd size of this post, I'm going to break it down by sub-headings for some sanity.

My Background

In the past I've run a homeserver from UnRAID, and for the last year or so, I've been running 3 TrueNAS SCALE servers (one of which I intend to keep as a TrueNAS SCALE system acting as a basic NAS for my wife, at least for the time being). My desktop PC runs both Linux (currently EndeavourOS) and Windows 11, dual-booted via different SSDs. I've been a techie for most of my life, and now work in IT for an MSP.

I share this to give a basic background of my competency and experience level. I'd describe myself as on the beginner end of intermediate skilled Linux user, or perhaps a more advanced beginner. My homelab has been invaluable for me as a growing IT professional.


Why Proxmox

Here's my situation for which I'm seeking out some clarity: I've built this server, with the intent to largely jettison most of one of my TrueNAS SCALE servers (currently a backup + NVR server) and turn it into a nearly-identical node. Lastly, I'll be turning my main TrueNAS SCALE server into a PVE node as well once I get proper command over PVE.

The impetus for all of this is the TrueCharts ditching of TrueNAS SCALE due to iXsystems' decision to stop supporting Kubernetes-based apps and switch entirely to Docker/Docker-Compose. TrueCharts is heading right into Talos Linux to keep the Kubernetes-based structure going (since they were never solely a TrueNAS SCALE team).

This is where I admit my "cheating" - I've never really had to learn Docker. Even on UnRAID, I just used the "Community Apps" to install apps - never really messed with any Docker configurations. And TrueNAS SCALE charts didn't require much more than inputting some values in app options (like with UnRAID).

The non-TrueCharts apps don't feel as fleshed out (they plainly have fewer configuration options) and not even all the apps I used are available there. I was overwhelmed/confused by Kubernetes - maybe something for a different day for me.

For the time being - Docker. The silver lining here is my being forced to finally learn it/container systems.


New Proxmox Server Details

Here are the specs of this system:

• AMD Ryzen 5 7600
• 64GB DDR5 RAM
• Nvidia GeForce RTX 3050 6GB
  • For Frigate - some of my cameras are H.265-only, so I transcode footage to H.264 for compatibility with many things.
  • For Jellyfin transcoding.
• 1x240GB Samsung PM893 SATA SSD
  • Boot drive, nothing else intended, in ZFS.
• 2x2TB Crucial T500 M.2 PCIe 4.0 NVMe SSDs
  • Virtual machines and containers, nothing else intended, in ZFS RAID1.
  • The most important VM, aside from perhaps one for Docker, is Home Assistant.
• 1x4TB Crucial MX500 SATA SSD
  • NVR footage from Frigate, nothing else intended, in ZFS.
  • SMB/NFS shared on the network for me to access from computers/VMs.
• 4x8TB Western Digital Red Plus SATA HDDs
  • Media and documents storage for container apps (Jellyfin, Nextcloud, Navidrome, Immich, Kavita, Audiobookshelf, etc.) to utilize, in ZFS RAID2.
  • SMB/NFS shared on the network for me to access from computers/containers/VMs.
• Google Coral PCIe TPU
  • For Frigate's object detection.

My main server (the one which will be the last node to be added) is more powerful (but also with 4x8TB drives, though those are U.2 PCIe 3.1 NVMe SSDs), but the other server I intend to retrofit will be largely the same as this above-mentioned, with a notable difference of an Nvidia RTX A2000 6GB video card (rather similar to the one above, but different).

I intend to have high-availability going on down the road, hence the total-overkill redundancy of things. Yes, I've spent a pretty penny here and there, but I've also been smart about deals, refurbished items, etc.


The Point - Storage Questions

So, to get to the point (sorry!) - I do not have any HBAs (all my PCIe slots are taken up anyway), and virtualizing TrueNAS SCALE or something similar to create storage pools and SMB/NFS shares seems like it is adding more complexity and abstraction layers - besides I'd like to run things more natively and practice some Linux CLI. What is the best way to set these all up, and what in the world is going on with the "Datacenter" section in PVE?

ChatGPT and guides online have said I can go under this server's disk section and create a ZFS pool by selecting drives and a desired RAID state - I have done this. But then to create the actual datasets (such as "ISOs" or "media" or "documents") I need to do "zfs create ____" CLI commands?

Doing that seems to work when i list ZFS pools in the shell - and the "Disks" section creation seems to also render that pool in the "Datacenter" storage section. But:

1. The creation of datasets via "zfs create ____" doesn't appear there?
2. Anything that does appear there, whether created as a "directory" or listed as "ZFS" forces a "content" choice - ZFS items only allow two choices - "disk image" and "container" - but choosing directory gives a few other choices, some of which are non-obvious in what they mean ("snippets?").
   1. On this note, if I don't want to store any ISOs or really anything on my boot drive, can I just remove that listing from here, and designate something else on the dual M.2 SSDs for that? Would I do this by just creating a "directory" in this section, or by the "zfs create ____" CLI commands - or both?

Is this section even relevant for my datasets?

Moreover, if I create all these datasets via "zfs create ____" and then follow whatever guides to run SMB/CIFS and NFS services on Proxmox to share them to the network, what is the "correct" or "best practice" way to link them to containers and/or VMs?

And lastly - should LXCs be used over Docker container apps on a VM? Given my entire...essay above, does one seem more useful than the other? I know I can't map storage directly into a VM and then try to use it elsewhere in other VMs or containers, as this can reportedly cause corruption - this is not the case for LXC-mapped stuff, right?


Help I've Fallen and I Can't Get Up

I have watched so many random guides, argued with ChatGPT, and read blog posts. I've waded through the Proxmox wiki and mostly left more confused.

Is there any good place to go for beginners to do this all correctly? I had a hell of a time even getting a clear answer on if it's ideal to have a separate boot drive than VM/container drives (Reddit posts helped...).

 

[pr0927]

Bump, one thing I see mention of is "Cockpit" in an LXC. Would folks recommend this in factoring the above?

 

[jdancer]

Migrated off TrueNAS SCALE to Proxmox because didn't have full CLI functionality. Used the LXC *Arr scripts from here https://tteck.github.io/Proxmox

I am using privileged containers because didn't want to configure UID/GUID remapping. Using Homarr as the jumping point to other *Arr LXCs.

Boot drive is an Intel Optane using ZFS RAID-0. Media drives are ZFS RAID-1. I also have a scratch drive for *Arr pre/post-processing using ZFS RAID-0.

You can use the GUI to configure ZFS for data/VMs but prefer using the CLI for /dev/disk/by-id naming of using /dev/sda? Easier to troubleshoot disks that way.

 

[pr0927]

Oh yeah was planning on using some of those helper scripts.

Do you recommend LXC over Docker in a VM?

And what's the drawback to using privileged containers versus unprivileged?

Yeah,I think CLI seems to be the right thing - I take it I don't "need" to have a dataset for media to be listed under "storage" in the "Datacenter" section?

And to follow-up on the Cockpit thing (LXC?) - would that basically help accomplish the same SMB/NFS control that CLI would do, and if so, is it recommended?

 

[Dunuin]

Do you recommend LXC over Docker in a VM?

Officially recomended is running Docker in a VM. From time to time a PVE upgrade will break docker LXCs. Already happended multiple times in the past. So if you want your docker services to be reliable, use a VM even if that got more overhead.

And what's the drawback to using privileged containers versus unprivileged?

Highest to lowest isolation: VM > unprivileged LXC > privileged LXC
In other words, the privileged LXC it is the least secure option you could choose and I wouldn't use it for anything you want public facing.
But because it is less isolated, you don't have to care about user/group remapping and stuff like using NFS/SMB shares will work.

I take it I don't "need" to have a dataset for media to be listed under "storage" in the "Datacenter" section?

You usually only want to have datasets as a PVE storage if you actually intend to put backups/ISOs/templates/snippets(/virtualdisks) on it.

 

[pr0927]

Officially recomended is running Docker in a VM. From time to time a PVE upgrade will break docker LXCs. Already happended multiple times in the past. So if you want your docker services to be reliable, use a VM even if that got more overhead.

Haha, OK, guess I need to get beyond "easy mode" and not rely on these helper scripts. xD

That said, do you think doing the storage SMB and NFS stuff through LXC in Cockpit would be good idea?

Since I intend to run Jellyfin and whatnot (publicly-facing), and I don't have an HBA, NFS sharing would be the move here, right?

On my TrueNAS SCALE systems, I had run NFS and SMB services, and then linked the storage pools for each app via NFS, and maybe just because it was all on the same local system, there was seemingly zero lag-time over a direct "host-path."

Highest to lowest isolation: VM > unprivileged LXC > privileged LXC
In other words, the privileged LXC it is the least secure option you could choose and I wouldn't use it for anything you want public facing.
But because it is less isolated, you don't have to care about user/group remapping and stuff like using NFS/SMB shares will work.

So I suppose this begs the question - for unprivileged LXC, caring about user/group mapping becomes mandatory - is this pretty difficult? What users are you even mapping, just system operator ones and admin accounts?

You usually only want to have datasets as a PVE storage if you actually intend to put backups/ISOs/templates/snippets(/virtualdisks) on it.

Ah OK, makes sense - so I should just leave any datasets I create via CLI "zfs create" commands for storage aside from those categories alone and not try to add them there.

 

[Dunuin]

So I suppose this begs the question - for unprivileged LXC, caring about user/group mapping becomes mandatory - is this pretty difficult?

See: https://pve.proxmox.com/wiki/Unprivileged_LXC_containers

What users are you even mapping, just system operator ones and admin accounts?

All users that need to access bind-mounted folders.