Proxmox storage architecture

[Jaspergie]

Hi all,

I am going to install proxmox for the first time this weekend on an entirely newly built system. The system is the following:
-Intel i7 11700 CPU
-64GB DDR4 3200MHz RAM (Non ECC, obviously since my CPU)
-2 TB M.2 NVMe SSD (TLC)
-My MoBo (and CPU) supports up to another 64GB extra RAM + has one extra M.2 slot and 6 SATA ports

What I plan to do with the system is mainly focuses on 2 parts:
- Host personal services, such as Jellyfin (and supporting services), Nextcloud, Bitwarden etc.
- Host my private homelab consisting for example out of a Windows Server for AD etc, Windows client(s), Vulnerable client, hacking system, Logging and monitoring stack, and other nerd stuff related to projects i do or will do in the future.

I plan to host the first parts in containers (LXC), so for example my Jellyfin can directly use my iGPU instead odd passthrough. The homelab will probably run multiple dedicated VMs. I will install Proxmox and all containers/VMs/data stores etc on the one SSD I have and then later add more SSDs when I need more space or when I get one as a christmas present haha.

I plan to use a traditional dedicated back-up solution. So probably I'll use Proxmox Back-up Server to store/back-up everything on an external disk/back-up solution.
I don's plan to use a Raid system, since I dont think I need it.

Now my questions are:

• I noticed the option to choose a filesystem architecture during the Proxmox install, which got me reading a bit about ZFS (never heard of it before) and a lot of people seem very positive about it:
  • Is ZFS still recommended for me even though I have SSDs and will use no RAID?
  • If so, I guess I should preferably already select ZFS during the Proxmox install? Or should I only use ZFS for the "data storage" parts?
• That made me thinking: What do you think is the "best" way to implement and divide (logical) disk volumes?
  • Should I mage one big logical volume and expand that everythime I add an SSD? (I guess that is very good possible with ZFS)
  • Or should I make on volume for the OS (Proxmox) and one for evything else? Or should I also divide 2 logical volumes for the VMs and then a separeate one for all the data (my files, video's for jellyfin etc). How do you guys make your volume architecture? How big should the OS volume be?

I hope someone with the same situation/usecase or with more knowledge than me can give me some advice. Off course I am willing to put more time into reading about ZFS, but only if it is suitable for me, so I want to check that first.

Thank you!

Jasper

 

[Jaspergie]

Or is it easier/better to create a TrueNas instance, store all my files there adn then share the folders to the specific other services such as Jellyfin, Calibre, Nextcloud? I however only have one disk right now, is that gonna be a problem for TrueNas?

 

[LnxBil]

I don's plan to use a Raid system, since I dont think I need it.

Hopefully, this is not going to bite in ...

The hardware is relatively beafy, whereare you have only a TLC SSD, which you will not have very much fun with with ZFS. Consumer SSDs have a lot of problems with ZFS, including performance and durability.

 

[Jaspergie]

Haha no, I am actually look for advice. So, what are your recommendations:

• Don't use ZFS at all? I wasn't planning to do so at first, but other people suggested that even with Raid0 ZFS brings extra advantages in data integrity, snapshots, back-ups. Also because I am not familiar with it, it is maybe fun to at least learn a bit about it already (not fully, as you already said). Don't I need ZFS for TrueNas also?
• Or can I also just already add a second SSD for storage? One that is better suited for ZFS:
  • What are good non-consumer SSD's then? For example the WD red series?
  • You specifically also mention TLC. I know QLC is the worst option, but I read that many enterprise SSDs also use TLC. Would you recommend to use SLC, eMLC or MLC?
  • What are other specs I need to keep an eye one when buying one? The one I have for example has already quite a high "Total Bytes Written (TBW)" of 1,2PB, almost all specs are close to a "WD Red SA500 2,5" 2TB" (1,3PB) for example.
• Would it be "ok" to use the existing SSD to store the VMs, ISOs etc. and then use a more enterprise SSD for storage?

 

[LnxBil]

Don't use ZFS at all?

No, I love it and it's great. You may run into performance problems on consumer grade SSDs and some of them die VERY fast.

What are good non-consumer SSD's then?

Any enterprise SSD you can find. For my homelab / myself I always buy used enterprise SSDs and I just bought a 960 GB Samsung for 39 euros, so they can also be very cheap. AFAIK, all WD RED stuff is still not enterprise level, just prosumer ... still better than consumer but still lacking features like power loss protection etc.

What are other specs I need to keep an eye one when buying one?

Any enterprise SSD will be better than any non-enterprise one. If you want speed, use SAS or NVMe, but most of the time, you will not have the need for that speed (and pricetag).

 

[Jaspergie]

Good evening,

Thank you for your answers. I agree that the storage is a bit limited now in relation to the rest of the system. This is because I started out with the idea to just host a "simple" hobby homelab. But along the way I kept thinking of other usages, such as Jellyfin, Nextcloud (whish ideally require some filesharing).

So please hear me out:

My current situation and plan for the "initial" hardware setup is currently as follows:

• I currently only have 1 disk, a 2TB NVMe M.2 SSD use, with another NVMe M.2 slot free. Furthermore, my MoBo and Case can have 6 x 2,5" SATA600 disks.
• I'll keep using my current NVMe M.2 SSD (1,2PB TBW) for Proxmox host OS and VMs
• I'm about to buy 1 or 2 used enterprise 4TB SSDs with 90-100% health (this one https://www.amazon.com/Intel-S4500-SSDSC2KB038T701-3-84TB-Enterprise/dp/B075HRYWC4) . I can buy them here for around 200 euros each (which per TB around the same price as you payed). I'll use these ones for storage and I would like to buy more in the future to "expand" my storage when I need more space.
• I'll use an external harddisk to do weekly back-ups of everything (All disks. Host OS, VMs, Cofigs etc and Storage)

The considerations and idea's to configure them:
So my initial plan was to not use any RAID at all: my data is only personal, not time/availability critical and I want the system to be as cheap as possible, since it is mostly just a hobby homeserver/homelab. However, I am planning to keep all my files there as some sort of NAS and in Nextcloud and I also would like to not lose all or a part of my VMs that I have been working on a lot in case of a disk failure or some other issue. Therefor I was planning to keep weekly back-ups so I could just copy pasta all my data back on a new disk or overwrite the damaged files (no actual need to have it back-up within minutes or hours).

• Ideas/options for the host NVMe SSD - It is a consumer SSD, but it has a "decent" TBW so I will use it. However, the wearout remarks made me think about how to configure it:
  1. I could just configure it "as is" with ZFS in RAID0 and have already some advantages. ZFS might be overkill in this case.
  2. However you made me think: what if the disk fails or if the wear-out is at critical levels and I want to "smoothly/quickly" replace the disk. I could buy a new SSD reinstall Proxmox and replace back-ups, which would work but probably it could become a big hassle. For just around 100 euros I could buy a second SSD. Enterpise NVMe are too expensice, but this one had more than 4PF TBW: https://www.amazon.com/Transcend-MTE220S-Solid-State-TS2TMTE220S/dp/B0875JV151?th=1. Then I add this one later and update the RAID0 to RAID1. In case of fail, the mirror will make the replacement much smoother. I consider it extra costs, but 100 euro is not that much and it might be worth it: more reliable, safer, more stable. I could also put my most critical data on this drive. And I get to make the most out ZFS (learning a lot). However, also this might be overkill?
  3. The last option is to just use the 1 NVMe SSD and configure it with xfs/ext4. I will not the most smooth replacement, but the wearout will be much less? Or do they also wearout fast with ext4.xfs? Is the wear-out a proxmox/vm thing? What would in my specific usecase be the biggest "drawback"/"disadvantage" when using ext4/xfs instead of ZFS?
• Ideas/options for the storage SATA SSDs - I was planning to buy extra disks anyways, so the 1 or 2 enterprise disks I don't consider "extra costs". Buying these good Enterprise used ones are around the same price as new consumer ones. I am still having doubts how to configure them though:
  • RAID: In the end I want to have at least room for 10TB of data, so 3x4 TB disks for around 600 euros. Having that in RAID1/Mirror would require me to buy 3 extra disks for 600 euros. This is way out my budget and feels like madness for a simple hobby homeserver/homelab. RAIDz feels like more hassle to recover, so plan to go just for RAID0 and back-up and recover the entire disks when required.
  • NAS/Share: The storage will be mainly used by other VMs as Jellyfin and Nextcloud, but also networkdrive for my laptop. I still have to decide between going for TrueNAS passthrough the disks to that VM or just setup SMB/NFS from Proxmox myself
  • Filesystem: When I not using TrueNAS, with the RAID0 setup, what would be the recommended filesystem for these storage SSDs? Should keep it the same as the Proxmox OS filesystem?
  • Disk addition: I am starting with 1 storage SSD where I want to make shares for Jellyfin, Nextcloud etc. However, when the disk space is low I want to be able to add a disk so the shares can just store more data without any issues. The disks should be 1 logical volume I guess? What is the best way to set this up and is ZFS or Ext better in relation to this?

Could you or anyone with much more experience with Proxmox please give me some advice and recommendations in the mentioned options above? (or recommended options that I didn't mention are also welcome off course)