NVMe allocation for new homelab

[JLuizPJr]

Guys, I'm brand new to proxmox since I'm retiring my 2013 Synology and creating a new homelab. My objective is to have a quite reliable system for the next 10 years, running trueNAS and quite a few VMs and LXCs.

I'm using a 7950x on a miniITX X670E-I Asus Strix. I'll use 2x onboard 2TB NVMe for boot in mirror and I'll have a ASUS Hyper M.2 card with 4x 2TB NMVe, with bifurcation enabled on the mobo.

My question is about the 4x2TB allocation: should I create a RAIDZ on proxmox and allocate a simple virtual SCSI to trueNAS OR should I passthrough the 4 NMVe direct to trueNAS, making it available thru SMB/CFIS to proxmox? I spent the last two days thinking about the pros and cons and still in doubt. Appreciate your time!!

 

[JLuizPJr]

There is the third option which is using the 6 NVMe in a single spool… I don’t know why, but I’m not comfortable with that…

 

[Dunuin]

First, you really shouldn't use consumer SSDs with ZFS. There are only 5 Enterprise M.2 models with 2TB available and all of them are 22110 and might not fit all M.2 slots.

Raidz is fine for TrueNAS but not great to store VMs.

Its best to avoid ZFS on top of ZFS as ZFS got massive overhead and overhead will exponentially multiply and not simply add up. Thats why you usually passthrough the disks as TrueNAS requires you to use ZFS. Other option would be to skip TrueNAS, create the ZFS on the PVE host and run an LXC as your fileserver with bind-mounted dataset. Or something like a OpenMediaVault VM on top of ZFS that will allow you to use some less demanding filesystems like ext4/xfs.

 

[JLuizPJr]

First, you really shouldn't use consumer SSDs with ZFS. There are only 5 Enterprise M.2 models with 2TB available and all of them are 22110 and might not fit all M.2 slots.

Raidz is fine for TrueNAS but not great to store VMs.

Its best to avoid ZFS on top of ZFS as ZFS got massive overhead and overhead will exponentially multiply and not simply add up. Thats why you usually passthrough the disks as TrueNAS requires you to use ZFS. Other option would be to skip TrueNAS, create the ZFS on the PVE host and run an LXC as your fileserver with bind-mounted dataset. Or something like a OpenMediaVault VM on top of ZFS that will allow you to use some less demanding filesystems like ext4/xfs.

Thank you Dunuin!! Along the years, as price goes down, I expect to replace the consumer SSDs with Enterprise grade ones. Your ZFS on top of ZFS argument is really strong.

 

[Dunuin]

Along the years, as price goes down, I expect to replace the consumer SSDs with Enterprise grade ones.

I hope you at least didn't bought QLC consumer SSDs. Would have been better to spend less on this expensive CPU and use that money to not cheap out on storage when building a NAS.

 

[JLuizPJr]

I hope you at least didn't bought QLC consumer SSDs. Would have been better to spend less on this expensive CPU and use that money to not cheap out on storage when building a NAS.

I'm using Kingston KC3000 2TB... it's TLC with a Phison E18 controller and 3.2PBW lifespan.

 

[Dunuin]

Kingston KC3000

Ok, at least TLC. But 3.2 PB is the 4 TB version. The 2 TB only got 1.6 PB TBW.
Try to avoid running DBs or similar stuff that is doing sync writes as the SSDs got no power-loss protection and therefore can't cache these writes in DRAM for wear optimization. If you try hard, with the wrong workload, you could kill those SSDs within a few months.
1.6 PB TBW sound like a lot, but if it could continuously write with the advertised write performance of 7GB/s (hint: it wont...performance will heavily drop after a few seconds...should be more like 1.6GB/s which is still very fast) this would mean these 1.6 PB TBW would be exceeded after 63.5 hours of writing...or 11.6 days if you calculate with the real achievable write performance.
In reality you won't write that much of data but could still be a lot because most people underestimate the write amplification of sync writes, nested filesystem and so on.