Speccing PBS server

[dignus]

Hi all,
We're currently thinking about moving one cluster to proxmox. Storage used will be about 130 TB. I read about and tested PBS, works like a charm. But not on spinning rust.

First question: what compression / dedup rate can I realisticly expect? The 130 TB storage in use are 90% Ubuntu 22.04 VM's.

I would like to re-use some "older" hardware we have laying around. One of the pieces of this would be a whole lot of Samsung PM863a drives. These are fast, but "only" SATA SSD instead of NVME. Would this be good enough? I did some testing, seems OK, but I can't mimic backing up the 130 TB.

When it comes to CPU & memory, what is the recommendation for this?
I would love to use ZFS because of it's feature set and simplicity in expanding/replacing stuff. I'd disable compression and checksumming, basic leaving me with something that does perform closer to ext4. Did someone else test this already?

 

[Dunuin]

I'd disable compression and checksumming, basic leaving me with something that does perform closer to ext4. Did someone else test this already?

The default LZ4 compression of ZFS basically comes free and still saves some space, even with chunks already compressed with ZSTD. And regarding disableing checksumming...keep in mind that the verify job of PBS can only detect corrupted data, but it can't fix these corruptions like ZFS with checksumming could do. Especially important as PBS deduplicated chunks and nothing is stored twice. Lets say you got 100 backup snapshots of a VM and a chunk corrupts that is part of the 80 oldest backup snapshots. Then this single chunk will prevent you from restoring any of these 80 backup snapshots. And you can't fix that chunk by creating a new backup, as it isn`t existing on the PVE node anymore for the last 20 backups.

 

[dignus]

Yeah, agreed. But any other filesystem would have the same issue, but yes, I do prefer to keep it on if performance allows it. ZFS is the best for a reason.

 

[dignus]

Hm, quiet in this thread. Can anyone advise me on cpu/mem and the sata vs nvme question?
I’ll happily buy nvme, but it would be a waste not to use this 70 TB of fine ssd’s.