Purchasing a Server Exclusively for PBS (hardware discussion)

[BriBriBri]

Hi All,

We're looking to purchase a server new, explicitly for PBS. Which is a bit of a big commitment for us. So I thought I'd run the specs by everyone here in case I'm missing anything. . .

SUPERMICRO AS-2124US-TNRP Server/Chassis
2 x 16 Core AMD EPYC 7313 Gen 3 Processor
256GB DDR4-3200 ECC RAM
24 x SAMSUNG PM9A3 2.5" U.2 7.68TB PCIe 4.0 x4 NVMe

Planning on putting the twenty-four SSDs in zfs RAID-Z2 for approximately 168TB of raw space.

Our current PBS operates on 24 drives of spinning rust (Seagate ST10000NM0086-2AA101) with some consumer-grade SSDs as special devs to help speed them up. It's in a ZFS stripe of mirrors config totalling 120TB useable space. Due presumably to the spinning rust, we've been noticing poor performance (tapes writing only 50MB/s to 100MB/s when tape drive can sustain 300MB/s). Same with disk-to-disk backups (maxing out at well below GbE over 10gig links). Likewise, our verification jobs take several days. The server is literally always working on a backlog of verification jobs.

Our hope, of course, is that we'll see much better performance with the specs above (going all-SSD, and even all-nvme). But I'd love to hear any critiques. We don't mind going a bit overkill. But I'm especially concerned with. . . well. . . underkill! We only get one shot at this for the next several years!

Thanks in advance for anyone who takes the time to reply!

Bri

 

[BriBriBri]

already thinking maybe I should double the RAM to 512GB for 168TB of ZFS RAID-Z2?

 

[fabian]

more RAM is always good for ZFS!

I'd really recommend re-evaluating this part here:

Planning on putting the twenty-four SSDs in zfs RAID-Z2 for approximately 168TB of raw space.

while this would maximize space, it's also really a lot of eggs in one basket and not very good performance wise.

something like 3x8 raidz-2 would only "sacrifice" 4 more disks (~30TB more space for parity), but roughly triple your iops, and improve resiliancy and resilvering times. of couse, 2x12 or 4x6 are also valid options - https://wintelguy.com/zfs-calc.pl might help you getting a feeling for what happens when you change the pool layout.

what is the bottle neck in your current setup? verification needs both fast random I/O (since it needs to read lots of chunks) but also CPU power (for calculating the checksums). it also helps to batch verification so that individual chunks are not verified multiple times in a short time span - https://bugzilla.proxmox.com/show_bug.cgi?id=4714 has some background info.

 

[BriBriBri]

Hi Fabian,

Apologies for the huge delay in responding. Other things came up but we did finally get our new hardware and I'm getting ready to put pbs 3.0 on it.

I'm jostling this thread awake again just in case others making a similar decision/purchase find it useful.

We ended up with twelve 15.3TB nvme SSDs instead of twenty-four 7.68TB. Leaving twelve bays open for eventual expansion.

My thinking is that putting them all in raidz-2 might "strike the balance" between my concerns for easily just dumping all our replicas/backups into one large datastore (organized via namespaces) and your (valid!) concerns about optimizing iops. At twelve SSDs total, would you agree?

Regarding the bottleneck in our current/old setup? I always just assumed, based upon the repeated warnings against traditional HDDs, it was the "spinning rust" HDDs along with the relatively old CPU (Xeon E5-2630 v4). I'm hoping for a really nice performance uplift with the two Genoa 9124 DP/UP 16C/32T 3.0G CPUs and the nvme SSDs (Micron 7450 SSD 2.5" NVMe PCIe4 15.3TB).

It's really the first time I've ever run a built-to-order server rather than cobbling things together from spares. So I'm excited!

Thanks for your feedback. And, again, sorry that I took so long to reply!

I'll try to post the eventual results here when things are up and running.

 

[guletz]

HI,

Some suggestions:

- do not put all eggs in a single basket(I would buy 2 x servers for PBS), because if some bug(hardware) will hit your only PBS server you are out of game
- 12 x SSD, in raidz2, is suboptimal, as @fabian also suggested(iops is bad), but also the resilver(in case of a ssd will be broken), and also zfs scrub will not be so fast
- I would use a draid zfs in case of raidz2, for the fast resilver case
- PBS is fast, if you have a decent cpu, and sufficient RAM, and a good iops(raidz1-2 is bad)
- PBS is metadata intensive(especial for verification jobs), so you need to optimise your zfs pool for this case(L2 arc/cache for metadata only is one possibility)
- your ideea about a future expansion from 12 x SSD raidz2 will not be happend, because you can not do this with any raidz2(you could only if you will use stripped mirror/raidz1/2)

Good luck/Bafta !

 

[BriBriBri]

- do not put all eggs in a single basket(I would buy 2 x servers for PBS), because if some bug(hardware) will hit your only PBS server you are out of game

Well, it is a backup server, after all. We would still have the data on the live servers. And we backup to tape weekly. Given this, purchasing a second server seems extravagant. While safeguarding data is paramount, at some point, diminishing returns and budgets do matter.

- your ideea about a future expansion from 12 x SSD raidz2 will not be happend, because you can not do this with any raidz2(you could only if you will use stripped mirror/raidz1/2)

Apologies. I thought it would be obvious that I am referring to server capacity expansion (not zfs pool). Which is the point where we'd entertain the creation of a new/separate zfs pool.