Questions about material sizing for a "highly" stressed PBS

[Adri-M]

Hello.

I would like to switch from 2 slow instances of PBS to an efficient one (described below). I'd like to have your feedback on the consistency of the hardware with my usage.

Here is my actual situation:

On the first instance, I currently centralize the backup of 6 PVE clusters located on remote sites to a central site (Total ~80Vm LXC/KVM ).​

I back up all VMs (KVM and LXC) 4 times a day, with fairly long retention periods (360 hourly, 15 weekly, 6 monthly). In total, this represents 4.5TB.​

The volume is mounted by NFS on a ZFS NAS (truenas).​

The global "Prune Job" task takes ~20min, the "Garbage" task takes about 12h, and I no longer perform integrity checks because the VM is totally saturated due to diskIOs. I also have IO Delay and Load Average values that are too high (probably due to the slow filesystem), as I have the same problem with 48 x Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz (2 Sockets).​

On the second PBS instance I'm backing up the main cluster of my infrastructure: 7 nodes for ~150Vm LXC/KVM. I've made a second instance to separate the server load and above all to use another ZFS NAS while waiting for an investment. The issues are quite similar (Slow IO, High CPU, GUI list of backups very very slow...)​

Here is what i want to buy to unify my PBS instance and finally be able to have a fluid GUI listing of my thousands of functional backups and the possibility of carrying out regular integrity checks without it taking days/week or simply broke the system.

Intel Xeon-Gold 6434 3.7GHz 8-core 195W (mono socket)
256Gb RAM ECC
PBS Storage: 5 x 3.2TB NVMe Gen4 High Performance Mixed Use
PBS System: NS204i-u ( NVMe HotPlug Boot Optimized StorageDevice)
2x10Gbps LACP for network.

Question is:
Does this configuration seem harmonious to you?
Is it better to use integrated PBS ZFS management (Creating a RAIDZ1 ou RAIDZ2 pool via GUI using the 5 Nvme)
Should we prioritize CPU frequency or cores?
I've read everything and its opposite on the Internet. I'm not planning to add any SpecialDevice, Zil or L2Arc to this ZFS NVME pool. What do you think? Are there any important parameters to specify before making the ZFS pool? Also, if you see another method more suited to PBS for storage, I'm interested.

Last but not least: If it's best to manage disks in ZFS using PBS, then I'm forced to install PBS directly on the hardware. It's impossible to virtualize it while maintaining the same performance, isn't it?

I thank you for reading this long thread and hope to benefit from your expert comments

Adrien

 

[proxstormer]

mmm... your problem is not the backup, it is the restore.
Only one core doing the restore, only one restore at time ... you will take ages to restore your clusters.
Divide and rule.

 

[VictorSTS]

- Install PBS on baremetal. Backup /etc/proxmox-backup often. The datastore can be connected to PBS easily if you ever reinstall.
- RAIDz performance is limited to the performance of 1 drive. Those drives might be fast enough for you. If unsure get more drives and use RAID10.
- No need for zil, special device or l2arc, will add no performance for an all SSD datastore. Well, maybe l2arc if you add a couple tb ram
- Each task in PBS is siglethreaded, but there might be somewhat high concurrency (many backups at once, restores, GC, verify), so you'll need to get an average of simultaneous tasks and get the fastest CPU with that amount of cores that your budget allows.
- Do verifies asap. Even if ZFS protects from bit rot, a single missing chunk for whatever reason can render many backups useless and you won't notice unless verifies are run regularly.

 

[Adri-M]

- Install PBS on baremetal. Backup /etc/proxmox-backup often. The datastore can be connected to PBS easily if you ever reinstall.
- RAIDz performance is limited to the performance of 1 drive. Those drives might be fast enough for you. If unsure get more drives and use RAID10.
- No need for zil, special device or l2arc, will add no performance for an all SSD datastore. Well, maybe l2arc if you add a couple tb ram
- Each task in PBS is siglethreaded, but there might be somewhat high concurrency (many backups at once, restores, GC, verify), so you'll need to get an average of simultaneous tasks and get the fastest CPU with that amount of cores that your budget allows.
- Do verifies asap. Even if ZFS protects from bit rot, a single missing chunk for whatever reason can render many backups useless and you won't notice unless verifies are run regularly.

Thank you!

- It was planned to go on barematal, now it's sure
- RAIDz performance: Aware of that but with NVME High Performance Mixed Use i hope RAID10 is not required to have good perf
- l2arc on Intels optane I've already done, but for PBS it would be super luxury! Unfortunately not in the budget

About CPU:
Thank you for pointing that out. I had noticed a kind of task paralelization since I see cores loaded according to the actions, but I have knowing difficulties; if I will have more benefit with 12 cores at 3.2Ghz instead of 8 at 3.7Ghz (for example). These two CPUs are close, but same question remain with 16 cores at 2.6... It's hard to say!
Reading your suggestions, given that I have quite a few different servers that will be using PBS + routine tasks, 8 fast Cores as I imagined might not be ideal. What do you think?

- Do verifies asap.
I know... I have some local snap + tar.gz per night correctly and safely stored

 

[proxstormer]

- Install PBS on baremetal. Backup /etc/proxmox-backup often. The datastore can be connected to PBS easily if you ever reinstall.
- RAIDz performance is limited to the performance of 1 drive. Those drives might be fast enough for you. If unsure get more drives and use RAID10.
- No need for zil, special device or l2arc, will add no performance for an all SSD datastore. Well, maybe l2arc if you add a couple tb ram
- Each task in PBS is siglethreaded, but there might be somewhat high concurrency (many backups at once, restores, GC, verify), so you'll need to get an average of simultaneous tasks and get the fastest CPU with that amount of cores that your budget allows.
- Do verifies asap. Even if ZFS protects from bit rot, a single missing chunk for whatever reason can render many backups useless and you won't notice unless verifies are run regularly.

I have a question with the cores and the threads, are you able to run a GC on 8 cores at same time?

 

[VictorSTS]

I have a question with the cores and the threads, are you able to run a GC on 8 cores at same time?

No. One GC is just one singlethreaded task. You may run parallel GC on different datastores or namespaces. Still, GC isn't CPU intensive, it's mostly limited by storage performance.

 

[VictorSTS]

if I will have more benefit with 12 cores at 3.2Ghz instead of 8 at 3.7Ghz (for example). These two CPUs are close, but same question remain with 16 cores at 2.6... It's hard to say!
Reading your suggestions, given that I have quite a few different servers that will be using PBS + routine tasks, 8 fast Cores as I imagined might not be ideal. What do you think?

As pointed out before: get an average of simultaneous tasks and get the fastest CPU with that amount of cores that your budget allows.
Adding the type of tasks is important too: verify requieres more cpu than a gc, i.e.
PBS isnt too cpu hungry. Maybe your opinion is biased due to your high iowait due to your slow NFS storage and think otherwise. I have a few PBS with around 80TB each on an 8 core ryzen and average cpu usage is around 40% with hdd raidz + ssd special device.

 

[Dunuin]

Last but not least: If it's best to manage disks in ZFS using PBS, then I'm forced to install PBS directly on the hardware. It's impossible to virtualize it while maintaining the same performance, isn't it?

You know that it is possible to install PBS + PVE both bare metal on the same machine?

 

[Adri-M]

Thank Dunuin. Sure! I was thinking more about virtualization. Thank you

Thank you VictorSTS et al. for for those helpful comments.