Backup Server Hardware Specs & Storage Sizing

[odobrev]

Hello.

First of all, I'm sorry if my English is not good enough, I'm not a native English speaker but I will try my best to make myself understandable so don't hesitate to ask me to rewrite some explanations if they're not clear enough.

We've been using Hyper-V Core 2016 at work with some more or less sketchy backup solutions (some custom scripts at the beginning, then it became unmanageable and we bought Synology NASes for Active Backup for Business which is working OK right now).
The current Hyper-V Server specifications are these :

CPU	2 x Xeon Silver 4116 so 24 Physical Cores
RAM	256 Gb
Storage	3.1 Tb used (RAID 10 of 4 x Kingston DC500R 3.84 Tb)

The number of running VMs is 29 with 3 different SCADA software.
The average CPU load is about 7-10% (yearly stats collected by Zabbix Server). // MAX 64 % during updates
The average Disk Read Rate is 106 r/s. // Max 4903.
The average Disk Write Rate is 107 w/s. // Max 4063
The average Disk Utilization is 3.7%. // Max 96%
(We do have by government obligation a lot more software than people using it...)

We backup every hour on a Synology NAS with ABB (RS1221RP+ using BTRFS deduplication) on RAID 10 of Exos X16 HDDs with a Raid 1 Kingston DC500R (3.84 Tb) SSD Cache => Takes about 3-5 minutes depending on the number of changes (around 7-10Gb per backup).
We keep every hourly version for 48 hours, then daily for one month.
This server is replicated daily to a second backup server situated in another building and there is a 2 X 10 Gbps private direct fiber link between these buildings (can be upgraded if needed, we do have 8 more fibers available).

I've encountered many problems with this Hypervisor and since I've been using Proxmox VE and Proxmox Backup Server at home and on some smaller projects at work and it made me smile instead of wanting to throw the server in the trash every day like it does with Hyper-V, I'm trying to make a quote in order to replace the existing solution by a Proxmox powered one.

The PVE expected server specifications are these :

CPU	32 Cores @3Ghz (or more depending on prices)
RAM	390 Gb (or more depending on prices)
Storage	MAX 8 Tb used (We might go for Micron 7450 MAX or Micron 5400 MAX depending on budget)

The number of VMs might grow to about 45.

The plan is :
1. To get about the same backup plan as the current one 48 hourly versions + daily for a month on two Backup Servers.
2. To be able to backup hourly at a decent speed (about 15 minutes max would be good).
3. To be able to replicate between primary and secondary Backup Server hourly at decent speeds.

The problem is that I can't find enough specific information especially on storage sizing to be able to reliably define a specifications sheet for a Proxmox Backup Server.
If there is a post that I didn't find and that can help me. I would be glad if someone could point me in this direction.
By feeling I would go for at least a 16 Core @3Ghz CPU and 64 Gb of RAM with an EXT4 FS. But could it be less ?
As I understood from the forum posts, HDDs are a no go in production so we're planning to use at least Micron 5400 MAX SSDs or better if budget allows it for PBS.
Is there any formula to calculate how much storage do I need for Proxmox Backup Server depending on how much Gb of Data blocks change every hour or is there a general formula like N GB used in production x Ration (0.5 or 2 or 3 or ?) x Number of backup versions to keep ?

Also would something like Micron 5400 Max be enough for our needs since about same perfoming Kingston DC500R drives are more than enough right now or does Proxmox VE / Backup Server require more IOPS ?

Thank you for your feedback.

 

[tcabernoch]

I'll be watching this thread. I'm not sure if the exact specs or formula you are looking for exists. If it does exist, I want to know about it.

First, and primary question you have that nobody can answer is how much space will you need? We don't know. You don't know. And you won't until you run some backups. A PBS system that's allowed to retain a couple weeks worth of backups will start hitting _unreal_ de-dupe ratios of over 30:1. Your hourly backup scheme is gonna add practically nothing each time it runs.
The real question is how much delta do you have, I see you have some estimates of the delta your current backup system is managing, but expect wildly different results out of PBS.
So ... How much space do you need? Dunno. Spin up a system and try some testing. Or make a guess. Maybe survey folks as to how many VMs they are backing up, how much retention is allowed, and how much it fills up. If you search this forum for the terms 'pruning', GC, or 'garbage collection', you'll see lots of discussions of retention and datastore size.

Yes, you want all SSD on the PBS server. Or whatever you can afford. I build SSD/HDD special vdev aka Fusion pools, but I did some heavy testing to validate that first.

PBS doesn't multiplex much. Seems to use one or two threads per backup. High core-count servers aren't really an advantage. Turn off hyperthreading. Again, I haven't seen a formula, but my experience tells me PBS needs roughly 4 cores per PVE doing simultaneous backup. Backing up 4 PVE servers at the same time? 16 (full physical) cores should be fine.

RAM ... Um it likes RAM, but isn't too much of a hog. Your proposed 64gb is gonna be plenty for many scenarios.

And this EXT4 FS ... Well, go with what ya know. In the Proxmox world, you see a lot of ZFS. ZFS let's you do a lotta wonderful things, and I use it everywhere. Maybe you don't want to learn obscure database/filesystem concepts, and EXT4 is indeed fine for you.

I hope this info from my own experience is helpful. Somebody might have actual vendor approved info. If so, listen to them first.

 

[odobrev]

I'll be watching this thread. I'm not sure if the exact specs or formula you are looking for exists. If it does exist, I want to know about it.

First, and primary question you have that nobody can answer is how much space will you need? We don't know. You don't know. And you won't until you run some backups. A PBS system that's allowed to retain a couple weeks worth of backups will start hitting _unreal_ de-dupe ratios of over 30:1. Your hourly backup scheme is gonna add practically nothing each time it runs.
The real question is how much delta do you have, I see you have some estimates of the delta your current backup system is managing, but expect wildly different results out of PBS.
So ... How much space do you need? Dunno. Spin up a system and try some testing. Or make a guess. Maybe survey folks as to how many VMs they are backing up, how much retention is allowed, and how much it fills up. If you search this forum for the terms 'pruning', GC, or 'garbage collection', you'll see lots of discussions of retention and datastore size.

Yes, you want all SSD on the PBS server. Or whatever you can afford. I build SSD/HDD special vdev aka Fusion pools, but I did some heavy testing to validate that first.

PBS doesn't multiplex much. Seems to use one or two threads per backup. High core-count servers aren't really an advantage. Turn off hyperthreading. Again, I haven't seen a formula, but my experience tells me PBS needs roughly 4 cores per PVE doing simultaneous backup. Backing up 4 PVE servers at the same time? 16 (full physical) cores should be fine.

RAM ... Um it likes RAM, but isn't too much of a hog. Your proposed 64gb is gonna be plenty for many scenarios.

And this EXT4 FS ... Well, go with what ya know. In the Proxmox world, you see a lot of ZFS. ZFS let's you do a lotta wonderful things, and I use it everywhere. Maybe you don't want to learn obscure database/filesystem concepts, and EXT4 is indeed fine for you.

I hope this info from my own experience is helpful. Somebody might have actual vendor approved info. If so, listen to them first.

First of all, thank you for sharing your knowledge and taking your time to answer.

I actually do have a PVE and PBS server running on a lot smaller load.
It took me some time to compile data from logs but here are some results.

First, let's define the environment :
PVE :

CPU	Atom C3758
RAM	64 GB
Storage	2 x 480 GB Kingston DC500M
FS	local-zfs MIRROR with LVM-THIN
Network Speed	1 Gbps

PBS :

CPU	Xeon E3-1226 v3
RAM	16 GB
Storage	4 x Slow consumer HDD
FS	ZFS rpool with 2 mirrored VDEVS
Network Speed	1 Gbps

Now, let's define the backup load :

Total VMs Size on Disk	250 GiB so around 269 GB
Average Backup Time (from the last 24 hourly Backup logs)	2 minutes 5 seconds
Average Transferred Backup Size from PVE to PBS (from the last 24 hourly Backup logs)	7.39 GB
Number of VMs to backup	7

It's a mixed load since most VMs don't change much (freepbx without call recording, wiki, unms, etc.) but there is one "heavy" load from Zabbix server which collects 5222 data points with a performance indicator of 90.46 values polled every second.

Let's now see what happens in PBS data store :

If my calculations are OK, seems like a 1.95 ratio from "real VMs size" to "PBS datastore usage".

Also if I follow the recommendations I saw on different forum posts (and my own crashes on first PBS usages), I should always let about 20% of storage unused.

So If I follow my own tests based on this small workload, I should apply a 2.15 ratio from "VMs real size" to "PBS storage to buy".
So for a 8 TB VMs storage used, I should get at least 17.2 TB of PBS storage for Data Store.
But that's the thing, even if the backup strategy is the same, can't define if I can scale this workload around the same for the bigger server.


As for ZFS, I'm all keen to using it on PVE but not so sure about PBS ?
I read a lot of posts and as I understood, PBS does not use ZFS de-duplication but it's own software based one such a QNAP does.
ZFS brings some good stuff such as replication between servers (really good if you don't want to go all in for Ceph with a small HA Cluster), de-duplication, compression, etc. but is also a software based solution that does complicate some simple operations such as HOT DRIVE replacement which is really easy with a hardware raid (and then you don't really need ZFS anymore).

What do you think, should I try to go with a 2.15 ratio or wait for some more experienced people to check on this post ?

 

[tcabernoch]

Your fuzzy math all looks fine to me. Call it an educated guess instead of a calculation.
Here's the thing. If those are all Windows VMs, and you added a Linux VM to backup, your entire first backup will be copied to PBS with little to no dedupe.
On the other hand, if you added another Windows backup, even if its an entirely brand-new VM installed from scratch, you are going to get a significant dedup ratio regardless.
Or the other way around. Whatever. The point is that you'll see a LOT of dedupe out of completely unrelated machines if they have the same root OS.
And then, if you throw encrypted VMs or backups into the mix, its gonna skew your ratio again.
So, maybe you have estimated low. Maybe you've estimated high. Give yourself some wiggle room.

Also ... I'm sure its obvious, the Dedup Factor listed by PBS depends entirely on your retention and how many backup runs exist.
You were talking about hourly backups. PBS is gonna claim that it has a full copy, made out of recycled parts, except for the delta in the last hour.
So for just one VM with zero delta over 24 hours, its gonna claim something like 23:1. Obviously, that's BS. So ya, PBS gives wildly different stats than other calculation methods.

Perhaps someone else will chime in. Good luck.

 

[Falk R.]

As for ZFS, I'm all keen to using it on PVE but not so sure about PBS ?
I read a lot of posts and as I understood, PBS does not use ZFS de-duplication but it's own software based one such a QNAP does.
ZFS brings some good stuff such as replication between servers (really good if you don't want to go all in for Ceph with a small HA Cluster), de-duplication, compression, etc. but is also a software based solution that does complicate some simple operations such as HOT DRIVE replacement which is really easy with a hardware raid (and then you don't really need ZFS anymore).

I can only agree with the previous speaker.
ZFS also has other functions such as bit rot detection.
The dedup of the PBS is not ZFS based, so the dedup also works if you have an EXT4 or XFS datastore.
I like to use ZFS in the PBS, but if I have an old Veeam server with not so good disks or SSDs but with an good raid controller, then I use xfs on the hardware raid.
I always use a second PBS instance to replicate the backups and the remote sync only ever fetches the changed chunks after deduplication.
Since the secondary PBS does a pull, I configure the firewall on the server so that no incoming connections are allowed. There is no better ransomware protection than a completely closed firewall.