Server sizing and speed

[felipe]

Hi,

we now have a 90TB backup servers just as simple nfs storage. With a lot of 8TB disks in one big hardware raid. Now we want PBS. In the docs it says its better with SSD'S (of course) but we also want to have about 90TB and with SSD's it is really expansive. With about 12-16 spinners in a big raid setup it should be fast enough for backuping 2-3 vms at the same time and restore 2-3 vms at the same time?
we only use kvm so its 4mb chunks am i right?
also the question is zfs vs hardware raid. zfs in this case needs a lot more ram ( i guess about 100gb total?). so its more expensive then a hardware raid controller
what are your thoughts?

thank you

 

[Dominic]

be fast enough for backuping 2-3 vms at the same time and restore 2-3 vms at the same time?

The question is just how long restoring those 3 VMs takes. Not all HDDs are equal. You will only get a reliable answer to this question by benchmarking your storage.

As a rule of thumb, 1GB RAM per TB RAW disk space for ZFS. See the ZFS performance tips in our administration guide.

 

[felipe]

So restoring is more io bound then backup? Because restoring was never a problem bevore. Have you got any numbers how many iops it takes?
Using Enterprise SSD's for backup is a little bit expensive. As i wrote generally a big raid with 16 spinners with 7,2k disks will have about 100 iops. if teh chunks are not 4k it should be fast. but how does the restore work?

 

[Dominic]

So restoring is more io bound then backup?

Sorry, that's not what I meant. VMs will backup and restore if the storage is slow or fast. The process of backup and restore will just take longer if the storage is slower.

but how does the restore work?

There is a whole section about restoring data in the official documentation.

 

[felipe]

At the moment we are quite happy with just backuping to nfs. it works really reliable. but backup is very slow for big data because all needs to be send to the storage via lan. so i guess with the new backup server it will be way faster. (after the first initial backup)

do you have any numbers how much iops we need for backup / restore? because now our nfs share has 12 hdds and there is absolutlely no hdd limit. the only limit is the 10g network. but as i read thats different for the pbs. but i have now idea how many iops we need vor lets say for 2 times 200MB restore at the same time.

the question what is fast in your definition... a raid with 16 x 7,2k hdds ist still very fast but not if you need thousandsof iops...

its a calculation for disaster recovery how much time we need to restore all vms

 

[felipe]

i was playing around with pbs now as a virtual machine and ceph storage as backend (ssd pool)
the backup and restore speed was quite slow (only 30-40mb) i dont know why. network is 10G.
(delta speed on second backup is impressive)
the pbs was at low cpu and had only 60 iops. i dont know why it was slow now still have to test.
but again what can i espect from pbs in terms of speed?

 

[wahmed]

First thing to do is to run benchmark on the PBS server and see how fast your server actually is. The output should give you some indication.

# proxmox-backup-client benchmark

 

[felipe]

this is the speed on a test VM:
TLS speed: 560.44 MB/s
SHA256 speed: 348.40 MB/s
Compression speed: 456.07 MB/s
Decompress speed: 771.66 MB/s
AES256/GCM speed: 1534.76 MB/s
Verify speed: 243.81 MB/s

on a new production system i guess it will be a lot faster.
does anybody have figures how fast his storage is? specially the restore is important. because backup is sooo fast after the initial backup i dont care.
proxmox writes to use ssd's but a backup store with about 70-80TB in enterprise SSD ist very expensive. so my idea is use 20 spndles in a big raid. or alternatively use it with zfs special device and 2 ssds with 3,84TB.
we will not backup lxc containers . is there any difference in chunck sizes between backing up kvm and lxc?

 

[felipe]

does nobody have examples of production servers? and sizing?

 

[tom]

does nobody have examples of production servers? and sizing?

See also https://forum.proxmox.com/threads/how-fast-is-your-backup-datastore-benchmark-tool.72750/

 

[felipe]

i now this page. but what does this really say? i did the test on a bps vm. it was in the middle field of the tests. but i did not note much difference between an attached ssd storage and sata storage. and thats exactly what i want to know, TLS speed is allmost the same hdd and ssd both between 550 and 580 MB/s.
to me i looks like this test checks almost only cpu speed. on a production machine i would give a lot of cpu so thats not the point.

the point is do you have any calculation of iops regarding the backup/restore/garbage collection? for kvm backups?

and the main point is i want to calculate if 20 spindles in a big raid setup are fast enogh for restorig 2-3 vms with around 100MB's. i did not order that hardware because i dont want to spend a lot of money seeing that it is to slow.

alternativele adding 2 or 4 ssds as special devices. but how big should they be in percentage of the main storage?

 

[dipendra.chauhan]

Can anyone please help me with a backup sizing tool for proxmox backupserver. How to size the backup storage required for a specific retention needs ?

 

[UdoB]

How to size the backup storage required for a specific retention needs ?

Only you can know how much data you have - and how much data changes every day (per backup interval). The size you need in the PBS is roughly like this:

P = plain data size you have now (or plan to have in the near future)
N = newly modified data per backup interval
K = number of backups kept
R = reserved = 20% of the capacity left free forever

C = Capacity = P + N*K + R

PBS does compression; I have skipped that aspect here. The effect can go from "needs only 50%" for light data to "needs 99%" for already compressed files.

In my specific world I can get a "deduplication factor" of 25 --> each new added backup adds 4 percent of its size.

 

[wahmed]

I agree with what @UdoB said in the previous post.

In addition to that, here is a scenario that should help you understand slightly better and size your PBS:
In this scenario, the space used by all VM is 50TB. It is a rough estimate. Hope this helps.

Retention is 3 daily, 2 weekly, 6 monthly, 5% daily change.

1. Initial Full Backup:
   50TB (initial full backup)
2. Daily Incremental Backups:
   Daily changes: 50TB * 5% = 2.5TB
   With dedup/compression: 2.5TB * 50% = 1.25TB per daily backup
3. Weekly Backups:
   Weekly backups are incremental from the last backup of the previous week.
   Weekly change (assuming 7 days): 50TB * 5% * 7 = 17.5TB
   With dedup/compression: 17.5TB * 50% = 8.75TB per weekly backup
4. Monthly Backups:
   Monthly backups are incremental from the last backup of the previous month.
   Monthly change (assuming 30 days): 50TB * 5% * 30 = 75TB
   With dedup/compression: 75TB * 50% = 37.5TB per monthly backup

Storage Requirements

1. Initial Full Backup:
   50TB * 50% = 25TB
2. Last 3 Daily Backups:
   3 * 1.25TB = 3.75TB
3. 2 Weekly Backups:
   2 * 8.75TB = 17.5TB
4. 6 Monthly Backups:
   6 * 37.5TB = 225TB

Total Estimated Storage:
25TB + 3.75TB + 17.5TB + 225TB = 271.25TB

 

[UdoB]

I agree with what @UdoB said in the previous post.

Thanks.

the space used by all VM is 50TB.
Retention is 3 daily, 2 weekly, 6 monthly, 5% daily change.

Monthly Backups:
Monthly backups are incremental from the last backup of the previous month.
Monthly change (assuming 30 days): 50TB * 5% * 30 = 75TB
With dedup/compression: 75TB * 50% = 37.5TB per monthly backup

I do not buy this. I would really not expect an "every day 5%" to generate 75TB to backup - between any two dates of the month or a year. 100 percent is 50TB - and that is the absolute maximum change in data, isn't it?

The other extreme would be "it stays just at 5%" which is also wrong, for sure. Only the monitoring software can tell the truth...

 

[wahmed]

I do not buy this. I would really not expect an "every day 5%" to generate 75TB to backup -

That is correct. It is not a concrete number. Only the administrator of the environment can tell how much new data is being written on day to day basis. 5% in my opinion is a safer overall figure for the sake of calculation with little bit of overestimation. There are some environment which does generate roughly 17% avg new data day to day. There are also environment which generates around 0.20% new data daily. 5% figure aside, the calculation has been used on 33 real Proxmox environment where similar calculation with various daily change rate has been used.

 

[dipendra.chauhan]

Discussions and details have been really helpful. Thankyou @wahmed and @UdoB