Server sizing and speed

felipe

Active Member
Oct 28, 2013
221
6
38
Hi,

we now have a 90TB backup servers just as simple nfs storage. With a lot of 8TB disks in one big hardware raid. Now we want PBS. In the docs it says its better with SSD'S (of course) but we also want to have about 90TB and with SSD's it is really expansive. With about 12-16 spinners in a big raid setup it should be fast enough for backuping 2-3 vms at the same time and restore 2-3 vms at the same time?
we only use kvm so its 4mb chunks am i right?
also the question is zfs vs hardware raid. zfs in this case needs a lot more ram ( i guess about 100gb total?). so its more expensive then a hardware raid controller
what are your thoughts?

thank you
 

Dominic

Proxmox Retired Staff
Retired Staff
Mar 18, 2019
1,388
177
68
be fast enough for backuping 2-3 vms at the same time and restore 2-3 vms at the same time?
The question is just how long restoring those 3 VMs takes. Not all HDDs are equal. You will only get a reliable answer to this question by benchmarking your storage.

As a rule of thumb, 1GB RAM per TB RAW disk space for ZFS. See the ZFS performance tips in our administration guide.
 

felipe

Active Member
Oct 28, 2013
221
6
38
So restoring is more io bound then backup? Because restoring was never a problem bevore. Have you got any numbers how many iops it takes?
Using Enterprise SSD's for backup is a little bit expensive. As i wrote generally a big raid with 16 spinners with 7,2k disks will have about 100 iops. if teh chunks are not 4k it should be fast. but how does the restore work?
 

Dominic

Proxmox Retired Staff
Retired Staff
Mar 18, 2019
1,388
177
68
So restoring is more io bound then backup?
Sorry, that's not what I meant. VMs will backup and restore if the storage is slow or fast. The process of backup and restore will just take longer if the storage is slower.

but how does the restore work?
There is a whole section about restoring data in the official documentation.
 

felipe

Active Member
Oct 28, 2013
221
6
38
At the moment we are quite happy with just backuping to nfs. it works really reliable. but backup is very slow for big data because all needs to be send to the storage via lan. so i guess with the new backup server it will be way faster. (after the first initial backup)

do you have any numbers how much iops we need for backup / restore? because now our nfs share has 12 hdds and there is absolutlely no hdd limit. the only limit is the 10g network. but as i read thats different for the pbs. but i have now idea how many iops we need vor lets say for 2 times 200MB restore at the same time.

the question what is fast in your definition... a raid with 16 x 7,2k hdds ist still very fast but not if you need thousandsof iops...

its a calculation for disaster recovery how much time we need to restore all vms
 
Last edited:

felipe

Active Member
Oct 28, 2013
221
6
38
i was playing around with pbs now as a virtual machine and ceph storage as backend (ssd pool)
the backup and restore speed was quite slow (only 30-40mb) i dont know why. network is 10G.
(delta speed on second backup is impressive)
the pbs was at low cpu and had only 60 iops. i dont know why it was slow now still have to test.
but again what can i espect from pbs in terms of speed?
 
Last edited:

symmcom

Renowned Member
Oct 28, 2012
1,084
36
68
Calgary, Canada
www.symmcom.com
First thing to do is to run benchmark on the PBS server and see how fast your server actually is. The output should give you some indication.
Code:
# proxmox-backup-client benchmark
 

felipe

Active Member
Oct 28, 2013
221
6
38
this is the speed on a test VM:
TLS speed: 560.44 MB/s
SHA256 speed: 348.40 MB/s
Compression speed: 456.07 MB/s
Decompress speed: 771.66 MB/s
AES256/GCM speed: 1534.76 MB/s
Verify speed: 243.81 MB/s

on a new production system i guess it will be a lot faster.
does anybody have figures how fast his storage is? specially the restore is important. because backup is sooo fast after the initial backup i dont care.
proxmox writes to use ssd's but a backup store with about 70-80TB in enterprise SSD ist very expensive. so my idea is use 20 spndles in a big raid. or alternatively use it with zfs special device and 2 ssds with 3,84TB.
we will not backup lxc containers . is there any difference in chunck sizes between backing up kvm and lxc?
 

felipe

Active Member
Oct 28, 2013
221
6
38
i now this page. but what does this really say? i did the test on a bps vm. it was in the middle field of the tests. but i did not note much difference between an attached ssd storage and sata storage. and thats exactly what i want to know, TLS speed is allmost the same hdd and ssd both between 550 and 580 MB/s.
to me i looks like this test checks almost only cpu speed. on a production machine i would give a lot of cpu so thats not the point.

the point is do you have any calculation of iops regarding the backup/restore/garbage collection? for kvm backups?

and the main point is i want to calculate if 20 spindles in a big raid setup are fast enogh for restorig 2-3 vms with around 100MB's. i did not order that hardware because i dont want to spend a lot of money seeing that it is to slow.

alternativele adding 2 or 4 ssds as special devices. but how big should they be in percentage of the main storage?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!