Get stats on backups

[kurdam]

Hi,

I was wondering if there is a way to get a backup report like veeam where you can have a report on a pool or even better globally on the total volume of data uploaded on the PBS server.
I found a way to have it per VM on the PBS interface but i can't find any more general view.

I'm asking that because we are looking to rent a S3 compatible storage (we are planning on using the upcoming tape backup to S3 to sync all our backups to a bucket). I was taking with their support and they asked me what volume of data we are uploading each day on our server. I'm currently not able to answer this question without looking in each individual jobs and adding them up.

It would be nice to have a more general view on the backups and to see globally what happened, how much more space has been used or freed on the datastores, an average on network usage ...

 

[dcsapak]

problaby not exactly what you mean but we do collect the usage data of the datastore over time into an rrd database. with that data you can get an idea how much the storage requirement grow and shrink over time
this is also exported when you use an external metric server (influxdb)

 

[kurdam]

Yes i'm using the graph you provide in the datastore view to predict and monitor my datastore growth but i think it would be nice to have a bit more infos than that, maybe in an advanced tab somewhere. Just a way for us to analyze a bit more what is going on in term of ressources consumption.
We are monitoring our servers health with Zabbix. Maybe by looking into the stats there i will be able to have a bit more infos.

 

[dcsapak]

mhmm.. what exactly more infos would you need then? we cannot give a growth rate by backup group, because this would be very misleading since all chunks are deduplicated across the whole datastore
so the only real usage statistic is the datastore usage

 

[fabian]

https://bugzilla.proxmox.com/show_bug.cgi?id=4506

 

[kurdam]

Something like this :



But on all jobs or a pool job on a selected period (like you do with the cog wheel where you can select the numbers of days) to have a rough idea of the total volume changed, uploaded...

But more important for me is the volume of datas transferred.
If you imagine that someday it will be possible to add an S3 storage as a datastore, the amount of datas transferred counts in the price of the S3 so it would be nice to have a gross overview of that metric as well as to know that each day, on that specific pool a day of retention added is about 50 GB more on my storage.
It's important to know how much traffic you upload on your wan link to determine its speed (100M, 1G, 10G) and not impact your end users by saturating the wan link with the backup export or by not managing to upload all your datas in 24h.
tldr: it could be a nice tool to predict the impact of backup exports on the network infrastructure as well as the cost of exporting to a S3 storage.

 

[fabian]

there already are upload stats, but those obviously only capture that particular run. in the client log/output:

INFO: root.pxar: had to backup 48.646 MiB of 1.267 GiB (compressed 7.515 MiB) in 1.47s
INFO: root.pxar: average backup speed: 33.017 MiB/s
INFO: root.pxar: backup was done incrementally, reused 1.22 GiB (96.3%)

in the manifest (which also contains the sizes of the backup contents as a comparison), from the client's PoV:

 "unprotected": {
    "chunk_upload_stats": {
      "compressed_size": 8121427,
      "count": 8,
      "duplicates": 0,
      "size": 51607814
    },

in the server-side task log (from the server's PoV, this would also show any chunks that already existed on the server that the client didn't know about and uploaded):

2024-04-26T10:46:53+02:00: POST /dynamic_close
2024-04-26T10:46:53+02:00: Upload statistics for 'root.pxar.didx'
2024-04-26T10:46:53+02:00: UUID: a263f20de4994493bcfca3ab689481f4
2024-04-26T10:46:53+02:00: Checksum: a9dd0b2ec7e4a84d0bde8019c06e4bbbca4f2d2b2fb16a04e577901acadf982d
2024-04-26T10:46:53+02:00: Size: 1360563006
2024-04-26T10:46:53+02:00: Chunk count: 306
2024-04-26T10:46:53+02:00: Upload size: 51008706 (3%)
2024-04-26T10:46:53+02:00: Duplicates: 299+0 (97%)
2024-04-26T10:46:53+02:00: Compression: 15%
2024-04-26T10:46:53+02:00: successfully closed dynamic index 2