Questions about log retention

copec

New Member
Jun 24, 2023
18
2
3
We are testing out PBS to use with PVE. We need to be able to prove that we performed backups on certain historical dates in order to prove that we are keeping an operational backup system in place as part of a certification process. I'm looking at the /var/log on PBS, as well as the UI, and the documentation. Is there a historical log kept of all the backups that have taken place?
 
Is there a historical log kept of all the backups that have taken place?
Yes, but that will be rotated.

Currently, the backup server creates a task log entry for every backup – you can easily search them by going to Administration -> Tasks, there you can enable a filter for the "backup" task type.

Those logs are saved in /var/log/pve/tasks , with two characters from the UPID's pstart (process start time, the third field of a UPID:<node>:<pid>:<pstart>:<task-id>:<start-time>:<worker-type>:<worker-id>:<username> task ID) as directory-index to avoid having to many files in a single directory.

For those the following retention is happening:
There are archive files that act as an index of (finished) task entries, as long as a task log is referenced by one of those it's kept, otherwise it will get removed.

The archive logs are rotated, by a default mechanism of "rotate if bigger than 512 KiB in size" and if such a rotation happens only the newest 20 archive-files will be kept – this amounts to roughly 100'000 task entries that are saved.

But, you can also override this behavior by setting a task-log-max-days from the Proxmox Backup Server's node.cfg, if that's set then the archive's will be searched and only those will be removed that contain tasks that are older than the now - max-days cut-off.

You can set that config using proxmox-backup-manager node update --task-log-max-days 365 (here ensuring that all task that started in the last 365 days will be kept).

Additionally, Proxmox VE uploads the client task-log from a backup also to the backup server, but that's saved within the backup snapshot itself, so if that's removed by a user that log is also gone, so probably not useful for your audit purpose.
 
FWIW, you could also add some tooling that uses the Proxmox Backup Server periodically saves the list of created backups in some directory, or database of your own.

Even using the debug tooling we provide could be enough, e.g.:
proxmox-backup-debug api get /nodes/localhost/tasks --typefilter=backup --start=0 --limit=500 --output-format json-pretty

That would give you the last 500 tasks of type backup - there you have information like what user made a backup, if the task finished with OK status, how long it took, and the worker ID which allows one to map this backup to a datastore and backup group, so should be enough for most such auditing purposes.
 
  • Like
Reactions: VictorSTS and copec

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!