Disk usage under /var/log/proxmox-backup

[NomadCF]

We're seeing some high disk usage under /var/log/proxmox-backup/. It's varying from a few MB to +200G. What this is temporary data and does /var/logs make the most sense for this data to be temporarily stored ?

 

[Dunuin]

PBS is writing some lines for each chunk that is written. You probably got millions of chunks so millions of log lines get created when doing stuff.
I also searched for a way to change the log severenity but looks like it isn't possible to drop those info level log lines.

 

[NomadCF]

That's extraordinarily weird, as I wouldn't expect log data of any kind to be transient like that.

It's a simple enough matter of switching rsyslog to ignore those lines if in fact that's what it is.

 

[ggoller]

Hey,
the loglevel right now is fixed on "info" for api/proxy. There is a patch in progress for controlling it using an env variable.
Preliminary patch: https://lists.proxmox.com/pipermail/pbs-devel/2023-August/006486.html (not merged and probably subject to change, just fyi)

 

[NomadCF]

Hey,
the loglevel right now is fixed on "info" for api/proxy. There is a patch in progress for controlling it using an env variable.
Preliminary patch: https://lists.proxmox.com/pipermail/pbs-devel/2023-August/006486.html (not merged and probably subject to change, just fyi)

This is wonderful to hear, but in the mean time is would be the impact if these log files weren't able to be written to ? Say due to filling the disk ?

 

[steff123]

Same problem here.
@ggoller Is there a cleanup by default in PBS? The main part of my files was older than 90 days.

I delete the old stuff with

find /var/log/proxmox-backup -mtime +90 -delete

and activated

proxmox-backup-manager node update --task-log-max-days 90

And hope that's not breaking something.

 

[ggoller]

Hi,
nope, this should be safe...
Deleting old stuff with

find /var/log/proxmox-backup -mtime +90 -delete

and activating:

proxmox-backup-manager node update --task-log-max-days 90

is a good idea.

The only thing that could be a problem here would be deleting a task log that is referenced by a jobstate file (a running job): https://bugzilla.proxmox.com/show_bug.cgi?id=4895
But if you only delete files older than 90 days you should be fine.

 

[NomadCF]

Our current issue stems from using PBS-client to back up approximately 22TB of data every day. This process creates substantial log directories, as mentioned earlier. These extensive logs are not only consuming a significant amount of our storage space but also wearing down our SSDs to the point where we had to transition to using HDDs to store the log files. To mitigate this problem, we are seeking a way to either disable these logs.

 

[ggoller]

Our current issue stems from using PBS-client to back up approximately 22TB of data every day. This process creates substantial log directories, as mentioned earlier. These extensive logs are not only consuming a significant amount of our storage space but also wearing down our SSDs to the point where we had to transition to using HDDs to store the log files. To mitigate this problem, we are seeking a way to either disable these logs.

When the patch is merged you will be able to set the LogLevel to `OFF`.

 

[PelleHanses]

When the patch is merged you will be able to set the LogLevel to `OFF`.

Is this path merged yet? The fills my disk every day now with 90GB data (then the root disk is full).
If it is merged - where do I set the log level?

If it is not merged, can I move my log files from /var/log/proxmox-backup/tasks to a Datastore (just making a link to that folder instead of /var/log/proxmox-backup/tasks)? Is it a really bad idea or can it be acceptable until the patch is introduced?

 

[ggoller]

Patch is not yet merged, but it should be fine (AFAIK) to link log output in another dir.

 

[sistemi-union]

Hi all, I'm having the same exact problem on a production environment. Last year we had to replace the system NVMe due to SMART error, now we're in the same situation and were investigating. Here are the statistics:



The weird 661TB written in 6937 hours (~290 days) is completely crazy. At the moment, I moved and then soft-linked /var/log/proxmox-backup/ in an HDD partition under the zfs storage pool, but I suspect that an update could "break" the changes.

@ggoller could you provide us any release date for the patch to be public?

Thanks in advance

 

[ggoller]

Hi!
sadly I can't give you a release date. This is quite an extensive patch series and has yet to be reviewed properly on the mailing list.
Soft-linking /var/log/proxmox-backup/ should work fine though. I don't think we will be changing this path anytime soon!

 

[stv.Bundesadmin]

Glad that there is a pending patch, can't wait for it to land ;-)

On the source-side, a remote-sync-job causes a lot of storage growth & IOPS on /var/log/proxmox and brutally spams the journalctl with useless info (endless lines of...)
"proxmox-backup-proxy[4544]: download chunk "/data/DATA/.chunks/4dd5/4dd58422849710b079a0716bd..."