Disk usage under /var/log/proxmox-backup

NomadCF

Active Member
Dec 20, 2017
27
1
43
42
We're seeing some high disk usage under /var/log/proxmox-backup/. It's varying from a few MB to +200G. What this is temporary data and does /var/logs make the most sense for this data to be temporarily stored ?
 
PBS is writing some lines for each chunk that is written. You probably got millions of chunks so millions of log lines get created when doing stuff.
I also searched for a way to change the log severenity but looks like it isn't possible to drop those info level log lines.
 
That's extraordinarily weird, as I wouldn't expect log data of any kind to be transient like that.

It's a simple enough matter of switching rsyslog to ignore those lines if in fact that's what it is.
 
Last edited:
Same problem here.
@ggoller Is there a cleanup by default in PBS? The main part of my files was older than 90 days.

I delete the old stuff with
Code:
find /var/log/proxmox-backup -mtime +90 -delete

and activated
Code:
proxmox-backup-manager node update --task-log-max-days 90

And hope that's not breaking something.
 
Hi,
nope, this should be safe...
Deleting old stuff with
Code:
find /var/log/proxmox-backup -mtime +90 -delete
and activating:
Code:
proxmox-backup-manager node update --task-log-max-days 90
is a good idea.

The only thing that could be a problem here would be deleting a task log that is referenced by a jobstate file (a running job): https://bugzilla.proxmox.com/show_bug.cgi?id=4895
But if you only delete files older than 90 days you should be fine.
 
Last edited:
Our current issue stems from using PBS-client to back up approximately 22TB of data every day. This process creates substantial log directories, as mentioned earlier. These extensive logs are not only consuming a significant amount of our storage space but also wearing down our SSDs to the point where we had to transition to using HDDs to store the log files. To mitigate this problem, we are seeking a way to either disable these logs.
 
Our current issue stems from using PBS-client to back up approximately 22TB of data every day. This process creates substantial log directories, as mentioned earlier. These extensive logs are not only consuming a significant amount of our storage space but also wearing down our SSDs to the point where we had to transition to using HDDs to store the log files. To mitigate this problem, we are seeking a way to either disable these logs.
When the patch is merged you will be able to set the LogLevel to `OFF`.
 
When the patch is merged you will be able to set the LogLevel to `OFF`.
Is this path merged yet? The fills my disk every day now with 90GB data (then the root disk is full).
If it is merged - where do I set the log level?

If it is not merged, can I move my log files from /var/log/proxmox-backup/tasks to a Datastore (just making a link to that folder instead of /var/log/proxmox-backup/tasks)? Is it a really bad idea or can it be acceptable until the patch is introduced?
 
Last edited:
Patch is not yet merged, but it should be fine (AFAIK) to link log output in another dir.
 
Hi all, I'm having the same exact problem on a production environment. Last year we had to replace the system NVMe due to SMART error, now we're in the same situation and were investigating. Here are the statistics:

1715293182230.png

The weird 661TB written in 6937 hours (~290 days) is completely crazy. At the moment, I moved and then soft-linked /var/log/proxmox-backup/ in an HDD partition under the zfs storage pool, but I suspect that an update could "break" the changes.

@ggoller could you provide us any release date for the patch to be public?

Thanks in advance :)
 
Hi!
sadly I can't give you a release date. This is quite an extensive patch series and has yet to be reviewed properly on the mailing list.
Soft-linking /var/log/proxmox-backup/ should work fine though. I don't think we will be changing this path anytime soon!
 
Glad that there is a pending patch, can't wait for it to land ;-)

On the source-side, a remote-sync-job causes a lot of storage growth & IOPS on /var/log/proxmox and brutally spams the journalctl with useless info (endless lines of...)
"proxmox-backup-proxy[4544]: download chunk "/data/DATA/.chunks/4dd5/4dd58422849710b079a0716bd..."
 

Attachments

  • pbs-sync-source-high-log-iops.jpg
    pbs-sync-source-high-log-iops.jpg
    310.2 KB · Views: 5

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!