Disk usage under /var/log/proxmox-backup

N

NomadCF

Active Member
Dec 20, 2017
29
1
43
43
We're seeing some high disk usage under /var/log/proxmox-backup/. It's varying from a few MB to +200G. What this is temporary data and does /var/logs make the most sense for this data to be temporarily stored ?
 
PBS is writing some lines for each chunk that is written. You probably got millions of chunks so millions of log lines get created when doing stuff.
I also searched for a way to change the log severenity but looks like it isn't possible to drop those info level log lines.
 
That's extraordinarily weird, as I wouldn't expect log data of any kind to be transient like that.

It's a simple enough matter of switching rsyslog to ignore those lines if in fact that's what it is.
 
Last edited:
  • Like
Reactions: Kodey, Neobin and Dunuin
Same problem here.
@ggoller Is there a cleanup by default in PBS? The main part of my files was older than 90 days.

I delete the old stuff with
Code: 
find /var/log/proxmox-backup -mtime +90 -delete

and activated
Code: 
proxmox-backup-manager node update --task-log-max-days 90

And hope that's not breaking something.
 
Hi,
nope, this should be safe...
Deleting old stuff with
Code: 
find /var/log/proxmox-backup -mtime +90 -delete
and activating:
Code: 
proxmox-backup-manager node update --task-log-max-days 90
is a good idea.

The only thing that could be a problem here would be deleting a task log that is referenced by a jobstate file (a running job): https://bugzilla.proxmox.com/show_bug.cgi?id=4895
But if you only delete files older than 90 days you should be fine.
 
Last edited:
Our current issue stems from using PBS-client to back up approximately 22TB of data every day. This process creates substantial log directories, as mentioned earlier. These extensive logs are not only consuming a significant amount of our storage space but also wearing down our SSDs to the point where we had to transition to using HDDs to store the log files. To mitigate this problem, we are seeking a way to either disable these logs.
 
NomadCF said:
Our current issue stems from using PBS-client to back up approximately 22TB of data every day. This process creates substantial log directories, as mentioned earlier. These extensive logs are not only consuming a significant amount of our storage space but also wearing down our SSDs to the point where we had to transition to using HDDs to store the log files. To mitigate this problem, we are seeking a way to either disable these logs.
Click to expand...
When the patch is merged you will be able to set the LogLevel to `OFF`.
 
ggoller said:
When the patch is merged you will be able to set the LogLevel to `OFF`.
Click to expand...
Is this path merged yet? The fills my disk every day now with 90GB data (then the root disk is full).
If it is merged - where do I set the log level?

If it is not merged, can I move my log files from /var/log/proxmox-backup/tasks to a Datastore (just making a link to that folder instead of /var/log/proxmox-backup/tasks)? Is it a really bad idea or can it be acceptable until the patch is introduced?
 
Last edited:
Patch is not yet merged, but it should be fine (AFAIK) to link log output in another dir.
 
Hi all, I'm having the same exact problem on a production environment. Last year we had to replace the system NVMe due to SMART error, now we're in the same situation and were investigating. Here are the statistics:

1715293182230.png

The weird 661TB written in 6937 hours (~290 days) is completely crazy. At the moment, I moved and then soft-linked /var/log/proxmox-backup/ in an HDD partition under the zfs storage pool, but I suspect that an update could "break" the changes.

@ggoller could you provide us any release date for the patch to be public?

Thanks in advance :)
 
Hi!
sadly I can't give you a release date. This is quite an extensive patch series and has yet to be reviewed properly on the mailing list.
Soft-linking /var/log/proxmox-backup/ should work fine though. I don't think we will be changing this path anytime soon!
 
  • Like
Reactions: uiffiu
Glad that there is a pending patch, can't wait for it to land ;-)

On the source-side, a remote-sync-job causes a lot of storage growth & IOPS on /var/log/proxmox and brutally spams the journalctl with useless info (endless lines of...)
"proxmox-backup-proxy[4544]: download chunk "/data/DATA/.chunks/4dd5/4dd58422849710b079a0716bd..."
 

Attachments

  • pbs-sync-source-high-log-iops.jpg
    pbs-sync-source-high-log-iops.jpg
    310.2 KB · Views: 14
I've been researching this issue and have a heads-up about an upcoming feature of systemd.exec . With version 253 (currently pve and pbs are using version 252) there will be a new option "LogFilterPatterns=~extended regular expression" to discard log entries based on the MESSAGE= field of the incoming structured message. This option can be added multiple times to the [Service] section of a system service. I could not test, but hopefully this will take care of the systemd journal spamming (chunks) by pbs.

As for the /var/log/proxmox-backup/ folder, I was hoping that release 3,3 of pbs would include patch https://lists.proxmox.com/pipermail/pbs-devel/2023-August/006486.html . ggoller, please provide a status update for this patch.
 
The patch you linked is outdated, I ended up moving this stuff and migrating to tracing directly. The main tracing infrastructure has already been merged, so in the latest version (3.3) you should be able to set the PBS_LOG environment variable to your max log level. So for example you could use systemctl edit proxmox-backup.service or/and systemctl edit proxmox-backup-proxy.service and insert Environment="PBS_LOG=error" in the [Service] section. E.g.:
Code: 
[Service]
Environment="PBS_LOG=error"

Apologies for not updating the documentation, will submit a patch soon.
More fine-grained control is also planned in the future: https://bugzilla.proxmox.com/show_bug.cgi?id=6081 (because the systemd version of matching messages is kind of janky :) )
 
There is a reference about PBS_LOG in the current documentation, but it is intended to set logging verbosity for the command-line tools. I didn't think it could be used for the services running on the pbs server. Will try. In the meantime, I used another tweak of systemd.exec, adding:

Code: 
[Service]
LogRateLimitIntervalSec=1s
LogRateLimitBurst=16

to the proxmox-backup-proxy.service. With this setting over 1100 copies of this message:

Code: 
Jan 20 09:10:08 pbs111 proxmox-backup-proxy[1235]: UPLOAD END 4194304 bytes

are suppressed from a single run of "proxmox-backup-client benchmark".
 
Unfortunately it does not work in my case. When adding:

Code: 
[Service]
Environment="PBS_LOG=off"

to the proxmox-backup-proxy.service, and running "proxmox-backup-client benchmark" on a pve node, the first and last messages are dropped, but all the "UPLOAD" messages (>1400) are still logged in the pbs server:

Code: 
This message is suppressed >>-->Jan 20 10:20:51 pbs111 proxmox-backup-proxy[2082]: starting new benchmark on datastore 'PBSdatastore' from ::ffff:192.168.212.131: "host/benchmark/2025-01-20T15:20:51Z"
Jan 20 10:20:51 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
Jan 20 10:20:51 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
Jan 20 10:20:51 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
Jan 20 10:20:51 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
.
over 1400 of those messages are still logged
.
Jan 20 10:20:56 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
Jan 20 10:20:56 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
Jan 20 10:20:56 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
Jan 20 10:20:56 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
Jan 20 10:20:56 pbs111 proxmox-backup-proxy[2082]: UPLOAD END 4194304 bytes
This message is suppressed >>-->Jan 20 10:20:56 pbs111 proxmox-backup-proxy[2082]: benchmark finished successfully
 
Just had a look at the code and it seems this line is printed manually to stdout using println without using our usual logging (https://git.proxmox.com/?p=proxmox-...a1d8a1b54051b4c5ce63f5c66f1ce6cf;hb=HEAD#l270). I will send a patch soon. (I generally have to go through the whole code base another time and convert all the println's and eprintln's to real logs.)

In the meantime you can add StandardOutput=null to the [Service] section in the systemd override file. Also note two things: I don't recommend having the loglevel "off" as you won't see anything; and there is currently an issue that the result of a task will not be written correctly if the loglevel is higher than info (this means that the status will be unknown, although the task will be run correctly. This will be fixed with: https://lore.proxmox.com/pbs-devel/20250120102657.290339-1-g.goller@proxmox.com/T/#u).
 
Last edited:
Shouldn't it be possible to use logrotate to control the amount and size of files in /var/log/proxmox-backup/?
 
You must log in or register to reply here.
Top Bottom