How to stop syncjob flooting syslog

M

mike2012

Renowned Member
Apr 29, 2015
42
1
73
Is there a posibility to stop the flooting of syslog when doing a syncjob with following lines
.
.
Dec 02 08:37:47 .... proxmox-backup-proxy[5327]: GET /chunk
Dec 02 08:37:47 .... proxmox-backup-proxy[5327]: download chunk "/mnt/datastore/......."
.
.

Package versions :
proxmox-backup: 2.3-1 (running kernel: 5.15.74-1-pve)
proxmox-backup-server: 2.3.1-1 (running version: 2.3.1)
pve-kernel-5.15: 7.2-14 pve-kernel-helper: 7.2-14
pve-kernel-5.13: 7.1-9 pve-kernel-5.11: 7.0-10
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.60-2-pve: 5.15.60-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ifupdown2: 3.1.0-1+pmx3 libjs-extjs: 7.0.0-1
proxmox-backup-docs: 2.3.1-1
proxmox-backup-client: 2.3.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-xtermjs: 4.16.0-1
smartmontools: 7.2-pve3 zfsutils-linux: 2.1.6-pve1
 
  • Like
Reactions: logics
To be precise: The sync job on host A (which pulls data from remote host B) does cause lots of proxmox-backup-proxy logging on host B, not on host A.

I have the same problem, because several servers sync data from a host about every 5-10 minutes, so on the host that data is being pulled from I get over 10 millions of lines weekly in each file /var/log/syslog, /var/log/messages, /var/log/user.log like this:

Code: 
Mar 27 07:12:02 myhost proxmox-backup-proxy[372948]: GET /chunk
Mar 27 07:12:02 myhost proxmox-backup-proxy[372948]: download chunk "/mnt/xxx/yyy/.chunks/4812/4812f61562bc33191f2b67fc4241ee7bdaace3bf6c80e908a609400d5e545419"

So this is the size of my logs:

Code: 
myhost:/var/log# treesize
8.5 GB .
3.1 GB ./journal
2.8 GB ./proxmox-backup
6.6 MB ./pve
389.0 KB ./pveproxy
165.0 KB ./apt
14.0 KB ./vzdump
...

with /bin/treesize being a small script

Bash: 
#/bin/sh
du -k --max-depth=1 --exclude=./{mnt,proc,sys,dev} | sort -nr | awk '
     BEGIN {
        split("KB,MB,GB,TB", Units, ",");
     }
     {
        u = 1;
        while ($1 >= 1024) {
           $1 = $1 / 1024;
           u += 1
        }
        $1 = sprintf("%.1f %s", $1, Units[u]);
        print $0;
     }
    '


In detail:

Code: 
myhost:/var/log# ls -l --sort=size
total 2723026
-rw-r----- 1 root     adm             1464480086 Apr  2 00:00 syslog.1
-rw-r----- 1 root     adm             1461321988 Apr  2 00:00 messages.1
-rw-r----- 1 root     adm             1453339684 Apr  2 00:00 user.log.1
-rw-r----- 1 root     adm              199448555 Apr  3 09:33 syslog
-rw-r----- 1 root     adm              190345787 Mar 12 00:00 syslog.4.gz
-rw-r----- 1 root     adm              189312770 Mar 12 00:00 messages.4.gz
-rw-r----- 1 root     adm              187180469 Mar 12 00:00 user.log.4.gz
-rw-r----- 1 root     adm              176000160 Mar 19 00:00 syslog.3.gz
-rw-r----- 1 root     adm              174973873 Mar 19 00:00 messages.3.gz
-rw-r----- 1 root     adm              173039564 Mar 26 00:00 syslog.2.gz
-rw-r----- 1 root     adm              172901808 Mar 19 00:00 user.log.3.gz
-rw-r----- 1 root     adm              172005297 Mar 26 00:00 messages.2.gz
-rw-r----- 1 root     adm              169877585 Mar 26 00:00 user.log.2.gz
-rw-r----- 1 root     adm              134262887 Apr  3 09:32 daemon.log
-rw-r----- 1 root     adm               65062269 Apr  3 09:33 messages
-rw-r----- 1 root     adm               63347347 Apr  2 17:44 user.log
-rw-r----- 1 root     adm               31611850 Aug 23  2022 syslog.6.gz
-rw-rw-r-- 1 root     utmp              18701432 Apr  3 09:32 lastlog
-rw-r----- 1 root     adm                9060203 Aug 22  2022 syslog.7.gz
-rw-r----- 1 root     adm                7984111 Apr  2 00:00 kern.log.1
-rw-r----- 1 root     adm                7983461 Apr  2 00:00 ufw.log.1
-rw-r----- 1 root     adm                3521684 Apr  2 00:00 auth.log.1
-rw-r----- 1 root     adm                2579205 Apr  2 00:00 daemon.log.1
-rw-r--r-- 1 root     root               2049472 Aug 28  2022 faillog
-rw-r----- 1 root     adm                1967061 Mar 19 00:00 ufw.log.3.gz
-rw-r----- 1 root     adm                1718463 Apr  3 09:33 kern.log
-rw-r----- 1 root     adm                1600728 Apr  3 09:33 ufw.log
-rw-r----- 1 root     adm                1001535 Mar  5 00:00 ufw.log.4.gz
-rw-r----- 1 root     adm                 992632 Mar 12 00:00 kern.log.4.gz
-rw-r----- 1 root     adm                 975374 Mar 19 00:00 kern.log.3.gz
-rw-r----- 1 root     adm                 963214 Mar 26 00:00 kern.log.2.gz
-rw-r----- 1 root     adm                 963214 Mar 26 00:00 ufw.log.2.gz
-rw-r----- 1 root     adm                 709293 Apr  3 09:33 auth.log
-rw-r----- 1 root     adm                 215852 Mar 26 00:00 daemon.log.2.gz
-rw-r----- 1 root     adm                 211768 Mar 19 00:00 daemon.log.3.gz
-rw-r----- 1 root     adm                 210477 Mar 12 00:00 daemon.log.4.gz
-rw-r----- 1 root     adm                 150873 Mar 26 00:00 auth.log.2.gz
-rw-r----- 1 root     adm                 150101 Mar 19 00:00 auth.log.3.gz
-rw-r----- 1 root     adm                 148793 Mar 12 00:00 auth.log.4.gz

myhost:/var/log# grep -c ".*" syslog.1
11714765
myhost:/var/log# grep -c "GET /chunk\|download chunk" syslog.1
11449872

myhost:/var/log# grep -c ".*" messages.1
11688043
myhost:/var/log# grep -c "GET /chunk\|download chunk" messages.1
11449872

myhost:/var/log# grep -c ".*" user.log.1
11657979
myhost:/var/log# grep -c "GET /chunk\|download chunk" user.log.1
11449872

(By the way: when checking out these big files in Windows I recomment LogExpert, because Notepad++ can't handle such big files at all.)

So as you can see /var/log/syslog, /var/log/messages and /var/log/user.log can each grow to over 1 GB in size for just 1 week of logging. Each contain the same GET /chunk and download chunk messages - and those lines make up 98% of their content.

I am not using nested Datastores, and I am not concerned about the size of /var/log/proxmox-backup as mentioned in https://forum.proxmox.com/threads/proxmox-backup-logs.98220/, but I care about syslog and the other 2 files mentioned, because I want to keep them small for being easily searchable.

Can anyone help me out here?
 
I guess the BackupReader used by sync should not set the debug flag ;) could you file an issue at bugzilla.proxmox.com ? thanks!
 
  • Like
Reactions: logics
yes, @ggoller sent a few patches - the scope got a bit bigger as he started reworking the whole server side logging infrastructure..
 
yes, the plan is to improve the control over what gets logged where/when, and expose the knobs to allow opt-in debug logs.
 
  • Like
Reactions: ggoller, VictorSTS and t0mz
Hello,

any news on this?

Seems to hit me too, running logcheck, now after upgrade to pve 8

(proxmox-backup-server:amd64/bookworm 3.1.2-1 uptodate)
 
Looks like it didn't make it into PBS v3.2 either?

I need to do something about this and am considering setting up either a dedicated server/VM/LXC and have PBS access the backup drives over the network or in case performance doesn't work out stop using PBS for the time being.
 
You must log in or register to reply here.
Top Bottom