syslog filling up with "apply old journal log rrd.journal-xxxxxxxx"

[bearded-donkey]

Hi,

Recently my syslog started filling up with:

2024-06-13T07:16:57.399492+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665b0f69
2024-06-13T07:16:57.400883+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665b1673

I just cleared out the syslog file which was 7GB. Following a reboot, and tail -f /var/log/syslog its filling up again with millions of entries.

Does anyone have any idea whats happening?

I searched but found no solutions on the form.

Thanks,

 

[Zerstoiber]

I've seen similar errors after changing timeserver/timezone.
RRD-Files are for the performance-graphs. If they get created with wrong timestamps, later updates to those files will be "confused".
I've deleted the rrd-files somewhere within /var/lib/rrdcached/ and the errors went away.

Is your timezone correct?
2024-06-13T07:16

 

[bearded-donkey]

Thanks for coming back.

My server datetime is spot on.

Thu Jun 13 09:43:38 AM NZST 2024


As for that directory, I don't have it in /var/lib:

     root@proxmox-backup-server  /var/lib                                                                                     10ms  09:44
❯ l
apt
chrony
dbus
dhcp
dictionaries-common
dpkg
git
grub
ifupdown2
logrotate
man-db
misc
nfs
pam
postfix
private
proxmox-backup
python
samba
shells.state
smartmontools
systemd
ucf
vim

I do see a directory that may be related here, containing a bunch of files (only showing the first 15, but there are 7873 files in total):

/var/lib/proxmox-backup/rrdb

     root@proxmox-backup-server  /var/lib/proxmox-backup/rrdb                                                                 39ms  09:46
❯ ll | head -15
total 1.8G
drwxr-xr-x 4 backup backup 304K Jun  4 08:33 ./
drwxr-xr-x 6 backup backup 4.0K Jun 12 03:11 ../
drwxr-xr-x 4 backup backup 4.0K Nov 20  2023 datastore/
drwxr-xr-x 2 backup backup 4.0K Jun  4 08:33 host/
-rw-r--r-- 1 backup backup  66M Jun 13 09:46 rrd.journal
-rw-r--r-- 1 backup backup 220K Dec 21 04:17 rrd.journal-6583056d
-rw-r--r-- 1 backup backup 4.0K Dec 21 04:47 rrd.journal-65830c75
-rw-r--r-- 1 backup backup    0 Dec 21 05:17 rrd.journal-6583137d
-rw-r--r-- 1 backup backup    0 Dec 21 05:47 rrd.journal-65831a85
-rw-r--r-- 1 backup backup    0 Dec 21 06:17 rrd.journal-6583218e
-rw-r--r-- 1 backup backup    0 Dec 21 06:47 rrd.journal-65832896
-rw-r--r-- 1 backup backup    0 Dec 21 07:17 rrd.journal-65832f9e
-rw-r--r-- 1 backup backup    0 Dec 21 07:47 rrd.journal-658336a6
-rw-r--r-- 1 backup backup    0 Dec 21 08:17 rrd.journal-65833dae

Should I try to clear directory this out?


Thanks,


EDIT:

Looks like the system is looping over these files again and again. I searched for one of the filenames in the syslog, and there are MANY MANY entries of the same filename at different times, e.g (filtered on a single filename):

This is just a small subset:

2024-06-13T09:35:49.706725+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:36:09.833090+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:36:29.703668+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:36:50.111180+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:37:09.466452+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:37:29.747533+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:37:51.177155+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:38:09.933683+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:38:30.020049+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:38:49.735102+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:39:10.705024+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:39:29.521582+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:39:51.057870+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:40:09.890731+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:40:30.193931+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:40:49.588658+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:41:10.200397+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:41:30.657436+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:41:50.429958+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:42:10.255652+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:42:30.721508+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:42:49.981384+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:43:10.058419+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:43:30.717337+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:43:50.696472+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:44:12.494603+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:44:29.874202+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:44:51.199490+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:45:10.597731+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:45:30.773753+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:45:51.301464+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:46:10.579469+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:46:31.381535+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8
2024-06-13T09:46:50.545526+12:00 proxmox-backup-server proxmox-backup-proxy[606]: apply old journal log rrd.journal-665d31e8

 

[dcsapak]

hi,

can you post the end of the log where the 'apply old journal' lines stop? my guess is there should be an error somewhere which aborts it and does not clean up the journal files

 

[bearded-donkey]

hi,

can you post the end of the log where the 'apply old journal' lines stop? my guess is there should be an error somewhere which aborts it and does not clean up the journal files

Thanks for getting back to me.

I had to solve the issue and couldn't deal with the growing logs all the time, so I deleted all the rrd related files and cleaned up the journal files. The problem has gone away, but I had to be happy with losing the graphs.

From what I remember I did not see any errors in those logs, it was just iterating over those journal files over and over.

If this ever happens again, I will take a look and post something again.


Thanks,