[SOLVED] Logs problem after upgrade from 5 to 6

[patefoniq]

Hi,

I just upgraded my two instances of pmg from version 5 to 6 without any problems. Unfortunately after that I have problems with logs:

From GUI:

command '/usr/bin/pmg-log-tracker -v -l 2000 -s 1567136160 -e 1567202400 -n -g' failed: exit code 255 (500)

From shell:

root@antyspam:~# /usr/bin/pmg-log-tracker -v -l 2000 -s 1567137300 -e 1567202400 -n -g
unable to access log files

Logs permissions:

drwxr-xr-x 10 root     root         4096 Aug 29 00:00 ./
drwxr-xr-x 11 root     root         4096 Oct  5  2018 ../
drwxr-xr-x  2 root     root         4096 Aug 29 08:34 apt/
drwxr-xr-x  2 clamav   clamav       4096 Aug 26 06:25 clamav/
drwxr-xr-x  2 www-data www-data     4096 Mar  1 10:13 pmgproxy/
drwxrwxr-t  2 root     postgres     4096 Aug 27 17:07 postgresql/
drwx------  2 root     root         4096 Aug 27 16:51 private/
drwxr-xr-x  3 root     root         4096 Mar  1 10:10 pve/
drwxr-x---  2 root     adm          4096 Mar  5  2018 samba/
drwxr-xr-x  2 root     root         4096 Apr  6 09:18 sysstat/
-rw-r--r--  1 root     root        20109 Aug 27 17:07 alternatives.log
-rw-r--r--  1 root     root         3114 Aug  9 09:45 alternatives.log.1
-rw-r--r--  1 root     root          485 Jun 24 11:11 alternatives.log.2.gz
-rw-r--r--  1 root     root          507 May 10 09:39 alternatives.log.3.gz
-rw-r--r--  1 root     root          507 Apr 30 09:02 alternatives.log.4.gz
-rw-r--r--  1 root     root         2525 Mar  3 12:28 alternatives.log.5.gz
-rw-r-----  1 root     adm        101214 Aug 30 07:17 auth.log
-rw-r-----  1 root     adm         39551 Aug 26 06:25 auth.log.1
-rw-r-----  1 root     adm          1729 Aug 18 06:25 auth.log.2.gz
-rw-r-----  1 root     adm          2289 Aug 12 06:25 auth.log.3.gz
-rw-r-----  1 root     adm          1734 Aug  4 06:25 auth.log.4.gz
-rw-rw----  1 root     utmp            0 Aug  1 06:25 btmp
-rw-------  1 root     utmp            0 Jul  1 06:25 btmp.1
-rw-r-----  1 root     adm        456762 Aug 30 07:15 daemon.log
-rw-r-----  1 root     adm        580877 Aug 26 06:25 daemon.log.1
-rw-r-----  1 root     adm         27008 Aug 18 06:25 daemon.log.2.gz
-rw-r-----  1 root     adm         35899 Aug 12 06:25 daemon.log.3.gz
-rw-r-----  1 root     adm         27594 Aug  4 06:25 daemon.log.4.gz
-rw-r-----  1 root     adm        134375 Aug 30 04:32 debug
-rw-r-----  1 root     adm        121864 Aug 26 06:01 debug.1
-rw-r-----  1 root     adm          4172 Aug 18 05:58 debug.2.gz
-rw-r-----  1 root     adm          5477 Aug 12 05:58 debug.3.gz
-rw-r-----  1 root     adm          4402 Aug  4 05:58 debug.4.gz
-rw-r--r--  1 root     root       366862 Aug 29 08:35 dpkg.log
-rw-r--r--  1 root     root        19468 Jul 29 10:25 dpkg.log.1
-rw-r--r--  1 root     root         1531 Jun 24 11:11 dpkg.log.2.gz
-rw-r--r--  1 root     root         1915 May 17 08:31 dpkg.log.3.gz
-rw-r--r--  1 root     root         1941 Apr 30 09:02 dpkg.log.4.gz
-rw-r--r--  1 root     root        35674 Apr  1 06:17 dpkg.log.5.gz
-rw-r--r--  1 root     root         3584 Aug 27 17:00 faillog
-rw-r--r--  1 root     root         2130 Aug 27 17:00 fontconfig.log
-rw-r-----  1 root     adm             0 Aug 29 00:00 kern.log
-rw-r-----  1 root     adm        369868 Aug 27 17:18 kern.log.1
-rw-r-----  1 root     adm         18775 Aug 21 11:01 kern.log.2.gz
-rw-r-----  1 root     adm         37114 Jul 16 08:58 kern.log.3.gz
-rw-r-----  1 root     adm         18780 Jun 24 11:12 kern.log.4.gz
-rw-rw-r--  1 root     utmp        32704 Aug 30 07:08 lastlog
-rw-r-----  1 root     adm         53577 Aug 30 07:05 mail.err
-rw-r-----  1 root     adm        106609 Aug 26 06:14 mail.err.1
-rw-r-----  1 root     adm          7415 Aug 18 06:17 mail.err.2.gz
-rw-r-----  1 root     adm         10773 Aug 12 06:16 mail.err.3.gz
-rw-r-----  1 root     adm          8236 Aug  4 06:16 mail.err.4.gz
-rw-r-----  1 root     adm      13954578 Aug 30 07:23 mail.info
-rw-r-----  1 root     adm      32138880 Aug 26 06:23 mail.info.1
-rw-r-----  1 root     adm       2085959 Aug 18 06:24 mail.info.2.gz
-rw-r-----  1 root     adm       3123068 Aug 12 06:24 mail.info.3.gz
-rw-r-----  1 root     adm       2387370 Aug  4 06:25 mail.info.4.gz
-rw-r-----  1 root     adm      13954578 Aug 30 07:23 mail.log
-rw-r-----  1 root     adm      32138880 Aug 26 06:23 mail.log.1
-rw-r-----  1 root     adm       2085959 Aug 18 06:24 mail.log.2.gz
-rw-r-----  1 root     adm       3123068 Aug 12 06:24 mail.log.3.gz
-rw-r-----  1 root     adm       2387370 Aug  4 06:25 mail.log.4.gz
-rw-r-----  1 root     adm        132593 Aug 30 07:05 mail.warn
-rw-r-----  1 root     adm        274369 Aug 26 06:14 mail.warn.1
-rw-r-----  1 root     adm         14429 Aug 18 06:17 mail.warn.2.gz
-rw-r-----  1 root     adm         20877 Aug 12 06:16 mail.warn.3.gz
-rw-r-----  1 root     adm         16059 Aug  4 06:24 mail.warn.4.gz
-rw-r-----  1 root     adm        458130 Aug 30 07:15 messages
-rw-r-----  1 root     adm        508709 Aug 26 06:25 messages.1
-rw-r-----  1 root     adm         17028 Aug 18 06:25 messages.2.gz
-rw-r-----  1 root     adm         23075 Aug 12 06:25 messages.3.gz
-rw-r-----  1 root     adm         17509 Aug  4 06:25 messages.4.gz
-rw-r-----  1 root     adm             0 Aug 28 00:00 syslog
-rw-r-----  1 root     adm      11804313 Aug 30 07:23 syslog.1
-rw-r-----  1 root     adm        300053 Aug 19 06:25 syslog.10.gz
-rw-r-----  1 root     adm        261784 Aug 18 06:25 syslog.11.gz
-rw-r-----  1 root     adm        347503 Aug 17 06:25 syslog.12.gz
-rw-r-----  1 root     adm        265111 Aug 16 06:25 syslog.13.gz
-rw-r-----  1 root     adm        464585 Aug 15 06:25 syslog.14.gz
-rw-r-----  1 root     adm        459756 Aug 14 06:25 syslog.15.gz
-rw-r-----  1 root     adm        452193 Aug 13 06:25 syslog.16.gz
-rw-r-----  1 root     adm        272725 Aug 12 06:25 syslog.17.gz
-rw-r-----  1 root     adm        326776 Aug 11 06:25 syslog.18.gz
-rw-r-----  1 root     adm        440571 Aug 10 06:25 syslog.19.gz
-rw-r-----  1 root     adm        446698 Aug  9 06:25 syslog.20.gz
-rw-r-----  1 root     adm        516339 Aug  8 06:25 syslog.21.gz
-rw-r-----  1 root     adm        429763 Aug  7 06:25 syslog.22.gz
-rw-r-----  1 root     adm        656071 Aug  6 06:25 syslog.23.gz
-rw-r-----  1 root     adm        254858 Aug  5 06:25 syslog.24.gz
-rw-r-----  1 root     adm        266090 Aug  4 06:25 syslog.25.gz
-rw-r-----  1 root     adm        418801 Aug  3 06:25 syslog.26.gz
-rw-r-----  1 root     adm        431173 Aug  2 06:25 syslog.27.gz
-rw-r-----  1 root     adm        437525 Aug  1 06:25 syslog.28.gz
-rw-r-----  1 root     adm        465140 Jul 31 06:25 syslog.29.gz
-rw-r-----  1 root     adm        418744 Aug 27 06:25 syslog.2.gz
-rw-r-----  1 root     adm        538143 Jul 30 06:25 syslog.30.gz
-rw-r-----  1 root     adm        245738 Aug 26 06:25 syslog.3.gz
-rw-r-----  1 root     adm        256747 Aug 25 06:25 syslog.4.gz
-rw-r-----  1 root     adm        396627 Aug 24 06:25 syslog.5.gz
-rw-r-----  1 root     adm        479805 Aug 23 06:25 syslog.6.gz
-rw-r-----  1 root     adm        858233 Aug 22 06:25 syslog.7.gz
-rw-r-----  1 root     adm        841577 Aug 21 06:25 syslog.8.gz
-rw-r-----  1 root     adm        776883 Aug 20 06:25 syslog.9.gz
-rw-r-----  1 root     adm             0 Aug 29 00:00 user.log
-rw-r-----  1 root     adm          5570 Aug 27 17:18 user.log.1
-rw-r-----  1 root     adm            91 Aug 21 11:01 user.log.2.gz
-rw-r-----  1 root     adm           105 Jul 16 08:58 user.log.3.gz
-rw-r-----  1 root     adm            91 Jun 24 11:12 user.log.4.gz
-rw-------  1 root     root        37651 Aug 27 17:19 vmware-vmsvc.log
-rw-rw-r--  1 root     utmp        14976 Aug 30 07:08 wtmp
-rw-rw-r--  1 root     utmp         5760 Jul 16 09:36 wtmp.1

Like always, yesterday it worked like a charm ;-)

 

[t.lamprecht]

The log-tracker normally runs as root, as you also did in your manual test, so permissions really should not be an issue here.

Single thing minimally strange is that the size of "syslog" is 0, I mean it could be just the exact moment after a log rotation, but even then mine never shows "0". Can you open it in the shell?

cat /var/log/syslog

 

[patefoniq]

I just rebooted the system and here is the output of ls syslog files:

-rw-r-----  1 root     adm      526K Jul 30 06:25 syslog.30.gz
-rw-r-----  1 root     adm      455K Jul 31 06:25 syslog.29.gz
-rw-r-----  1 root     adm      428K Aug  1 06:25 syslog.28.gz
-rw-r-----  1 root     adm      422K Aug  2 06:25 syslog.27.gz
-rw-r-----  1 root     adm      409K Aug  3 06:25 syslog.26.gz
-rw-r-----  1 root     adm      260K Aug  4 06:25 syslog.25.gz
-rw-r-----  1 root     adm      249K Aug  5 06:25 syslog.24.gz
-rw-r-----  1 root     adm      641K Aug  6 06:25 syslog.23.gz
-rw-r-----  1 root     adm      420K Aug  7 06:25 syslog.22.gz
-rw-r-----  1 root     adm      505K Aug  8 06:25 syslog.21.gz
-rw-r-----  1 root     adm      437K Aug  9 06:25 syslog.20.gz
-rw-r-----  1 root     adm      431K Aug 10 06:25 syslog.19.gz
-rw-r-----  1 root     adm      320K Aug 11 06:25 syslog.18.gz
-rw-r-----  1 root     adm      267K Aug 12 06:25 syslog.17.gz
-rw-r-----  1 root     adm      442K Aug 13 06:25 syslog.16.gz
-rw-r-----  1 root     adm      449K Aug 14 06:25 syslog.15.gz
-rw-r-----  1 root     adm      454K Aug 15 06:25 syslog.14.gz
-rw-r-----  1 root     adm      259K Aug 16 06:25 syslog.13.gz
-rw-r-----  1 root     adm      340K Aug 17 06:25 syslog.12.gz
-rw-r-----  1 root     adm      256K Aug 18 06:25 syslog.11.gz
-rw-r-----  1 root     adm      294K Aug 19 06:25 syslog.10.gz
-rw-r-----  1 root     adm      759K Aug 20 06:25 syslog.9.gz
-rw-r-----  1 root     adm      822K Aug 21 06:25 syslog.8.gz
-rw-r-----  1 root     adm      839K Aug 22 06:25 syslog.7.gz
-rw-r-----  1 root     adm      469K Aug 23 06:25 syslog.6.gz
-rw-r-----  1 root     adm      388K Aug 24 06:25 syslog.5.gz
-rw-r-----  1 root     adm      251K Aug 25 06:25 syslog.4.gz
-rw-r-----  1 root     adm      240K Aug 26 06:25 syslog.3.gz
-rw-r-----  1 root     adm      409K Aug 27 06:25 syslog.2.gz
-rw-r-----  1 root     adm       12M Aug 30 09:39 syslog.1
-rw-r-----  1 root     adm      201K Aug 30 09:44 syslog

Everything seems to be fine now:



It is nice to have such helpful community

 

[wimp9849]

Hi, i have the same problem after upgrade, i've tried rebooting yesterday and it solved the issue, but today i have the same error.

 

[patefoniq]

I agree reboot solves the problem only temporarily. Every day I get the same message in "tracking center":

command '/usr/bin/pmg-log-tracker -v -l 2000 -s 1568292780 -e 1568325600 -n -g' failed: exit code 255 (500)

And of course, file "syslog" has a 0 size. Size is back to normal only after a full OS restart.

 

[wimp9849]

So, i think we need to find if the problem is pmg or the new version of debian.

i tried adding manually a line to syslog with command: #logger test message but it is still empty, so i think the problem is on Debian / Rsyslog side.

I've reinstalled rsyslog with this command: #sudo apt-get install --reinstall rsyslog. Now is working, let's see if it stop again tomorrow

 

[Stoiko Ivanov]

* hmm - is rsyslog running (and installed properly)?:
** `ps auxwf |grep rsyslog`
** `systemctl status -l rsyslog`
** `dpkg -l |grep rsyslog`
* do you see the any mention of logrotate in your journal?

I hope this helps

 

[patefoniq]

**

root@antyspam:~# ps auxwf |grep rsyslog
623  0.0  0.0 225820  4692 ?        Ssl  Sep10   0:13 /usr/sbin/rsyslogd -n -iNONE

========================================

● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-09-10 15:10:07 CEST; 2 days ago
     Docs: man:rsyslogd(8)
           https://www.rsyslog.com/doc/
 Main PID: 623 (rsyslogd)
    Tasks: 4 (limit: 4915)
   Memory: 41.1M
   CGroup: /system.slice/rsyslog.service
           └─623 /usr/sbin/rsyslogd -n -iNONE

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

========================================

root@antyspam:~# dpkg -l |grep rsyslog
ii  rsyslog                        8.1901.0-1                   amd64        reliable system and kernel logging daemon

* do you see the any mention of logrotate in your journal?

NO

 

[towerman]

* hmm - is rsyslog running (and installed properly)?:
** `ps auxwf |grep rsyslog`
** `systemctl status -l rsyslog`
** `dpkg -l |grep rsyslog`
* do you see the any mention of logrotate in your journal?

I hope this helps

Neither do I

 

[Stoiko Ivanov]

this is odd ..
please post the output of:
* systemctl list-timers
* systemctl status -l logrotate
* cat /etc/logrotate.d/rsyslog
* logrotate -dv /etc/logrotate.conf

Thanks!

 

[patefoniq]

 

[Stoiko Ivanov]

* does the log remain emtpy after your restart `rsyslog`?
* what's the latest timestamp in /var/log/syslog.1?
* do you have current entries in your journal? (`journalctl -r`)

 

[patefoniq]

* After restart the "rsyslog" service, syslog file back to normal size and gets the full content.
* Sep 13 10:47:59 antyspam postfix/postscreen[7148]: DISCONNECT [63.83.73.232]:51776
* yes I have

After restarting the rsyslog service my "Tracking Center" backed to function normally.

 

[Stoiko Ivanov]

ok - seems logrotate runs, but fails to restart rsyslog:
please post:
* '/etc/logrotate.conf'
* '/etc/logrotate.d/rsyslog'

 

[patefoniq]

* '/etc/logrotate.conf'

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

* '/etc/logrotate.d/rsyslog'

/var/log/syslog
{
        rotate 30
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

 

[towerman]

* does the log remain emtpy after your restart `rsyslog`?
* what's the latest timestamp in /var/log/syslog.1?
* do you have current entries in your journal? (`journalctl -r`)

That's what I see

ok - seems logrotate runs, but fails to restart rsyslog:
please post:
* '/etc/logrotate.conf'
* '/etc/logrotate.d/rsyslog'

logrotate.conf

 see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

rsyslog

  GNU nano 3.2                /etc/logrotate.d/rsyslog

/var/log/syslog
{
        rotate 31
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

 

[Stoiko Ivanov]

invoke-rc.d rsyslog rotate > /dev/null

guess this is the problem

`invoke-rc.d` is a legacy program from sysvinit.
With buster this got dropped.
The upgrade probably asked that the file changed ...
you should have a file in /etc/logrotate.d/rsyslog.dpkg-dist (or similar suffix) - compare both
the line should be:

/usr/lib/rsyslog/rsyslog-rotate

instead of

 invoke-rc.d rsyslog rotate > /dev/null

hope this helps!

 

[patefoniq]

I've changed this option to the right one and restarted the server. We will see if this helps but I have the feeling it will be working fine now. When I started this discussion I felt somewhere inside that this instruction was wrong, but for being sure I preferred to ask here.

 

[towerman]

@Stoiko Ivanov, could you help me too please?

 

[Stoiko Ivanov]

@towerman - as far as i could see it's the very same issue (invoke-rc.d rsyslog rotate instead of /usr/lib/rsyslog/rsyslog-rotate).
or did i miss something?