Tracking Center not in sync

[Optimiza Seguridad]

Hello, Can you post the steps to sync PMG tracking center across nodes please

 

[guidonline]

I made my own solution with earlier ideas. Since I manage 4 big PMG servers, syncing and using cat with SSH on the fly is not an option. So I create a script to cache the syslog and merge them for all servers and changed the original: /usr/bin/pmg-log-tracker to place this cache file to the live (/var/log/syslog) location, load: /usr/bin/pmg-log-tracker-default and set the original syslog back (so this will not get corrupt).

Steps (logged in as root):

1. Back-up original: pmg-log-tracker

mv /usr/bin/pmg-log-tracker /usr/bin/pmg-log-tracker-default

2. Create new: /usr/bin/pmg-log-tracker file with this code:

mv /var/log/syslog /var/log/syslog-tmp
mv /tmp/syslog /var/log/syslog

/usr/bin/pmg-log-tracker-default "$@"

mv /var/log/syslog /tmp/syslog
mv /var/log/syslog-tmp /var/log/syslog

3. Give it execution rights.

chmod +x /usr/bin/pmg-log-tracker

Now create a script to generate: /tmp/syslog

1. Create file: /root/merge-syslog.sh with this code:

cat /var/log/syslog  | grep -E 'postfix' > /tmp/syslog
ssh root@mx2 -o ConnectTimeout=2 -o ConnectionAttempts=1 'cat /var/log/syslog | grep -E postfix' >> /tmp/syslog
ssh root@mx3 -o ConnectTimeout=2 -o ConnectionAttempts=1 'cat /var/log/syslog | grep -E postfix' >> /tmp/syslog
ssh root@mx4-o ConnectTimeout=2 -o ConnectionAttempts=1 'cat /var/log/syslog | grep -E postfix' >> /tmp/syslog
cat /tmp/syslog | sort -M > /tmp/syslog-2
uniq /tmp/syslog-2 /tmp/syslog

2. Give it execution rights.

chmod +x /root/merge-syslog.sh

3. Create cronjob, (runtime depends on time to execute /root/merge-syslog.sh)

*/10 * * * * /root/merge-syslog.sh

Warning, your data in the tracker-log can have a delay of 10 minutes with this example. With a small amount of mail, you can run it every 5 minutes or less.

Script is used at one node only, in my case mx1.

 

[PVerberne]

Hmmm,

After reading the solutions posted here, I still do not like the idea of having to potentially break the system because it interferes with the way proxmox mail works right now. Is it too much to ask to implement a function in the web interface that points to a different log file? When using the command line, pmg-log-tracker can already do this. That would be a win-win as far as I'm concerned. All I would have to do is collect all needed combined logs on one node and let the system look at that. When that file does not exist, the default file (/var/log/syslog) could be parsed instead. Looks like a small change to me, but hey ... I'm just a user .

Best regards.

 

[Skata]

I made my own solution with earlier ideas. Since I manage 4 big PMG servers, syncing and using cat with SSH on the fly is not an option. So I create a script to cache the syslog and merge them for all servers and changed the original: /usr/bin/pmg-log-tracker to place this cache file to the live (/var/log/syslog) location, load: /usr/bin/pmg-log-tracker-default and set the original syslog back (so this will not get corrupt).

Steps (logged in as root):

1. Back-up original: pmg-log-tracker

mv /usr/bin/pmg-log-tracker /usr/bin/pmg-log-tracker-default

2. Create new: /usr/bin/pmg-log-tracker file with this code:

mv /var/log/syslog /var/log/syslog-tmp
mv /tmp/syslog /var/log/syslog

/usr/bin/pmg-log-tracker-default "$@"

mv /var/log/syslog /tmp/syslog
mv /var/log/syslog-tmp /var/log/syslog

3. Give it execution rights.

chmod +x /usr/bin/pmg-log-tracker

Now create a script to generate: /tmp/syslog

1. Create file: /root/merge-syslog.sh with this code:

cat /var/log/syslog  | grep -E 'postfix' > /tmp/syslog
ssh root@mx2 -o ConnectTimeout=2 -o ConnectionAttempts=1 'cat /var/log/syslog | grep -E postfix' >> /tmp/syslog
ssh root@mx3 -o ConnectTimeout=2 -o ConnectionAttempts=1 'cat /var/log/syslog | grep -E postfix' >> /tmp/syslog
ssh root@mx4-o ConnectTimeout=2 -o ConnectionAttempts=1 'cat /var/log/syslog | grep -E postfix' >> /tmp/syslog
cat /tmp/syslog | sort -M > /tmp/syslog-2
uniq /tmp/syslog-2 /tmp/syslog

2. Give it execution rights.

chmod +x /root/merge-syslog.sh

3. Create cronjob, (runtime depends on time to execute /root/merge-syslog.sh)

*/10 * * * * /root/merge-syslog.sh

Warning, your data in the tracker-log can have a delay of 10 minutes with this example. With a small amount of mail, you can run it every 5 minutes or less.

Script is used at one node only, in my case mx1.

Thank you, your solution made my life easier =) I have a cluster with 3 larger devices. More than 20,000 clean e-mails go through each device every day. Even with the amount, your script has no problems!

 

[RichBone]

Hello

I found a simpler solution that runs without CronJob.

I have modified the file /usr/bin/pmg-tracker as follows:
/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@mx2 /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'

Best regards
Martin Krüger

 

[zenaconsult]

I found a simpler solution that runs without CronJob.

This is just excellent, well done !! So, in my case, on the master server mx02 I did this:

mv /usr/bin/pmg-log-tracker /usr/bin/pmg-log-tracker-default

cat >/usr/bin/pmg-log-tracker <<'SCRIPT_END'
#!/bin/bash
/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@mx01 /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'
ssh root@mx03 /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'
SCRIPT_END

chmod +x /usr/bin/pmg-log-tracker

Works perfectly.

 

[Bhazhinho]

Awesome work guys, only thing to keep in mind if Proxmox updates the default binary, we may need to fix it manually again.

 

[hk@]

Hello and thank you all for the inspiration on how to sync tracking-logs.
is this still needed in PMGv8? (I assume yes, as I just ran into a similiar issue on a test-cluster)
is there a sane way to publish all logs to all cluster members so anyone can search anywhere for mail tracking?
thx.