Tracking Center not in sync

[Thomas Hukkelberg]

A couple weeks ago I sat up a PMG cluster with 3 nodes. It works fine now, after having some initial problems syncing the database to the third node. I solved that by logging in to postgresql with psql and deleted entries from the cstatic-table. (The log complained about duplicate entries.) Then the sync between the nodes started working and errors in the logs went away. (To get into postgresql, I had to edit pg_hba.conf to trust localhost logons without password and reload postgresql)

Problem now is that the Tracking Center and Queue is not in sync across the nodes. Are they supposed to be in sync? Can't find any information about that in the manual or here in the forums. All nodes can ssh to each other and there is no errors in the logs. Also the all settings and Quarantine is synced, so apparently everything is working -- except Tracking Center and Queues...

I wonder where Tracking Center is getting it's info from? I guess it's a combination of database and log files. I believe the database contains all entries from all nodes, but the syslog only show local logs. Should the syslog be synced between the nodes with rsyslog? Can anyone enlighten me?

 

[dietmar]

Problem now is that the Tracking Center and Queue is not in sync across the nodes. Are they supposed to be in sync?

No. The tracking center search local files /var/log/syslog*, so that is not supposed to be in sync.

 

[Thomas Hukkelberg]

OK, I see. Will it be possible to set up an additional node for proxmox that receive syslogs vi arsyslog from all nodes. Will the resulting syslog be searchable by pmg-log-tracker?

 

[dietmar]

OK, I see. Will it be possible to set up an additional node for proxmox that receive syslogs vi arsyslog from all nodes. Will the resulting syslog be searchable by pmg-log-tracker?

No, that is currently not possible.

 

[Thomas Hukkelberg]

Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?

 

[Dragnell]

Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?

That look useful, i was until now believing that the tracking center was in sync across nodes, did you leave a how to behind ? Does still works for you ?

 

[Sommer]

Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?

Thomas,
Can you provide the steps to do tracking sync between nodes?
Thanks

 

[Sommer]

Well, I made my own solution.
The main ideia was to keep it simple, the less changes, the better.
I have 2 servers, 1 master and 1 node.

Create a script on both servers. First, on master server:

ssh root@serverNODE cat /var/log/syslog >> /var/log/syslog.1
cat /var/log/syslog.1 | sort -M > /var/log/tmp1
uniq /var/log/tmp1 /var/log/syslog.1

Second, on node server:

ssh root@serverMASTER cat /var/log/syslog >> /var/log/syslog.1
cat /var/log/syslog.1 | sort -M > /var/log/tmp1
uniq /var/log/tmp1 /var/log/syslog.1

Finally, include the line at /etc/crontab on both servers:

* * * * * root /path/script.sh

Now, I can use the tracking center on Master server and get results from Node server also.

 

[Sommer]

I woke up with a better solution.
Instead of copying logs from one server to another, messing up the files, I created my "own" pmg-log-tracker.

First you need to rename /usr/bin/pmg-log-tracker to something else, example pmg-log-tracker-default, on server Master and Node.

Then create /usr/bin/pmg-log-tracker script

/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@serverNODE /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'

"$@" is to pass all arguments to script

Spoiler: arguments

-v -l 2000 -s 1531882800 -e 1531969200 -n -g -f gmail.com

head -n -1 is to remove last line of first search results

Spoiler: last line

STATUS: aborted by limit (too many hits)

sed '/^#/ d' is to remove the header parameters of second search results

Spoiler: header parameters

# LogReader: 19220
# Query options
# Start: 2018-07-18 00:00:00 (1531872000)
# END: 2018-07-18 16:20:45 (1531930845)
# End Query Options

These filters are needed because PERL script that processes results failed with that unexpected characters.

 

[kyles2013]

I woke up with a better solution.
Instead of copying logs from one server to another, messing up the files, I created my "own" pmg-log-tracker.

First you need to rename /usr/bin/pmg-log-tracker to something else, example pmg-log-tracker-default, on server Master and Node.

Then create /usr/bin/pmg-log-tracker script

/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@serverNODE /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'

"$@" is to pass all arguments to script

Spoiler: arguments

-v -l 2000 -s 1531882800 -e 1531969200 -n -g -f gmail.com

head -n -1 is to remove last line of first search results

Spoiler: last line

STATUS: aborted by limit (too many hits)

sed '/^#/ d' is to remove the header parameters of second search results

Spoiler: header parameters

# LogReader: 19220
# Query options
# Start: 2018-07-18 00:00:00 (1531872000)
# END: 2018-07-18 16:20:45 (1531930845)
# End Query Options

These filters are needed because PERL script that processes results failed with that unexpected characters.

Hey Sommer,
I have gone ahead and tried to implement your script you have referenced above however I am running into an issue where the tracking center only shows the messages from the other node now instead of both. Any advice would be greatly appreciated.

 

[Sommer]

Hey Sommer,
I have gone ahead and tried to implement your script you have referenced above however I am running into an issue where the tracking center only shows the messages from the other node now instead of both. Any advice would be greatly appreciated.

Make sure your script has the two lines, first runs local, second runs remote

 

[kyles2013]

Yes that is correct I have uploaded my current pmg-log-tracker. The only thing I changed was the IP address for the serverip and added the TXT file extension so I could upload the file.
When I run this script it appears the second line overwrites the first.

 

[Sommer]

Yes that is correct I have uploaded my current pmg-log-tracker. The only thing I changed was the IP address for the serverip and added the TXT file extension so I could upload the file.
When I run this script it appears the second line overwrites the first.

Check original pmg script name, path, permissions.
Give me ls -l pmg* at /usr/bin

 

[kyles2013]

Check original pmg script name, path, permissions.
Give me ls -l pmg* at /usr/bin

Permissions are all set to 0755 with my file stored at /usr/bin/

 

[Sommer]

Permissions are all set to 0755 with my file stored at /usr/bin/

That is odd!
Try running pmg-log-tracker-default -l 1 and pmg-log-tracker -l 1 to see if you get any errors. Maybe you need to remove head and sed to see what is happening

 

[kyles2013]

That is odd!
Try running pmg-log-tracker-default -l 1 and pmg-log-tracker -l 1 to see if you get any errors. Maybe you need to remove head and sed to see what is happening

So when I run pmg-log-tracker -l 1 I am getting an error stating head: invalid number of lines: ‘1\r’

 

[Sommer]

So when I run pmg-log-tracker -l 1 I am getting an error stating head: invalid number of lines: ‘1\r’

and pmg-log-tracker-default -l 1?

 

[kyles2013]

and pmg-log-tracker-default -l 1?

It comes back with the query options and then the first email. Then it goes to STATUS: aborted by limit (too many hits)

 

[Sommer]

and pmg-log-tracker-default -l 1?

It seems your script have some strange character after head -n -1 \r

 

[kyles2013]

It seems your script have some strange character after head -n -1 \r

\r is a line return I believe? I have gone ahead and deleted everything from the script and manually typed it out and it still references the same error.
head: invalid number of lines: ‘1\r’