Tracking Center not in sync

Thomas Hukkelberg

Active Member
Mar 4, 2018
18
6
43
Oslo
A couple weeks ago I sat up a PMG cluster with 3 nodes. It works fine now, after having some initial problems syncing the database to the third node. I solved that by logging in to postgresql with psql and deleted entries from the cstatic-table. (The log complained about duplicate entries.) Then the sync between the nodes started working and errors in the logs went away. (To get into postgresql, I had to edit pg_hba.conf to trust localhost logons without password and reload postgresql)

Problem now is that the Tracking Center and Queue is not in sync across the nodes. Are they supposed to be in sync? Can't find any information about that in the manual or here in the forums. All nodes can ssh to each other and there is no errors in the logs. Also the all settings and Quarantine is synced, so apparently everything is working -- except Tracking Center and Queues...

I wonder where Tracking Center is getting it's info from? I guess it's a combination of database and log files. I believe the database contains all entries from all nodes, but the syslog only show local logs. Should the syslog be synced between the nodes with rsyslog? Can anyone enlighten me?
 
Problem now is that the Tracking Center and Queue is not in sync across the nodes. Are they supposed to be in sync?

No. The tracking center search local files /var/log/syslog*, so that is not supposed to be in sync.
 
OK, I see. Will it be possible to set up an additional node for proxmox that receive syslogs vi arsyslog from all nodes. Will the resulting syslog be searchable by pmg-log-tracker?
 
OK, I see. Will it be possible to set up an additional node for proxmox that receive syslogs vi arsyslog from all nodes. Will the resulting syslog be searchable by pmg-log-tracker?

No, that is currently not possible.
 
Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?
 
  • Like
Reactions: DerDanilo
Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?

That look useful, i was until now believing that the tracking center was in sync across nodes, did you leave a how to behind ? Does still works for you ?
 
Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?
Thomas,
Can you provide the steps to do tracking sync between nodes?
Thanks
 
Well, I made my own solution.
The main ideia was to keep it simple, the less changes, the better.
I have 2 servers, 1 master and 1 node.

Create a script on both servers. First, on master server:
Code:
ssh root@serverNODE cat /var/log/syslog >> /var/log/syslog.1
cat /var/log/syslog.1 | sort -M > /var/log/tmp1
uniq /var/log/tmp1 /var/log/syslog.1

Second, on node server:
Code:
ssh root@serverMASTER cat /var/log/syslog >> /var/log/syslog.1
cat /var/log/syslog.1 | sort -M > /var/log/tmp1
uniq /var/log/tmp1 /var/log/syslog.1

Finally, include the line at /etc/crontab on both servers:
Code:
* * * * * root /path/script.sh

Now, I can use the tracking center on Master server and get results from Node server also.
 
I woke up with a better solution.
Instead of copying logs from one server to another, messing up the files, I created my "own" pmg-log-tracker.

First you need to rename /usr/bin/pmg-log-tracker to something else, example pmg-log-tracker-default, on server Master and Node.

Then create /usr/bin/pmg-log-tracker script
Code:
/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@serverNODE /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'

"$@" is to pass all arguments to script
-v -l 2000 -s 1531882800 -e 1531969200 -n -g -f gmail.com
head -n -1 is to remove last line of first search results
STATUS: aborted by limit (too many hits)
sed '/^#/ d' is to remove the header parameters of second search results
# LogReader: 19220
# Query options
# Start: 2018-07-18 00:00:00 (1531872000)
# END: 2018-07-18 16:20:45 (1531930845)
# End Query Options
These filters are needed because PERL script that processes results failed with that unexpected characters.
 
I woke up with a better solution.
Instead of copying logs from one server to another, messing up the files, I created my "own" pmg-log-tracker.

First you need to rename /usr/bin/pmg-log-tracker to something else, example pmg-log-tracker-default, on server Master and Node.

Then create /usr/bin/pmg-log-tracker script
Code:
/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@serverNODE /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'

"$@" is to pass all arguments to script
-v -l 2000 -s 1531882800 -e 1531969200 -n -g -f gmail.com
head -n -1 is to remove last line of first search results
STATUS: aborted by limit (too many hits)
sed '/^#/ d' is to remove the header parameters of second search results
# LogReader: 19220
# Query options
# Start: 2018-07-18 00:00:00 (1531872000)
# END: 2018-07-18 16:20:45 (1531930845)
# End Query Options
These filters are needed because PERL script that processes results failed with that unexpected characters.
Hey Sommer,
I have gone ahead and tried to implement your script you have referenced above however I am running into an issue where the tracking center only shows the messages from the other node now instead of both. Any advice would be greatly appreciated.
 
Hey Sommer,
I have gone ahead and tried to implement your script you have referenced above however I am running into an issue where the tracking center only shows the messages from the other node now instead of both. Any advice would be greatly appreciated.
Make sure your script has the two lines, first runs local, second runs remote
 
Yes that is correct I have uploaded my current pmg-log-tracker. The only thing I changed was the IP address for the serverip and added the TXT file extension so I could upload the file.
When I run this script it appears the second line overwrites the first.
 

Attachments

  • pmg-log-tracker.txt
    122 bytes · Views: 78
Yes that is correct I have uploaded my current pmg-log-tracker. The only thing I changed was the IP address for the serverip and added the TXT file extension so I could upload the file.
When I run this script it appears the second line overwrites the first.

Check original pmg script name, path, permissions.
Give me ls -l pmg* at /usr/bin
 
Permissions are all set to 0755 with my file stored at /usr/bin/
That is odd!
Try running pmg-log-tracker-default -l 1 and pmg-log-tracker -l 1 to see if you get any errors. Maybe you need to remove head and sed to see what is happening
 
That is odd!
Try running pmg-log-tracker-default -l 1 and pmg-log-tracker -l 1 to see if you get any errors. Maybe you need to remove head and sed to see what is happening
So when I run pmg-log-tracker -l 1 I am getting an error stating head: invalid number of lines: ‘1\r’
 
It seems your script have some strange character after head -n -1 \r
\r is a line return I believe? I have gone ahead and deleted everything from the script and manually typed it out and it still references the same error.
head: invalid number of lines: ‘1\r’
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!