Tracking Center not in sync

Mar 4, 2018
11
2
8
Oslo
A couple weeks ago I sat up a PMG cluster with 3 nodes. It works fine now, after having some initial problems syncing the database to the third node. I solved that by logging in to postgresql with psql and deleted entries from the cstatic-table. (The log complained about duplicate entries.) Then the sync between the nodes started working and errors in the logs went away. (To get into postgresql, I had to edit pg_hba.conf to trust localhost logons without password and reload postgresql)

Problem now is that the Tracking Center and Queue is not in sync across the nodes. Are they supposed to be in sync? Can't find any information about that in the manual or here in the forums. All nodes can ssh to each other and there is no errors in the logs. Also the all settings and Quarantine is synced, so apparently everything is working -- except Tracking Center and Queues...

I wonder where Tracking Center is getting it's info from? I guess it's a combination of database and log files. I believe the database contains all entries from all nodes, but the syslog only show local logs. Should the syslog be synced between the nodes with rsyslog? Can anyone enlighten me?
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
17,003
467
103
Austria
www.proxmox.com
Problem now is that the Tracking Center and Queue is not in sync across the nodes. Are they supposed to be in sync?

No. The tracking center search local files /var/log/syslog*, so that is not supposed to be in sync.
 
Mar 4, 2018
11
2
8
Oslo
OK, I see. Will it be possible to set up an additional node for proxmox that receive syslogs vi arsyslog from all nodes. Will the resulting syslog be searchable by pmg-log-tracker?
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
17,003
467
103
Austria
www.proxmox.com
OK, I see. Will it be possible to set up an additional node for proxmox that receive syslogs vi arsyslog from all nodes. Will the resulting syslog be searchable by pmg-log-tracker?

No, that is currently not possible.
 
Mar 4, 2018
11
2
8
Oslo
Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?
 
  • Like
Reactions: DerDanilo

Dragnell

New Member
Jul 30, 2012
5
0
1
Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?

That look useful, i was until now believing that the tracking center was in sync across nodes, did you leave a how to behind ? Does still works for you ?
 

Sommer

Member
Jun 7, 2018
47
6
8
36
Um ok -- but I actually did set up an additional node yesterday -- and it works like a charm! Now we have all logs collected at one host that we use for tracking! Don't see why this is not done by PMG by default?
Thomas,
Can you provide the steps to do tracking sync between nodes?
Thanks
 

Sommer

Member
Jun 7, 2018
47
6
8
36
Well, I made my own solution.
The main ideia was to keep it simple, the less changes, the better.
I have 2 servers, 1 master and 1 node.

Create a script on both servers. First, on master server:
Code:
ssh root@serverNODE cat /var/log/syslog >> /var/log/syslog.1
cat /var/log/syslog.1 | sort -M > /var/log/tmp1
uniq /var/log/tmp1 /var/log/syslog.1

Second, on node server:
Code:
ssh root@serverMASTER cat /var/log/syslog >> /var/log/syslog.1
cat /var/log/syslog.1 | sort -M > /var/log/tmp1
uniq /var/log/tmp1 /var/log/syslog.1

Finally, include the line at /etc/crontab on both servers:
Code:
* * * * * root /path/script.sh

Now, I can use the tracking center on Master server and get results from Node server also.
 
  • Like
Reactions: DerDanilo

Sommer

Member
Jun 7, 2018
47
6
8
36
I woke up with a better solution.
Instead of copying logs from one server to another, messing up the files, I created my "own" pmg-log-tracker.

First you need to rename /usr/bin/pmg-log-tracker to something else, example pmg-log-tracker-default, on server Master and Node.

Then create /usr/bin/pmg-log-tracker script
Code:
/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@serverNODE /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'

"$@" is to pass all arguments to script
-v -l 2000 -s 1531882800 -e 1531969200 -n -g -f gmail.com
head -n -1 is to remove last line of first search results
STATUS: aborted by limit (too many hits)
sed '/^#/ d' is to remove the header parameters of second search results
# LogReader: 19220
# Query options
# Start: 2018-07-18 00:00:00 (1531872000)
# END: 2018-07-18 16:20:45 (1531930845)
# End Query Options
These filters are needed because PERL script that processes results failed with that unexpected characters.
 

kyles2013

New Member
Jul 29, 2018
9
1
1
26
I woke up with a better solution.
Instead of copying logs from one server to another, messing up the files, I created my "own" pmg-log-tracker.

First you need to rename /usr/bin/pmg-log-tracker to something else, example pmg-log-tracker-default, on server Master and Node.

Then create /usr/bin/pmg-log-tracker script
Code:
/usr/bin/pmg-log-tracker-default "$@" | head -n -1
ssh root@serverNODE /usr/bin/pmg-log-tracker-default "$@" | sed '/^#/ d'

"$@" is to pass all arguments to script
-v -l 2000 -s 1531882800 -e 1531969200 -n -g -f gmail.com
head -n -1 is to remove last line of first search results
STATUS: aborted by limit (too many hits)
sed '/^#/ d' is to remove the header parameters of second search results
# LogReader: 19220
# Query options
# Start: 2018-07-18 00:00:00 (1531872000)
# END: 2018-07-18 16:20:45 (1531930845)
# End Query Options
These filters are needed because PERL script that processes results failed with that unexpected characters.
Hey Sommer,
I have gone ahead and tried to implement your script you have referenced above however I am running into an issue where the tracking center only shows the messages from the other node now instead of both. Any advice would be greatly appreciated.
 

Sommer

Member
Jun 7, 2018
47
6
8
36
Hey Sommer,
I have gone ahead and tried to implement your script you have referenced above however I am running into an issue where the tracking center only shows the messages from the other node now instead of both. Any advice would be greatly appreciated.
Make sure your script has the two lines, first runs local, second runs remote
 

kyles2013

New Member
Jul 29, 2018
9
1
1
26
Yes that is correct I have uploaded my current pmg-log-tracker. The only thing I changed was the IP address for the serverip and added the TXT file extension so I could upload the file.
When I run this script it appears the second line overwrites the first.
 

Attachments

  • pmg-log-tracker.txt
    122 bytes · Views: 50

Sommer

Member
Jun 7, 2018
47
6
8
36
Yes that is correct I have uploaded my current pmg-log-tracker. The only thing I changed was the IP address for the serverip and added the TXT file extension so I could upload the file.
When I run this script it appears the second line overwrites the first.

Check original pmg script name, path, permissions.
Give me ls -l pmg* at /usr/bin
 

kyles2013

New Member
Jul 29, 2018
9
1
1
26
Check original pmg script name, path, permissions.
Give me ls -l pmg* at /usr/bin
Permissions are all set to 0755 with my file stored at /usr/bin/
 

Attachments

  • putty_2018-08-01_16-12-36.png
    putty_2018-08-01_16-12-36.png
    28.9 KB · Views: 30

Sommer

Member
Jun 7, 2018
47
6
8
36
Permissions are all set to 0755 with my file stored at /usr/bin/
That is odd!
Try running pmg-log-tracker-default -l 1 and pmg-log-tracker -l 1 to see if you get any errors. Maybe you need to remove head and sed to see what is happening
 

kyles2013

New Member
Jul 29, 2018
9
1
1
26
That is odd!
Try running pmg-log-tracker-default -l 1 and pmg-log-tracker -l 1 to see if you get any errors. Maybe you need to remove head and sed to see what is happening
So when I run pmg-log-tracker -l 1 I am getting an error stating head: invalid number of lines: ‘1\r’
 

kyles2013

New Member
Jul 29, 2018
9
1
1
26
It seems your script have some strange character after head -n -1 \r
\r is a line return I believe? I have gone ahead and deleted everything from the script and manually typed it out and it still references the same error.
head: invalid number of lines: ‘1\r’
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!