Trying to get accurate user count with pmgsh

[MiamiJack]

Hello All,

When I try to get a report of spam blocked for each of my domains, I use:
pmgsh get /statistics/receiver -month $1 -year $2 > $1-$2.json

I then have some code that splits it up by domain, but the problem is if someone does a dictionary attack on a domain, it then shows like I have 100 users instead of 10.

Is there a way to only get those that accepted email so we know those are legit users vs just bad email addresses?

Thanks!

 

[MiamiJack]

@Stoiko Ivanov

Hello, sorry to tag you on this, but I never got anywhere with this and the situation came up again.
I'm trying to determine email addresses that we delivered to, versus those rejected.
So if I have only 1 valid email address for a domain, lets say 1@abc.com, when I run the report it says 10 email address for that domain received email.

Any suggestions on how I can determine what did get delivered versus not?

Indirect question, pmgsh gives back the spamcount, that is what pmg rejected, quarantined or both?
Also I currently have RBL's at the postfix level, if I turn them on at the SA level, will those get rejected or quarantined? and will they show up in pmgsh data?

Thank you!!!

 

[Stoiko Ivanov]

Any suggestions on how I can determine what did get delivered versus not?

I would use the logs for this - maybe through the tracking center api - or directly.

Indirect question, pmgsh gives back the spamcount, that is what pmg rejected, quarantined or both?

rejected or quarantined - i.e. the final action of a mail according to your rule-system does not play a role here
spamcount_in and out are those mails that have more than a SA score of 3.

Also I currently have RBL's at the postfix level, if I turn them on at the SA level, will those get rejected or quarantined? and will they show up in pmgsh data?

I would suggest to keep some dnsbls in postfix/postscreen - it's faster and more efficient - those mails are included (as well as spamcount mails, virus mails, and greylisted mails that did not try delivery again) in junk_in and junk_out.

I hope this helps!

 

[MiamiJack]

I will try to address the above points individually:
I totally understand the performance improvement of blocking at the postfix level, and I commented out those SA configuration lines to utilize that early on, but there were 2 issues, 1- if a legitimate email was blacklisted, it could not be seen by the user to recover it, and 2- it's not being counted in what's being blocked which is my initial goal here.
This count justifies resources used and it's value.

I verified the count:
I created a script to grab the pmgsh output for September, it then does some organization and calculates totals.
According to the data from pmgsh my server spam count is 8,235 for all users, however a quick grep of the September logfile results in 53,308 "blocked using an RBL" this is ONLY RBL and not counting those items quarantined.

using this command:
grep -i "blocked using" /var/log/mail.log.1 | wc -l Result: 53,308

grep "rule: Quarantine" /var/log/mail.log.1 | wc -l grep: /var/log/mail.log.1: binary file matches Result: 5,722

grep -i "rule: bl" /var/log/mail.log.1 | awk '{print $11,$12}' | sort | uniq -c grep: /var/log/mail.log.1: binary file matches 19 outgoing Spam, 24 (rule: Block 161 (rule: BL-What-From) 65 (rule: BL-Who-Domains) 1,172 (rule: BL-Who-DomainsR) 1,232 (rule: BL-Who-IP-NW) 3 to virus 1 Viruses, 79DB680C60) 1 Viruses, 886098014B) 1 Viruses, C5E6681063)

grep "cannot find your reverse" /var/log/mail.log.1 | wc -l grep: /var/log/mail.log.1: binary file matches Result: 29,251

As you can see from multiple queries, PMG is doing much more than is reported.

I did turn a few RBL's off at the postfix level, and turn them back on in SA to see if there are benefits for recovery and count.

 

[Stoiko Ivanov]

As you can see from multiple queries, PMG is doing much more than is reported.

As said - mails are counted as spam if they have a SA score of 3 or above - the statistics do not take into account your ruleset.
The junk-mails include those mails that are blocked by postscreen...

 

[MiamiJack]

Sorry to post on this older thread, but all related and was trying to improve upon the information I'm extracting for stats.

When I run from the GUI -> Receiver, then pick incoming, month of march, whole month I pick a couple of users to get data from, and that data doesn't match the data from pmgsh statistics.

Example:
Receiver 1:
GUI Says 887 Total and 97 Spam
PMGSH Says 927 and 97 spam

Receiver 2:
GUI Says 960 Total and 116 Spam
PMGSH Says 1045 and 116 spam

Any ideas as to what can be the descrepency?

Thanks!

 

[dcsapak]

and that data doesn't match the data from pmgsh statistics.

what is the exact pmgsh command?

 

[MiamiJack]

what is the exact pmgsh command?

The command I use is:
pmgsh get /statistics/receiver -month 03 -year $2023

This would give me the month I'm researching so in this case all of March, then it results in the json file which I can verify each users stats against the GUI.

 

[MiamiJack]

The command I use is:
pmgsh get /statistics/receiver -month 03 -year $2023

This would give me the month I'm researching so in this case all of March, then it results in the json file which I can verify each users stats against the GUI.

I think part of the issue is that reporting is case sensitive....
Can we do anything command line or code wise to ignore case?

 

[dcsapak]

I think part of the issue is that reporting is case sensitive....
Can we do anything command line or code wise to ignore case?

no, but i don't think actually that this is the issue (at least from the output/commands you posted)

the gui and pmgsh uses the same api call, but probably with different parameters.

the gui calculates a start and end date (in unix epoch form) from the dropdown field, but that is always localized (iow. it respects your local browser timezone)
maybe it's just off because you and the gui query a different timespan

 

[MiamiJack]

no, but i don't think actually that this is the issue (at least from the output/commands you posted)

the gui and pmgsh uses the same api call, but probably with different parameters.

the gui calculates a start and end date (in unix epoch form) from the dropdown field, but that is always localized (iow. it respects your local browser timezone)
maybe it's just off because you and the gui query a different timespan

That is a possibility, however the case sensitive aspect is an issue since it will list the same email address 2 or 3 different ways depending on matching case.

 

[dcsapak]

again, look at the requests/responses from the gui and the pmgsh, both are using the same api, but different parameters, nothing in the code would suggest that we do any case sensitive/insensitive shenanigans

we never mangle e-mail adresses together with different cases AFAIR