Central Blocklist - Filters Not Working Reliably

[yummiweb]

PMG 9.0.4 (and and previous)

Dear Community, dear Developers,

The Proxmox Mailgateway is a truly helpful and very stable piece of software. Unfortunately, the filters behave somewhat unexpectedly—or rather, they sometimes fail to filter what they are supposed to. And because this is not decreasing but actually increasing, I would like to describe the problem today:

Emails from senders or domains that repeatedly send spam are radically blocked in my PMG. Email addresses or domain entries are added to the relevant blocklist for this purpose. This works in many cases, but unfortunately not in all.

Even though the corresponding domain entries have been added, some spammers still manage to bypass the filter. I investigated this and suspected that sometimes (perhaps) addresses enclosed in single quotes or angle brackets get through.

Since I didn't want to create a blacklist entry for every variant (or for every subdomain), I ended up using regular expressions and even created a script to generate these (rather complex) entries. The results look something like this (for "domain.tld" including all subdomains):

(?i)((['<]+)?.*@.*domain.tld(['>]+)?)(\s*)$

These entries covered the most common variations, making the filter considerably more reliable—but unfortunately not 100%. This level of reliability was sufficient for several years, and with each new PMG version, there was renewed hope that the filter itself would improve. Instead, I occasionally had to modify my regular expression entries because they no longer worked in the new version (I didn't have to change anything in the latest version, PMG 9.0.4).

These entries covered the common variants, and the filter became significantly more reliable—but unfortunately not 100%. What always seemed to work was blocking individual email addresses directly via an email entry, without using regular expressions.

However, in the current version (PMG 9.0.4), I've noticed that even this filter is no longer reliable. Specifically, we have one address that, despite the filter, is NOT blocked (and no, the address isn't on any welcome list, neither centrally nor personally).

The address looks like this: "harry66666666@domain.tld" and bypasses the filter EVERY SINGLE TIME instead of being blocked just once. Even a regular expression like this didn't help:

(?i)((['<]+)?harry66666666@domain.tld(['>]+)?)(\s*)$

And the sender isn't even a professional spammer using tricks, but simply an unwanted sender (a confused individual).

Therefore, I'd like to discuss this further and benefit from your experiences. I'd like to identify the causes in order to understand the situation and find a solution.

Thank you!

Regards, Yummiweb

 

[Badej]

Hi Yummiweb,

Try using Spamassassin. It allows for more fine-grained rule customization.
To correctly create a rule, you need original message headers. Perhaps they contain unique ones to catch spam.
Try it for example. It assigns a high score to TLDs found in a high percentage of spam mailings.

Add to /etc/mail/spamassassin/custom.cf

header  __DOMAIN_RISK_HIGH_FROM    From:addr =~ /\.(reisen|fit|ly|tt|fun|ir|buzz|pk|na|coupons|yachts|motorcycles|best|beauty|autos|sbs|pics|quest|rest|cfd|hair|click|pw|makeup|icu|lat|top|ovh|help|cam|ba|vip|site)$/i
header  __DOMAIN_RISK_HIGH_RPATH   Return-Path:addr =~ /\.(reisen|fit|ly|tt|fun|ir|buzz|pk|na|coupons|yachts|motorcycles|best|beauty|autos|sbs|pics|quest|rest|cfd|hair|click|pw|makeup|icu|lat|top|ovh|help|cam|ba|vip|site)$/i
header  __DOMAIN_RISK_HIGH_REPLY   Reply-To:addr =~ /\.(reisen|fit|ly|tt|fun|ir|buzz|pk|na|coupons|yachts|motorcycles|best|beauty|autos|sbs|pics|quest|rest|cfd|hair|click|pw|makeup|icu|lat|top|ovh|help|cam|ba|vip|site)$/i
meta      DOMAIN_RISK_HIGH    __DOMAIN_RISK_HIGH_FROM || __DOMAIN_RISK_HIGH_RPATH || __DOMAIN_RISK_HIGH_REPLY
describe  DOMAIN_RISK_HIGH    Cloudflare Radar: most abused TLDs (malicious/spam)
score     DOMAIN_RISK_HIGH    7.0

spamassassin -D --lint
systemctl restart pmg-smtp-filter.service

 

[yummiweb]

Hello Badej,

Thank you for your attempt at an answer, but unfortunately it doesn't address the core of my question – even though I explicitly mentioned that my example isn't about blocking a spammer at all.

So it's not about configuring spam filters or integrating spam lists, but about address-based blocklists.

I always find it frustrating when questions like "I'm having problems with component A." are met with answers like "Why don't you use something completely different?"

If there's absolutely no solution to a problem, it certainly makes sense to consider workarounds, but right now we're talking about unexpected behavior from a function – which is either just a matter of understanding or usability, or could even indicate a functional error or bug. I think it's sensible not to avoid this but rather to investigate it – either for yourself, for others, or for the developers.

Regards, Yummiweb

 

[smileluo]

use something like below format to match any main domain and subdomain:

.*@([^.]+\.)*chenyingwen\.cn$

 

[smileluo]

You can use it at who object,add a blacklist and use Regular Expression to add: .*@([^.]+\.)*chenyingwen\.cn$

 

[Badej]

In emails, there are technically two different "From" addresses:
Envelope From / Return-Path (RFC 5321. MailFrom)
Header From (RFC 5322. From)
Both need to be checked.

header   __CHENYINGWEN_FROM    From:addr =~ /@(?:[^.]+\.)*chenyingwen\.cn$/i
header   __CHENYINGWEN_RETURN  Return-Path:addr =~ /@(?:[^.]+\.)*chenyingwen\.cn$/i
header   __CHENYINGWEN_REPLY   Reply-To:addr =~ /@(?:[^.]+\.)*chenyingwen\.cn$/i
meta     BLOCK_CHENYINGWEN_ALL __CHENYINGWEN_FROM || __CHENYINGWEN_RETURN || __CHENYINGWEN_REPLY
describe BLOCK_CHENYINGWEN_ALL Total block for chenyingwen.cn (From/Return-Path/Reply-To)
score    BLOCK_CHENYINGWEN_ALL 10.0

The :addr header modifier only checks the email address in the From: header.

 

[yummiweb]

use something like below format to match any main domain and subdomain:

.*@([^.]+\.)*chenyingwen\.cn$

You're saying my regex isn't correct? The provided test field processes my regex perfectly and matches everything it's supposed to.

Your regex might work as well—but it blocks entire domains, and that wasn't the point. My example was about a single(!) sender address.

And a sender address is specific enough that it shouldn't even require a regex.

That's why I suspect this might be a bug—possibly caused by the sender structure.

You can use it at who object,add a blacklist and use Regular Expression to add: .*@([^.]+\.)*chenyingwen\.cn$

Did you just explain to me how to create a regular expression (regex) entry in a Who list? I already referred to regular expression entries in my original question—so what's the basis for assuming I don't know how and where to add them?

Please excuse me if I come across as a bit blunt, but I try to include all the essential information in my initial question—as it should be. Therefore, I find it somewhat "odd" that this information apparently wasn't read.

 

[Badej]

For GUI

It's best to use "What objects" for checking and use it in the filter.
Create a "Blacklist From" object (for example)
Add a Match Field

Field: From
Value: .*@([^.]+\.)*chenyingwen\.cn$

Field: Return-Path
Value: .*@([^.]+\.)*chenyingwen\.cn$

 

[Badej]

Documentation:
/pmg-docs/pmg-admin-guide.html#pmg_mailfilter_who

We have two important Who objects called Blacklist and Whitelist. These are used in the default ruleset to globally block or allow specific senders.
Note that Who objects match against the mail addresses given during the SMTP dialog:

• Who objects in the FROM category match against the sender’s mail address given in the MAIL command (the Envelope From).
• Who objects in the TO category match against the recipient’s mail address given in the RCPT command (the Envelope To).

These addresses are also visible in the logs and the Tracking Center.
If you want to match against the From: or To: header fields of the mail, use a What object with type Match Field instead.