PMG huge blacklist blocks server

[DerDanilo]

Apparently a huge blacklist of 400k mail addresses and somewhat about 100k domains in the ruledb seems to slow down PMG server tremendously. I am speaking of delays from hours to days, not just minutes.

I could not find anything in the documentation about limitations for the ruledb.

@proxmox team
What are the limits for the ruledb? Did you run them with huge DBs?

 

[tom]

what are you doing here exactly, how do you create this list?

 

[DerDanilo]

Well I am testing if PMG can handle huge blacklists. For example I imported an email blacklist from here.
The import took quiet long, which is okay considering the amount of objects.

After the import finished I have had to reboot PMG while some services seem to have crashed during the import and the email queues started filling up with connection errors to local services (something on 127.0.0.1).

So, is there any known limitation for the amount of objects in those lists? How fast will the system filter mails against those blacklists once they are in the PMG db?

 

[dietmar]

The system is simply not designed to handle such large lists inside the rule configuration. Would be much faster to use a .db file which can be read by postfix directly. Or write a special rule who object which can effectively compare such large lists...

 

[DerDanilo]

Thanks for this information @dietmar . Shouldn't this be mentioned in the manual/wiki?
As a admin I wouldn't bother using features like blacklists if they are available. This is somewhat missleading if there is no mentioning of what the limits for the rule configuration are.

Do you have a simple template on how to write a rule that checks incoming mails against textfile based blacklists? I am also using ramdisks to speed these kind of things up. Another solution could be to use some caching mechanism (redis, memcached) that hold those dbs, thus allowing even faster checks.

 

[dietmar]

Large blacklist are usually deployed at postfix level directly, so I have no idea why you want/need that in the rule configuration?

 

[tom]

I suggest to use DNSBL, easy to configure and fast.

You can configure DNSBL to block on the SMTP level, or you just let spamassassin adds a score if the IP is listed.

 

[DerDanilo]

I suggest to use DNSBL, easy to configure and fast.

You can configure DNSBL to block on the SMTP level, or you just let spamassassin adds a score if the IP is listed.

Of course this is enabled but it's not enough. Adding well known blacklists is even better. Unfortunately it takes quiet a while to train the spam lists to be efficient. If mailservers are behind PMG(s) and the quarantine user features are disabled (not usable with hundread to thousand users who don't know a thing about IT) it is quiet a challenge to usefully configure anti spam measures to be efficient.
Adding mail addresses blacklists is one more measure to fight against spam.

I don't understand why the ruledb is available if it is not capable of handling large/huge lists. There is no indicator anywhere. It could as well be that the ruledb writes postfix .db files and has postfix use those.
I would be happy if you'd point me in the right direction on how to deploy update-save postix blacklists that are usable with PMGs.

Thanks in advance!

 

[guletz]

100k domains

Hi,

For a such big value, I would try to offload postfix/PMG from this task! Instead I would use a dedicateted private DNS, where I would create(scripting) a NULL dns authority domain for each of 100k black list. This DNS must be used by postfix/PMG stack! Main advantage will
be the caching of DNS!

Good luck!

 

[heutger]

Of course this is enabled but it's not enough. Adding well known blacklists is even better. Unfortunately it takes quiet a while to train the spam lists to be efficient. If mailservers are behind PMG(s) and the quarantine user features are disabled (not usable with hundread to thousand users who don't know a thing about IT) it is quiet a challenge to usefully configure anti spam measures to be efficient.
Adding mail addresses blacklists is one more measure to fight against spam.

I don't understand why the ruledb is available if it is not capable of handling large/huge lists. There is no indicator anywhere. It could as well be that the ruledb writes postfix .db files and has postfix use those.
I would be happy if you'd point me in the right direction on how to deploy update-save postix blacklists that are usable with PMGs.

Thanks in advance!

But isn't it a bit dangerous to use foreign blacklists? Would be better to code a bit, so that users actions (e.g. via dovecot sieve) could invoke a sa-learn spam or ham call.

Alternatively the idea of @guletz is great to use a DNS blacklist. Also using your own regex files on postfix level therefor wouldn't be such great, also if using postmap for hashing won't be such performant, I'm afraid. Also rejecting in the rule system is no good idea because of the handling of the rule system (drop/block instead of reject) to be compliant with German Law.

 

[guletz]

... and a better ideea:

- firewall dns works great but it has a downside: time to block the spammer is too fast, and this is not good because you want a very long delay until the spammer will get a answer

For this reason and others (easy scripting for example), you could do a better job, using a cheap router (with under 100 euro in my case, I can block more then 30.000 of bad ip spammer). Maybe in your case you will need a better model with more cpus.
This dedicated ruter is in front of my mail-server. Daily I run a schedule that will import ip adresses from several black-lists (spamhaus is one of them) in a more then 30k ip list(under 5 min). Then with a simple ip firewall rule I drop any incoming connection, so the spammers will not get any answer wich is much better then a host not found using a firewall dns. Also in this case cache routing is helping a lot!
And you can do much more with this kind of router regarding mail server, but I do not want to say more to boring people.

Good luck!

 

[heutger]

... and a better ideea:

- firewall dns works great but it has a downside: time to block the spammer is too fast, and this is not good because you want a very long delay until the spammer will get a answer

For this reason and others (easy scripting for example), you could do a better job, using a cheap router (with under 100 euro in my case, I can block more then 30.000 of bad ip spammer). Maybe in your case you will need a better model with more cpus.
This dedicated ruter is in front of my mail-server. Daily I run a schedule that will import ip adresses from several black-lists (spamhaus is one of them) in a more then 30k ip list(under 5 min). Then with a simple ip firewall rule I drop any incoming connection, so the spammers will not get any answer wich is much better then a host not found using a firewall dns. Also in this case cache routing is helping a lot!
And you can do much more with this kind of router regarding mail server, but I do not want to say more to boring people.

Good luck!

Sadly this forum seems not to provide a voting system. I recently added fail2ban to my setup and I'm just wondering if it's the correct setup. Let me explain: I have no statistical data yet, but it feels a bit like I have more people trying to access my servers than before. My goal, getting my logs (and tracking center) more clean and preserve my performance from e.g. doing content checks for milter-rejected sources again and again, has been reached for sure, but it looks like "just testing" connections increased. I also was very much unsure on how to setup fail2ban action. I considered to do a silent drop of the connection as the spammers already ignored rejects by milter, postscreen hangups or other reasons like blocklists, pregreet violations etc., they also won't react on rejected tcp connections, but on the other hand, I thought that if doing a silent drop it looks a bit like the server is not available, so the mail will be queued and they will try it again, once the server is up again or in my situation the fail2ban rule expires. So I decided for rejecting with message. How do you think on that?

 

[guletz]

so the mail will be queued and they will try it again, once the server is up again or in my situation the fail2ban rule expires. So I decided for rejecting with message. How do you think on that?

Hi @heuther,

Do not do this You will consume more of your resurces(cpu/ram/etc), and after some point it will be worst. I see in my case, that the SPAM is less when I use DROP compared with REJECT!

 

[heutger]

Hi @heuther,

Do not do this You will consume more of your resurces(cpu/ram/etc), and after some point it will be worst. I see in my case, that the SPAM is less when I use DROP compared with REJECT!

Silent drop is just "no good behavior" and that was why fail2ban changed its behavior too. However I will try to check the quotes and may reconsider to change to drop.

 

[dcsapak]

Apparently a huge blacklist of 400k mail addresses and somewhat about 100k domains in the ruledb seems to slow down PMG server tremendously. I am speaking of delays from hours to days, not just minutes.

i tested this here in the lab a bit and had no problem with a rule with 400000 email or 100000 domains... service restarting did take some time and the memory usage has gone up (of course) but a mail was still processed in a 'normal' time (< 2 seconds per mail)

while it is still a bad idea to do this with the rule system, may i ask how many resources (ram/cpu) you allocated for that pmg instance?

 

[DerDanilo]

i tested this here in the lab a bit and had no problem with a rule with 400000 email or 100000 domains... service restarting did take some time and the memory usage has gone up (of course) but a mail was still processed in a 'normal' time (< 2 seconds per mail)

while it is still a bad idea to do this with the rule system, may i ask how many resources (ram/cpu) you allocated for that pmg instance?

Short answer only since I'm on vacation.

6 vCPUs, 3.7ghz Intel xeon, single socket, overcomittment of 1.5 on CPU cores.
8-16Gb RAM

I will reply to other answers when I'm back.