Spamhaus content filtering installation

Cadmailer said:
Can I install Spamhaus content filtering (i.e. HBL) directly into SpamAssassin based on its Github instructions or does PMG has its own unique instruction?
Click to expand...
Haven't tried this explicitly myself - but from a quick glance at the github instructions I think it should work with those.

Let us know if you run into any problems with them - and also how Spamhaus performs in your environment!
 
Hello Stoiko,

I was able to install Spamhaus DNSBL based on the Github instruction and it seems to be ok. This is the instruction given to me by the Spamhaus rep to install our commercial/paid Spamhaus DNSBL into SA. I assume it would replace the free and default SA Spamhaus DNSBL but I still have to confirm with Spamhaus tech support. I still have to test it to see if it works or if it performs well. I will keep you posted.

Thanks,
Brian
 
  • Like
Reactions: Stoiko Ivanov
Thanks for sharing your progress! - If you find the time a short report, after a few weeks on how it actually performs in the public internet would be very much appreciated!
 
When I installed the Spamhaus content filter to PMG, it was on a staging system. To test it, I sent test spam emails from Spamhaus, and got a couple of emails blocked, the rest went through, but I concluded Spamhaus was at least in there or installed properly. I backed up the staging system and restored the backup file to the 'production' PMG. I tested again but this time, none of the test spam emails were caught. I was wondering what is happening here, there's a couple of possibilities I'm considering:

1) The first email test result from Spamhaus was a false positive, that is, a couple of emails were caught but not by Spamhaus content filter, rather by the defaul SpamAssassin filters, like KAM, URIBL, rules, etc., so the Spamhaus content filter wasn't really integrated or installed. Is it possible the installation based on Github instruction could have been overwritten by the PMG template system?

2) The Spamhaus content filter configuration wasn't carried over to the production PMG by restoring the backup configuration from the staging PMG.

What do you think happened? Are there any config files I can check to see if the Spamhaus content filter was installed in PMG? Right now, I don't see any clues or indications of Spamhaus content filter working in PMG. Thanks.
 
Cadmailer said:
What do you think happened? Are there any config files I can check to see if the Spamhaus content filter was installed in PMG? Right now, I don't see any clues or indications of Spamhaus content filter working in PMG. Thanks.
Click to expand...
I usually call `spamassassin -Dall --lint 2>&1 |less` - this should show you what spamassassin reads as its config (pmg-smtp-filter uses SpamAssassin as perl-module, but the config-parsing should be the same)

Cadmailer said:
Is it possible the installation based on Github instruction could have been overwritten by the PMG template system?
Click to expand...
you can check what is actually rendered in /etc/mail/spamsassassin, and if your modifications are still there ...

from a quick glance the .pre and .cf files should not be part of the shipped templates (but as said - please verify this on your installatin)

Cadmailer said:
2) The Spamhaus content filter configuration wasn't carried over to the production PMG by restoring the backup configuration from the staging PMG.
Click to expand...
depends on how you created the backup (the list of files backed up by `pmgbackup` does not include all of /etc/mail/spamassassin (as PMG uses the templateing system for everything apart from custom.cf and pmg-scores.cf (which both get included in the backup))

I hope this explains it
 
Cadmailer said:
By the way, just one quick follow up question - if Spamhaus is in PMG, wouldn't I see some Spamhaus headers being added to the email that adds to the spam scoring?
Click to expand...
as said above - never got around to test this actively myself - so can only guess what exactly spamhaus spamassassin module does, but I'd guess so (PMG in general adds headers for the most part through the modify field action (apart from the ones that get added by postfix processing)
 
Stoiko Ivanov said:
I usually call `spamassassin -Dall --lint 2>&1 |less` - this should show you what spamassassin reads as its config (pmg-smtp-filter uses SpamAssassin as perl-module, but the config-parsing should be the same)
Click to expand...
I checked the read config, load plugins, or everything, on the output but didn't see any mention of Spamhaus or its config files, like sh.cf, SH.pm
Stoiko Ivanov said:
you can check what is actually rendered in /etc/mail/spamsassassin, and if your modifications are still there ...

from a quick glance the .pre and .cf files should not be part of the shipped templates (but as said - please verify this on your installatin)
Click to expand...
All the files I copied from Spamhaus to /etc/mail/spamassassin based on the Github instruction, are no longer there, like sh.cf, SH.pm, etc.. But I was thinking maybe that's normal because of PMG's templating system? Or are the Spamhaus files supposed to stay there the whole time in /etc/mail/spamassassin?
Stoiko Ivanov said:
depends on how you created the backup (the list of files backed up by `pmgbackup` does not include all of /etc/mail/spamassassin (as PMG uses the templateing system for everything apart from custom.cf and pmg-scores.cf (which both get included in the backup))

I hope this explains it
Click to expand...
I just used the Backup Now button under Backup/Restore menu, nothing special.

Thanks.
 
Cadmailer said:
I just used the Backup Now button under Backup/Restore menu, nothing special.
Click to expand...
I guess this is the problem then - as said - pmgbackup (or the GUI) does not backup all files in /etc/mail/spamassassin - so any manual modification needs to be carried over to the new system by you.
 
The first step in the installation is to check that the Spamhaus content filtering block list (HBL) is working, for example, you should get the result below:

$ dig +short abc123.hbl.dq.spamhaus.net
127.0.3.20

I didn't get this result in PMG console, it just returned an empty line. Note, my default DNS or 'DNS Server 1' points to the internal local Windows DNS Server on an old Windows 2008 server machine. Let's say the local DNS Server IP address is 192.168.10.10. When I run the command below, it works:

$ dig +short +nocookie @192.168.10.10 abc123.hbl.dq.spamhaus.net
127.0.3.20

It looks like there's a compatibillity issue with PMG and our internal local DNS Server. My concern is whether this issue is affecting the current PMG installation or its operation such that it's not using the 'DNS Server 1' for resolution but just the 'DNS Server 2' which is an external DNS (Google's 8.8.8.8). What do you think? Is our DNS settings even correct? Any suggestions?

Also, can you can explain what a 'Search domain' is? What is it being used for? For something that I think is quite important, the reference is very sparse on this ;-) Thanks.
 
Cadmailer said:
The first step in the installation is to check that the Spamhaus content filtering block list (HBL) is working, for example, you should get the result below:

$ dig +short abc123.hbl.dq.spamhaus.net
127.0.3.20

I didn't get this result in PMG console, it just returned an empty line
Click to expand...
your could try without the `+short` parameter to get a bit more information on why this might have failed. - see `man dig` for more information on the command

Cadmailer said:
It looks like there's a compatibillity issue with PMG and our internal local DNS Server.
Click to expand...
Sometimes DNS-resolvers do not return answers pointing to 127.0.0.0/8 for a (sensible or not) security reason - see e.g. https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html (there have been a few threads on that topic in the PMG forums...)
No Idea if your Windows 2008 does this or something similar.
You can also run a DNS-server directly on your PMG - that can help around such issues: https://pmg.proxmox.com/wiki/index....edicated_DNS_Resolver_on_Proxmox_Mail_Gateway

Cadmailer said:
installation or its operation such that it's not using the 'DNS Server 1' for resolution but just the 'DNS Server 2' which is an external DNS
Click to expand...
It used to be that SpamAssassin (which would be the module that you're trying to configure just used the first DNS-server from /etc/resolv.conf - but I think that changed with version 4.0.0 (they reworked DNS-resolution quite a bit for that) - however I don't think I have tested this recently.

Cadmailer said:
Also, can you can explain what a 'Search domain' is? What is it being used for? For something that I think is quite important, the reference is very sparse on this ;-) Thanks.
Click to expand...
In general the concept is explained somewhat here: https://en.wikipedia.org/wiki/Search_domain
for PMG the relevant part is the part on /etc/resolv.conf here: https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#_configuration_files_overview (the first entry is used to create the FQDN of your PMG)
more technical details can be found at `man 5 resolv.conf`
 
Stoiko Ivanov said:
your could try without the `+short` parameter to get a bit more information on why this might have failed. - see `man dig` for more information on the command
Click to expand...

I tried the command below with our internal Windows DNS and got an answer 'WARNING: recursion requested but not available'. However, recursion is enabled. I googled and read somewhere this happens with Windows DNS and linux clients and they suggested using the +nocookie parameter which works. Stupid question, does PMG use dig for DNS resolution? If so, then PMG may have a problem with our Windows DNS also. Running nslookup has no issues whatsoever in PMG. Anyways, it seems we're better off not using the Windows DNS altogether in PMG.

$ dig @192.168.10.10 abc123.hbl.dq.spamhaus.net

Stoiko Ivanov said:
Sometimes DNS-resolvers do not return answers pointing to 127.0.0.0/8 for a (sensible or not) security reason - see e.g. https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html (there have been a few threads on that topic in the PMG forums...)
No Idea if your Windows 2008 does this or something similar.
You can also run a DNS-server directly on your PMG - that can help around such issues: https://pmg.proxmox.com/wiki/index....edicated_DNS_Resolver_on_Proxmox_Mail_Gateway


It used to be that SpamAssassin (which would be the module that you're trying to configure just used the first DNS-server from /etc/resolv.conf - but I think that changed with version 4.0.0 (they reworked DNS-resolution quite a bit for that) - however I don't think I have tested this recently.


In general the concept is explained somewhat here: https://en.wikipedia.org/wiki/Search_domain
for PMG the relevant part is the part on /etc/resolv.conf here: https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#_configuration_files_overview (the first entry is used to create the FQDN of your PMG)
more technical details can be found at `man 5 resolv.conf`
Click to expand...
 
Cadmailer said:
Stupid question, does PMG use dig for DNS resolution? If so, then PMG may have a problem with our Windows DNS also.
Click to expand...
dig is a helper utility coming from the bind source - it's a quite feature rich DNS-client - but is normally not used to "do DNS resolution" by software ...
(deep down usually the libc-library calls getaddrinfo and similar are doing the DNS-resolution)

as it's not a common issue to have incompatibilities on how systems do DNS-resolution (at least I'm not aware of any reports) - and as there are very few threads on this topic with PMG I would assume that the issue might be with the windows DNS - but as said - I have no experience there.
 
Stoiko Ivanov said:
dig is a helper utility coming from the bind source - it's a quite feature rich DNS-client - but is normally not used to "do DNS resolution" by software ...
(deep down usually the libc-library calls getaddrinfo and similar are doing the DNS-resolution)

as it's not a common issue to have incompatibilities on how systems do DNS-resolution (at least I'm not aware of any reports) - and as there are very few threads on this topic with PMG I would assume that the issue might be with the windows DNS - but as said - I have no experience there.
Click to expand...
Sounds good! I read somewhere that Windows DNS doesn't support DNS cookies until Windows Server 2012. Since dig uses DNS cookies by default this likely explains the issue and why the +nocookie parameter works.

Thanks for checking this out with me, I will continue the Spamhaus reinstall and keep you posted.
 
Stoiko Ivanov said:
Thanks for sharing your progress! - If you find the time a short report, after a few weeks on how it actually performs in the public internet would be very much appreciated!
Click to expand...

It's been a while, but I can say that the commercial spamhaus DNSBL installation works based on the Github instruction. For example, I can see the hits based on the headers, like SH_ZRD_HEADERS_FRESH, SH_DBL_HEADERS, etc. You can send probes/test spam emails from Spamhaus, but it can't verify them even when the emails were blocked or quarantined properly. This is probably because PMG queues emails by default, therefore, it "accepts" all emails regardless if they are spam or not. We have been using the the paid/commericial DNSBL for about two months now and it's helping when it comes to the worst URLs and IPs out there. We use the DBL, ZRD, and HBL to scan the email content including the headers. I don't have any stats at the moment, I may do that next time, in the meantime I would like to know what you think.
 
Cadmailer said:
It's been a while, but I can say that the commercial spamhaus DNSBL installation works based on the Github instruction. For example, I can see the hits based on the headers, like SH_ZRD_HEADERS_FRESH, SH_DBL_HEADERS, etc. You can send probes/test spam emails from Spamhaus, but it can't verify them even when the emails were blocked or quarantined properly. This is probably because PMG queues emails by default, therefore, it "accepts" all emails regardless if they are spam or not. We have been using the the paid/commericial DNSBL for about two months now and it's helping when it comes to the worst URLs and IPs out there. We use the DBL, ZRD, and HBL to scan the email content including the headers. I don't have any stats at the moment, I may do that next time, in the meantime I would like to know what you think.
Click to expand...
I tried myself free version. Followed all instructions from github and it works fine but only for content filtering. All SMTP test emails were delivered because these should have been rejected during SMTP session (Spamassasin comes after). But all Content test emails were quarantined properly, you can see this in the logs
Code: 
2025-07-10T18:53:28.415166+02:00 pmg pmg-smtp-filter[936]: 240C6E686FF00601CB3: SA score=9/5 time=2.358 bayes=undefined autolearn=no autolearn_force=no hits=AWL(1.727),DMARC_MISSING(0.1),KAM_BADIPHTTP(2),KAM_DMARC_STATUS(0.01),NORMAL_HTTP_TO_IP(0.001),NUMERIC_HTTP_ADDR(0.001),SH_SBL_URL(6),SPF_HELO_PASS(-0.001),SPF_PASS(-0.001)
Spamhaus added 6 points to the score.
 
Last edited:
You must log in or register to reply here.
Top Bottom