Spamhaus content filtering installation

[Cadmailer]

Hello,

Can I install Spamhaus content filtering (i.e. HBL) directly into SpamAssassin based on its Github instructions or does PMG has its own unique instruction?

Thanks.

 

[Stoiko Ivanov]

Can I install Spamhaus content filtering (i.e. HBL) directly into SpamAssassin based on its Github instructions or does PMG has its own unique instruction?

Haven't tried this explicitly myself - but from a quick glance at the github instructions I think it should work with those.

Let us know if you run into any problems with them - and also how Spamhaus performs in your environment!

 

[Cadmailer]

Hello Stoiko,

I was able to install Spamhaus DNSBL based on the Github instruction and it seems to be ok. This is the instruction given to me by the Spamhaus rep to install our commercial/paid Spamhaus DNSBL into SA. I assume it would replace the free and default SA Spamhaus DNSBL but I still have to confirm with Spamhaus tech support. I still have to test it to see if it works or if it performs well. I will keep you posted.

Thanks,
Brian

 

[Cadmailer]

Spamhaus support told me Github installl instruction will replace default non-commercial Spamhaus content filtering with commercial one.

 

[Stoiko Ivanov]

Thanks for sharing your progress! - If you find the time a short report, after a few weeks on how it actually performs in the public internet would be very much appreciated!

 

[Cadmailer]

When I installed the Spamhaus content filter to PMG, it was on a staging system. To test it, I sent test spam emails from Spamhaus, and got a couple of emails blocked, the rest went through, but I concluded Spamhaus was at least in there or installed properly. I backed up the staging system and restored the backup file to the 'production' PMG. I tested again but this time, none of the test spam emails were caught. I was wondering what is happening here, there's a couple of possibilities I'm considering:

1) The first email test result from Spamhaus was a false positive, that is, a couple of emails were caught but not by Spamhaus content filter, rather by the defaul SpamAssassin filters, like KAM, URIBL, rules, etc., so the Spamhaus content filter wasn't really integrated or installed. Is it possible the installation based on Github instruction could have been overwritten by the PMG template system?

2) The Spamhaus content filter configuration wasn't carried over to the production PMG by restoring the backup configuration from the staging PMG.

What do you think happened? Are there any config files I can check to see if the Spamhaus content filter was installed in PMG? Right now, I don't see any clues or indications of Spamhaus content filter working in PMG. Thanks.

 

[Stoiko Ivanov]

What do you think happened? Are there any config files I can check to see if the Spamhaus content filter was installed in PMG? Right now, I don't see any clues or indications of Spamhaus content filter working in PMG. Thanks.

I usually call `spamassassin -Dall --lint 2>&1 |less` - this should show you what spamassassin reads as its config (pmg-smtp-filter uses SpamAssassin as perl-module, but the config-parsing should be the same)

Is it possible the installation based on Github instruction could have been overwritten by the PMG template system?

you can check what is actually rendered in /etc/mail/spamsassassin, and if your modifications are still there ...

from a quick glance the .pre and .cf files should not be part of the shipped templates (but as said - please verify this on your installatin)

2) The Spamhaus content filter configuration wasn't carried over to the production PMG by restoring the backup configuration from the staging PMG.

depends on how you created the backup (the list of files backed up by `pmgbackup` does not include all of /etc/mail/spamassassin (as PMG uses the templateing system for everything apart from custom.cf and pmg-scores.cf (which both get included in the backup))

I hope this explains it

 

[Cadmailer]

Awesome! I will check them out and get back to you, thanks!

 

[Cadmailer]

By the way, just one quick follow up question - if Spamhaus is in PMG, wouldn't I see some Spamhaus headers being added to the email that adds to the spam scoring?

 

[Stoiko Ivanov]

By the way, just one quick follow up question - if Spamhaus is in PMG, wouldn't I see some Spamhaus headers being added to the email that adds to the spam scoring?

as said above - never got around to test this actively myself - so can only guess what exactly spamhaus spamassassin module does, but I'd guess so (PMG in general adds headers for the most part through the modify field action (apart from the ones that get added by postfix processing)

 

[Cadmailer]

I usually call `spamassassin -Dall --lint 2>&1 |less` - this should show you what spamassassin reads as its config (pmg-smtp-filter uses SpamAssassin as perl-module, but the config-parsing should be the same)

I checked the read config, load plugins, or everything, on the output but didn't see any mention of Spamhaus or its config files, like sh.cf, SH.pm

you can check what is actually rendered in /etc/mail/spamsassassin, and if your modifications are still there ...

from a quick glance the .pre and .cf files should not be part of the shipped templates (but as said - please verify this on your installatin)

All the files I copied from Spamhaus to /etc/mail/spamassassin based on the Github instruction, are no longer there, like sh.cf, SH.pm, etc.. But I was thinking maybe that's normal because of PMG's templating system? Or are the Spamhaus files supposed to stay there the whole time in /etc/mail/spamassassin?

depends on how you created the backup (the list of files backed up by `pmgbackup` does not include all of /etc/mail/spamassassin (as PMG uses the templateing system for everything apart from custom.cf and pmg-scores.cf (which both get included in the backup))

I hope this explains it

I just used the Backup Now button under Backup/Restore menu, nothing special.

Thanks.

 

[Stoiko Ivanov]

I just used the Backup Now button under Backup/Restore menu, nothing special.

I guess this is the problem then - as said - pmgbackup (or the GUI) does not backup all files in /etc/mail/spamassassin - so any manual modification needs to be carried over to the new system by you.

 

[Cadmailer]

I guess this is the problem then - as said - pmgbackup (or the GUI) does not backup all files in /etc/mail/spamassassin - so any manual modification needs to be carried over to the new system by you.

That makes sense, I will install again and let you know. Thanks.

 

[Cadmailer]

The first step in the installation is to check that the Spamhaus content filtering block list (HBL) is working, for example, you should get the result below:

$ dig +short abc123.hbl.dq.spamhaus.net
127.0.3.20

I didn't get this result in PMG console, it just returned an empty line. Note, my default DNS or 'DNS Server 1' points to the internal local Windows DNS Server on an old Windows 2008 server machine. Let's say the local DNS Server IP address is 192.168.10.10. When I run the command below, it works:

$ dig +short +nocookie @192.168.10.10 abc123.hbl.dq.spamhaus.net
127.0.3.20

It looks like there's a compatibillity issue with PMG and our internal local DNS Server. My concern is whether this issue is affecting the current PMG installation or its operation such that it's not using the 'DNS Server 1' for resolution but just the 'DNS Server 2' which is an external DNS (Google's 8.8.8.8). What do you think? Is our DNS settings even correct? Any suggestions?

Also, can you can explain what a 'Search domain' is? What is it being used for? For something that I think is quite important, the reference is very sparse on this ;-) Thanks.

 

[Stoiko Ivanov]

The first step in the installation is to check that the Spamhaus content filtering block list (HBL) is working, for example, you should get the result below:

$ dig +short abc123.hbl.dq.spamhaus.net
127.0.3.20

I didn't get this result in PMG console, it just returned an empty line

your could try without the `+short` parameter to get a bit more information on why this might have failed. - see `man dig` for more information on the command

It looks like there's a compatibillity issue with PMG and our internal local DNS Server.

Sometimes DNS-resolvers do not return answers pointing to 127.0.0.0/8 for a (sensible or not) security reason - see e.g. https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html (there have been a few threads on that topic in the PMG forums...)
No Idea if your Windows 2008 does this or something similar.
You can also run a DNS-server directly on your PMG - that can help around such issues: https://pmg.proxmox.com/wiki/index....edicated_DNS_Resolver_on_Proxmox_Mail_Gateway

installation or its operation such that it's not using the 'DNS Server 1' for resolution but just the 'DNS Server 2' which is an external DNS

It used to be that SpamAssassin (which would be the module that you're trying to configure just used the first DNS-server from /etc/resolv.conf - but I think that changed with version 4.0.0 (they reworked DNS-resolution quite a bit for that) - however I don't think I have tested this recently.

Also, can you can explain what a 'Search domain' is? What is it being used for? For something that I think is quite important, the reference is very sparse on this ;-) Thanks.

In general the concept is explained somewhat here: https://en.wikipedia.org/wiki/Search_domain
for PMG the relevant part is the part on /etc/resolv.conf here: https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#_configuration_files_overview (the first entry is used to create the FQDN of your PMG)
more technical details can be found at `man 5 resolv.conf`

 

[Cadmailer]

your could try without the `+short` parameter to get a bit more information on why this might have failed. - see `man dig` for more information on the command

I tried the command below with our internal Windows DNS and got an answer 'WARNING: recursion requested but not available'. However, recursion is enabled. I googled and read somewhere this happens with Windows DNS and linux clients and they suggested using the +nocookie parameter which works. Stupid question, does PMG use dig for DNS resolution? If so, then PMG may have a problem with our Windows DNS also. Running nslookup has no issues whatsoever in PMG. Anyways, it seems we're better off not using the Windows DNS altogether in PMG.

$ dig @192.168.10.10 abc123.hbl.dq.spamhaus.net

Sometimes DNS-resolvers do not return answers pointing to 127.0.0.0/8 for a (sensible or not) security reason - see e.g. https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html (there have been a few threads on that topic in the PMG forums...)
No Idea if your Windows 2008 does this or something similar.
You can also run a DNS-server directly on your PMG - that can help around such issues: https://pmg.proxmox.com/wiki/index....edicated_DNS_Resolver_on_Proxmox_Mail_Gateway


It used to be that SpamAssassin (which would be the module that you're trying to configure just used the first DNS-server from /etc/resolv.conf - but I think that changed with version 4.0.0 (they reworked DNS-resolution quite a bit for that) - however I don't think I have tested this recently.


In general the concept is explained somewhat here: https://en.wikipedia.org/wiki/Search_domain
for PMG the relevant part is the part on /etc/resolv.conf here: https://pmg.proxmox.com/pmg-docs/pmg-admin-guide.html#_configuration_files_overview (the first entry is used to create the FQDN of your PMG)
more technical details can be found at `man 5 resolv.conf`

 

[Stoiko Ivanov]

Stupid question, does PMG use dig for DNS resolution? If so, then PMG may have a problem with our Windows DNS also.

dig is a helper utility coming from the bind source - it's a quite feature rich DNS-client - but is normally not used to "do DNS resolution" by software ...
(deep down usually the libc-library calls getaddrinfo and similar are doing the DNS-resolution)

as it's not a common issue to have incompatibilities on how systems do DNS-resolution (at least I'm not aware of any reports) - and as there are very few threads on this topic with PMG I would assume that the issue might be with the windows DNS - but as said - I have no experience there.

 

[Cadmailer]

dig is a helper utility coming from the bind source - it's a quite feature rich DNS-client - but is normally not used to "do DNS resolution" by software ...
(deep down usually the libc-library calls getaddrinfo and similar are doing the DNS-resolution)

as it's not a common issue to have incompatibilities on how systems do DNS-resolution (at least I'm not aware of any reports) - and as there are very few threads on this topic with PMG I would assume that the issue might be with the windows DNS - but as said - I have no experience there.

Sounds good! I read somewhere that Windows DNS doesn't support DNS cookies until Windows Server 2012. Since dig uses DNS cookies by default this likely explains the issue and why the +nocookie parameter works.

Thanks for checking this out with me, I will continue the Spamhaus reinstall and keep you posted.