[TUTORIAL] Advancing Proxmox Mail Gateway (especially Spam and Virus Detection)

[velocity08]

Hi Christian

Was wondering if you’ve tested this in a PMG cluster and if all the backend changes are copied across to the other cluster nodes?

“”Cheers
G

 

[Jarkko]

Hi for you all,

What do you think about F-Secure Linux Security, is it possible to use to this ?
Here is what i have in my mind
https://help.f-secure.com/product.html#business/releasenotes-business/latest/en/fsls-latest-en

 

[heutger]

Hi Christian

Was wondering if you’ve tested this in a PMG cluster and if all the backend changes are copied across to the other cluster nodes?

“”Cheers
G

Honestly I didn't. I had an enquiry to do it on a paid job, but seems, they aren't interested any more. From my point of view, you would need to to do most adjustments on both nodes as PMG will only copy their own things like templates and settings but additional modules like DCC, Pyzor, local DNS server, milter with copying the milter databases all need to be done by yourself.

 

[heutger]

Hi for you all,

What do you think about F-Secure Linux Security, is it possible to use to this ?
Here is what i have in my mind
https://help.f-secure.com/product.html#business/releasenotes-business/latest/en/fsls-latest-en

I considered F-Secure, but pricing isn't attractive. Avast is just beaten by Dr.Web, but unsure on how good Dr.Web is working.

 

[guletz]

Hi,

In my own opinion antivirus with signatures, as a technology is end of life. Now is almost impossible for most cases to block zero day virus. It os so easy to create new viruses with encrypted payload... as I read. And if you start to dig on Internet maybe you will see that more increses of new viruses. More new viruses / year for my point of view must need more human hour to deliver a antidot => higher prices for many antivirus tools.

But as I find in my own case (I do not want to say that this could be usable for any envitoment) I can live without any antivirus tools using this simple ideeas (it work for me for amost 7 years with only clamav, but clamav has seen let say only 1 maybe 2 viruses / year)


- no exe/bat/msi/js/etc for any mail
- only webmail and not a windows mail client
- any client who need to accses Internet will use a proxy server with the same no exe/bat/js/msi/etc file access
- now html viwer in webmail
- block as many countries as you can using dns or a smart firewall

And the most effective result is to teach your own clients and maybe to make with them some quitz like" who is able to identify one spam / malware virus from 100 good emails?" Go forward and motivate your user, like think that you have relatives(children, parents, and so on), and you can help them if you are willing to help them.

This ideas can be used with others better tools, but in the end a educated user is the best antivirus shield if you are willing to to spent time and to talk with them. Another side note is the fact that woman's are most carefully about this if you want to tech them.... or mybe I have a lot of luck with them

 

[Stoiko Ivanov]

To bad that avast changed the path to the executable from "/bin/scan" to "/usr/bin/scan". Proxmox Team needs to adjust the path to the executable here:

The patch got applied - version 5.2-5 should contain the fix once it gets released.

Thanks @heutger for the reminder on the mailing-list!

 

[Thomas k.]

The patch got applied - version 5.2-5 should contain the fix once it gets released.

Thanks @heutger for the reminder on the mailing-list!

Thanks Stoiko, thanks heutger!

Thomas

 

[heutger]

Hi,

In my own opinion antivirus with signatures, as a technology is end of life. Now is almost impossible for most cases to block zero day virus. It os so easy to create new viruses with encrypted payload... as I read. And if you start to dig on Internet maybe you will see that more increses of new viruses. More new viruses / year for my point of view must need more human hour to deliver a antidot => higher prices for many antivirus tools.

But as I find in my own case (I do not want to say that this could be usable for any envitoment) I can live without any antivirus tools using this simple ideeas (it work for me for amost 7 years with only clamav, but clamav has seen let say only 1 maybe 2 viruses / year)


- no exe/bat/msi/js/etc for any mail
- only webmail and not a windows mail client
- any client who need to accses Internet will use a proxy server with the same no exe/bat/js/msi/etc file access
- now html viwer in webmail
- block as many countries as you can using dns or a smart firewall

And the most effective result is to teach your own clients and maybe to make with them some quitz like" who is able to identify one spam / malware virus from 100 good emails?" Go forward and motivate your user, like think that you have relatives(children, parents, and so on), and you can help them if you are willing to help them.

This ideas can be used with others better tools, but in the end a educated user is the best antivirus shield if you are willing to to spent time and to talk with them. Another side note is the fact that woman's are most carefully about this if you want to tech them.... or mybe I have a lot of luck with them

Sure, don't rely on what your antivirus tool says about a suspicious attachment. However, modern viruses use the workaround of being not restricted to exe etc. but also word, excel, powerpoint, pdf or zip files and you can't restrict all of them. Also phishing or just browsing the internet with an insecure browser is already a touchpoint. So layer 8 (and user awareness) is the biggest security control, however, it's not working for all. Same for clientless doesn't work for all. So it's a good idea to have something to prevent from the mainstream malware, also having multiple solutions (on multiple touchpoint like Gateway, Client, Mail and other Servers) is a good idea, but for sure, you shouldn't rely on too much. So that's why I also tried to start with ClamAV and then just looked for another affordable solution, as it's just an extra.

 

[heutger]

Here comes something new!

As @AdamP asked in another thread about adjustments and mentioned over products, I also looked myself once again on how to optimize my whole setup by also reducing the amount of used systems. At this point I came over Warden AntiSpam and AntiVirus for Plesk and it's somehow similar to Proxmox, maybe @Stoiko Ivanov it would be a great idea to integrate PMG as well into Plesk as PMG looks more clean, professional and somehow nicer (e.g. a bit like MacOS vs. Windows^^). However, it came with some plugins and settings, I checked against my config and finally I decide to activate one more check: Phishing

For updating the feeds you need to signup with PhishTank and register an application (PMG is in use already ^^).

Then the following adjustments need to be done:

vi /etc/mail/spamassassin/v342.pre
vi /etc/mail/spamassassin/custom.cf
vi /etc/cron.hourly/phishing
chmod +x /etc/cron.hourly/phishing
/etc/cron.hourly/phishing

/etc/mail/spamassassin/v342.pre (uncommenting Phishing):

# This is the right place to customize your installation of SpamAssassin.
#
# See 'perldoc Mail::SpamAssassin::Conf' for details of what can be
# tweaked.
#
# This file was installed during the installation of SpamAssassin 3.4.1,
# and contains plugin loading commands for the new plugins added in that
# release.  It will not be overwritten during future SpamAssassin installs,
# so you can modify it to enable some disabled-by-default plugins below,
# if you so wish.
#
# There are now multiple files read to enable plugins in the
# /etc/mail/spamassassin directory; previously only one, "init.pre" was
# read.  Now both "init.pre", "v310.pre", and any other files ending in
# ".pre" will be read.  As future releases are made, new plugins will be
# added to new files, named according to the release they're added in.
###########################################################################

# HashBL - Use EBL email blocklist
loadplugin Mail::SpamAssassin::Plugin::HashBL

# ResourceLimits - assure your spamd child processes
# do not exceed specified CPU or memory limit
# loadplugin Mail::SpamAssassin::Plugin::ResourceLimits


# FromNameSpoof - help stop spam that tries to spoof other domains using
# the from name
# loadplugin Mail::SpamAssassin::Plugin::FromNameSpoof

# Phishing - finds uris used in phishing campaigns detected by
# OpenPhish or PhishTank feeds.
loadplugin Mail::SpamAssassin::Plugin::Phishing

# allow URI rules to look at DKIM headers if they exist
parse_dkim_uris 1

Adding the following the lines to the bottom of /etc/mail/spamassassin/custom.cf:

ifplugin Mail::SpamAssassin::Plugin::Phishing
phishing_openphish_feed /etc/mail/spamassassin/openphish-feed.txt
phishing_phishtank_feed /etc/mail/spamassassin/phishtank-feed.csv
body URI_PHISHING eval:check_phishing()
describe URI_PHISHING Url match phishing in feed
score URI_PHISHING 1.4
endif

/etc/cron.hourly/phishing (replace xxx through your application key):

#!/bin/sh

wget -O /etc/mail/spamassassin/openphish-feed.txt -q https://openphish.com/feed.txt
wget -O /etc/mail/spamassassin/phishtank-feed.csv.gz -q http://data.phishtank.com/data/xxx/online-valid.csv.gz
gunzip -f /etc/mail/spamassassin/phishtank-feed.csv.gz

 

[guletz]

Sure, don't rely on what your antivirus tool says about a suspicious attachment. However, modern viruses use the workaround of being not restricted to exe etc. but also word, excel, powerpoint, pdf or zip files and you can't restrict all of them. Also phishing or just browsing the internet with an insecure browser is already a touchpoint. So layer 8 (and user awareness) is the biggest security control, however, it's not working for all. Same for clientless doesn't work for all. So it's a good idea to have something to prevent from the mainstream malware, also having multiple solutions (on multiple touchpoint like Gateway, Client, Mail and other Servers) is a good idea, but for sure, you shouldn't rely on too much. So that's why I also tried to start with ClamAV and then just looked for another affordable solution, as it's just an extra.

Good points!

 

[heutger]

I did it again, something new (and I will do one more thing the next days, playing around a bit with fail2ban):

I just try again SPF on my private installation (on commercial installations I strongly recommend not(!) to use SPF) after getting one spam, which SPF would have been able to catch and got through my content filter as well (was a phishing try, so I also therefor added the above phishing feeds). Since this change, I got just one SPF catch in my logs meanwhile I saw on commercial that legit mails would have been catched (because of mailing lists, the reason why SPF is broken by design). You can check by yourself with enabled SPF by filtering with openspf or with disabled SPF by filtering with spf_fail - be surprised by the matches - and much more interesting with spf_softfail, which, if the sender would change from softfail to fail will all be rejected!

However, by playing around with SPF, I recognized, that there are not only SPF checks for the sender address domain performed but also SPF checks for the HELO domain. Funny thing is that Proxmox themselves seem to promote SPF very much (as should be enabled and be the successor to greylisting activation), but their own forum mails have no SPF HELO record and their own development mailing list has no SPF record at all. I also recognized, that some SPF checkers (and maybe some real implementations) don't well recognize the + signs for positive settings like +a or +mx, so I removed (also in my post above) the + signs for positive settings and just kept the soft fail tilde as extra sign.

Additional I now added records for my sending mail server HELO names (unfortunately most of my mail is been sent via my hosted exchange account, so most mail will still have no existing SPF HELO record) like this:

cs                       IN TXT     "v=spf1 a ip4:194.37.255.0/24 ip4:91.198.224.0/24 ~all"
mg                       IN TXT     "v=spf1 a a:heutger.net ip4:194.37.255.0/24 ip4:91.198.224.0/24 ~all"

You may wonder, why I didn't just set v=spf1 a -all and honestly it was my first thought to do so. But you need to reconsider, that you also send mails @ your HELO domains, e.g. PMG statistics been sent from the PMG hostname, so I need to adjust my records for mails coming from this particular subdomain.

So my mail gateway need to be able to send mails to my plesk server (which is cs) and this one need to be able to send mails through my cyren and hornet setup. As mails are ending at my hosted exchange account, no additional record is required. To take sure, that changes in the siren or hornet infrastructure without further notice don't result in my mails getting rejected, I also had chosen soft fail instead of hard fail, although for such mails the delivery path is well known.

Errata: include:antispameurope.com is not required at QualityHosting

 

[heutger]

Tadaa! Here comes something new again!

It's no spam protection itself, it's just to cleanup my logs a bit and maybe some spammers will see, that they are unwanted.

Therefor I installed fail2ban just for two typical annoying issues:

1. hangups via postscreen on servers, which are listed on many RBL or are rejected by postscreen for other reasons (so there is also no need always to recheck this ones on the RBL again and again, may also help with rate limits)
2. milter-rejected messages, which reach my miltered spamassassin score, it's untypical, if not really spam, that such messages will occur more than once, if it's no real spam and just a false-positive

Because of this really rare issues I also decide to be hard on banning, if they occur twice in a period of 24 hours, they will be rejected for 48 hours. I will now check, how my setup will work, funny thing is, that I saw two IPs already on both installations, so seems to be an usual "bad boy".

Here the steps performed and settings need to be done (last three ones you can use to check the status of fail2ban and the jails):

apt-get install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
vi /etc/fail2ban/filter.d/postfix-hangup.conf
vi /etc/fail2ban/filter.d/postfix-milter-reject.conf
vi /etc/fail2ban/jail.local
systemctl restart fail2ban
fail2ban-client status
fail2ban-client status postfix-hangup
fail2ban-client status postfix-milter-reject

/etc/fail2ban/filter.d/postfix-hangup.conf:

[Definition]

failregex = postscreen\[\d+\]: HANGUP .* from \[<HOST>\]:\d+

ignoreregex =

/etc/fail2ban/filter.d/postfix-milter-reject.conf:

[Definition]

failregex = milter-reject: END-OF-MESSAGE from .*\[<HOST>\]: 5.7.1 Blocked by SpamAssassin

ignoreregex =

/etc/fail2ban/jail.local (append this one to the end of the copied file):

[postfix-hangup]
enabled = true
port = smtp
filter = postfix-hangup
action = iptables[name=postfix-hangup, port=smtp, protocol=tcp]
logpath = /var/log/mail.log
bantime = 172800
findtime = 86400
maxretry = 2

[postfix-milter-reject]
enabled = true
port = smtp
filter = postfix-milter-reject
action = iptables[name=postfix-milter-reject, port=smtp, protocol=tcp]
logpath = /var/log/mail.log
bantime = 172800
findtime = 86400
maxretry = 2

 

[heutger]

And one more update:

You can decide, if you like to adopt my adjustments or not, but for me it's annoying to see always the same records in tracking center (if checking for quality of e.g. milter-reject) as well as giving any system performance to bad guys. After seeing .icu top level domain seems to have no legit customers, just sending spam over and over, I decided to reject .icu already on connection level (already considered to fail2ban them also, but they seem to use spam server farms, so I would need to reject whole networks, which I currently don't want to do). Googling for .icu spam, there are many records, also on this forum.

So here is what to do:

vi /etc/postfix/reject_tld
postmap /etc/postfix/reject_tld
vi /etc/pmg/templates/main.cf.in
pmgconfig sync --restart 1

/etc/postfix/reject_tld:

/\.icu$/ REJECT We reject all .icu domains

 

[heutger]

Something new ... again ;-)

I'm currently playing around a bit to optimize my setups. On the catch with 100% scores (minimum, at Mozilla Observatory I was able to catch 125%^^) and getting more and more green on Hardenize (I will post my experiences on my private blog once I have spare time, but I may prio to create my Advancing PMG repository first) I just considered to activate DKIM and together with SPF DMARC as well. It also came similar to my first steps with MTA-STS to improve my setup, that I read about it (again).

My personal opinion on DKIM (and DMARC) is somehow the same as SPF. It's broken by design (and I really don't understand, why the RFC authors make failures again and again) and ARC seems to try to patch the failures again similar as SRS tries to do. As recently also attending IETF meetings, also the promotion of DMARC is deceptive (similar to DANE). DKIM and DMARC doesn't help anything against spam. We all know the typical viagra, you won a million, I will send you another as well as hi my dear, want to get in love with you mails and they come from free mailers with valid SPF and DKIM, we all know the typical purchase this or that attendee or user list, they come from well setup (with valid SPF and DKIM) mail systems, so it's no sign for spam or no spam. Same for identity validation, which is done by DKIM and DMARC, for sure, you will know, that the sender is the real sender and for worse made phishing that's great, but if someone purchase a similar mail domain and using DKIM and DMARC, the signature will be valid again. However, S/MIME and PGP are still not widely spread, PGP always requires some additional or special tools, Web of Trust is also broken with the new key servers, and S/MIME still cost a few dollars for commercial usage, so many users are still not willing to use it, although they help for privacy (GDPR) or confidentiality (Trade Secrets Directive and others).

However, back on how I implemented. I currently run two setups of DKIM, one with my hosted exchange at QualityHosting, so I was just required by them to set a CNAME record, that the Hornet Security solution at them could be enabled for DKIM. DKIM allows to have parallel DKIM setups running, e.g. on branch offices, e.g. on periodical rotations (monthly, yearly, ...), e.g. on servers (mx1, mx2, ...), e.g. on additional services (zen desk, mail chimp, ...), they call it selectors. So my first step was really easy, if you're at QualityHosting, you just need to set the following record and ask them to activate DKIM signing:

hse._domainkey           IN CNAME   hse._domainkey.hornetsecurity.com.

But I also want to enable DKIM on my Plesk installation for all outgoing mails from my Plesk server. So I disabled the local DNS zones as I use an external DNS server, enabled DKIM in my mail server settings and in the mail settings for my domain. I then extracted the public key via command line on the plesk server:

openssl rsa -in /etc/domainkeys/heutger.net/default -pubout

and added it to my DKIM record for the plesk selector default:

default._domainkey       IN TXT     "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQD5swJ11LaZrNz02ebzlyblOthx7IafmMS628ElP70Ht3NlamqLR0XXId3S9yLxwRIHBxAubG2Jxg5S6Xs5J7gRJT2EuNFST7lXHOfHWFpoIHdoAT84bCGWdp0+vRTq7NixwtDXF/FHKGy6JrDA9m17N/5vCKtjTTxaaexXyeGKqwIDAQAB"

Last step was to setup my DMARC policy. I set up a free account with Postmark for the XML aggregated report to be processed for better interpretation (because of privacy don't use external services for detailed reports) at https://dmarc.postmarkapp.com and added the following to my DNS records:

_dmarc                   IN TXT     "v=DMARC1; p=quarantine; rua=mailto:xxx@dmarc.postmarkapp.com"

I then tested my setup with my other PMG gateway as well via https://www.port25.com/authentication-checker/

 

[heutger]

Funny fact about DKIM/DMARC: I really don’t understand, why it hasn’t been thought about integrating a third party to validate information, that would make it much better. However, I found about BIMI, which seems to fill this gap. Will play around with the next days. However, I also recognized another funny and worse fact: It looks like all the direct available implementations for me (Plesk, Hornet Security) use just a 1048 bit key and a SHA1 hash. That’s worse as both are well known to be unsecure. Anyone else already implemented DKIM/DMARC? Which hash algorithms/key sizes do you use?

Addendum 1: I just found, that 4096 bit keys will arise problems with DNS as they won't fit in UDP responses. Also there is no requirement for rekeying, it could be done, or not. All this facts emphasize, that DKIM and then also DMARC is broken by design. I'm planning also to play around with DNSSEC and DANE now, however, all of this is just for proof of concept and getting up to 100% score but I don't see any value in this features with all their drawbacks.

Addendum 2: I just found, that RFC 8463 also add EC support to DKIM, so it would be possible to use more secure keys that way without the limitation to DNS TCP responses. However, once if quant cryptographic devices exist with limitations, they won't differ between EC or primes, so the key length is then important again and it's then easier to break a small key size EC than a long key size prime, so I still prefer RSA over EC. However, there is still lack of implementation of DKIM signing solutions who support EC (which I have access to).

 

[DerDanilo]

@heutger
Would you be so kind an create full and complete, ordered guide on Github/Gitlab, your own blog or the Proxmox Wiki?
Also explaining what each option is good for?
Reading this whole thread is a mess and confusing as it's not possible to see what the latest state of usable configuration is.

I am happy to contribute if I should find the time to do so. Currently I am overloaded with work.
Maybe it's also possible to somewhat ansiblablize the whole adjustments to PMG, so others can apply them more easy.

Found this one so far:
https://www.heutger.net/proxmox-mail-gateway-mit-rspamd/

Thanks in advance!

 

[heutger]

@heutger
Would you be so kind an create full and complete, ordered guide on Github/Gitlab, your own blog or the Proxmox Wiki?
Also explaining what each option is good for?
Reading this whole thread is a mess and confusing as it's not possible to see what the latest state of usable configuration is.

I am happy to contribute if I should find the time to do so. Currently I am overloaded with work.
Maybe it's also possible to somewhat ansiblablize the whole adjustments to PMG, so others can apply them more easy.

Found this one so far:
https://www.heutger.net/proxmox-mail-gateway-mit-rspamd/

Thanks in advance!

@DerDanilo: Here are my plans:

I currently still test beside a few adjustments (BIMI, From Spoof Check, DNSSEC and DANE are on my roadmap, last one I just wait to be able to move my domains to a DNSSEC enabled environment) and discuss with some test sites about some optimization options like internet.nl, but once back from teaching ISO 27001 Foundation and Officer classes, my next plan is to set up our commercial productive installation with my IT stuff and meanwhile explaining them my adjustments I will set up a virtual machine with my adjustments as well.

From this virtual machine I will take all adjusted files and steps to do and will set them up on my Github repository, I already created. Once I did that, help is welcome on scripting to adjust that files based on the setup of someone (e.g. license keys, IDs, private and public keys, ...) need to be generated per installation. Scripting is nothing I would like to perform or provide.

For sure, I will then also document there everything I did.

As you could see in the article you mentioned above, my blog is not the best place to document my adjustments. I also got told, it's much easier, if I could provide files instead of code segments of adjustments, as it's required to find them in the files, performing a diff may be easier. I will just reconsider, if a wiki would be a good idea, but it would not provide possibility to discuss, ask questions or get support, so I believe Github is the best idea.

So stay tuned, will now come very soon as this thread is really a bit confusing now.

The base setup is done, next week I will start on: https://github.com/heutger/advancing-pmg

 

[ishan]

sa-update --nogpg --channel sa.schaal-it.net

Seems to be having issues -

channel: SHA512 verification failed, channel failed

 

[heutger]

sa-update --nogpg --channel sa.schaal-it.net

Seems to be having issues -

channel: SHA512 verification failed, channel failed

Sorry, but I can't confirm. I just tried by myself without errors. There may be other errors on your side. How about the other channels? Did you try via shell by yourself?

 

[heutger]

Funny fact about DKIM/DMARC: I really don’t understand, why it hasn’t been thought about integrating a third party to validate information, that would make it much better. However, I found about BIMI, which seems to fill this gap. Will play around with the next days. However, I also recognized another funny and worse fact: It looks like all the direct available implementations for me (Plesk, Hornet Security) use just a 1048 bit key and a SHA1 hash. That’s worse as both are well known to be unsecure. Anyone else already implemented DKIM/DMARC? Which hash algorithms/key sizes do you use?

I just found, that 4096 bit keys will arise problems with DNS as they won't fit in UDP responses. Also there is no requirement for rekeying, it could be done, or not. All this facts emphasize, that DKIM and then also DMARC is broken by design. I'm planning also to play around with DNSSEC and DANE now, however, all of this is just for proof of concept and getting up to 100% score but I don't see any value in this features with all their drawbacks.

I'm sorry, but I need to followup on this:

I got my report today and it confirmed the weaknesses of SPF and DKIM. Proxmox Development mailing list as well as Google Groups and some others fail with my SPF and DKIM settings and are complaint by the report. So SPF and DKIM and DMARC at all is nothing to fight against spam or increase identity or reputation with, it's just for the domain owners a good opportunity to identify, there their domain is been used or especially misused, maybe it's possible to fight against misuse from that side, but that's all. All what got told or promoted about DMARC beside that is not reality. As beside that report I got a mail, well signed and with valid SPF records via Google Mail containing a spam mail with an open 400+ recipients list.