Bad Spam recognition (compared to other solutions)

4920441

New Member
Dec 7, 2021
7
0
1
52
Hi,

I deployed two community editions of the proxmox mail gateway and though I tried to train spamassassin with severel gigabytes of SPAM (-> sa learn) and added the following DNSBL Site (dul.dnsbl.sorbs.net,ix.dnsbl.manitu.net,zen.spamhaus.org,bl.spamcop.net,b.barracudacentral.org)

But nevertheless the spam recogniton rate is rather low.

I get really easy to recognize spam e-mails on a daily basis.

What is the best practice to improve the recognition (way) more?

Thanks alot!

Cheers

4920441
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,690
992
163

itNGO

Active Member
Jun 12, 2020
386
78
28
43
Germany
it-ngo.com
Hi,

I deployed two community editions of the proxmox mail gateway and though I tried to train spamassassin with severel gigabytes of SPAM (-> sa learn) and added the following DNSBL Site (dul.dnsbl.sorbs.net,ix.dnsbl.manitu.net,zen.spamhaus.org,bl.spamcop.net,b.barracudacentral.org)

But nevertheless the spam recogniton rate is rather low.

I get really easy to recognize spam e-mails on a daily basis.

What is the best practice to improve the recognition (way) more?

Thanks alot!

Cheers

4920441
Hi,

in this case I am sorry, but I confirm, detection rate compared to Sophos UTM or Symantec Brightmail Gateway is really worse....

We used to have multiple rows in Spam-Detection. In front we use Sophos UTMs and until 6 Months ago we had Symantec Brighmail Gateway as second level detection. This works quite good, I would say around 99,99% detection rate overall after some configuration.

We replaced the SMG with PMG and detection rate got really worse for the second stage. I would say 7 of 10 Mails which pass the UTM at first level are even not detected on second stage in PMG. And many, many manual Blacklisting and Whitelisting is necessary now.

We continue to use PMG as we have clearly committed to open source wherever possible. But we really hope this is getting better in a future PMG-Update. Currently it is not really comparable to paid solutions from other vendors at all....
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,690
992
163
We continue to use PMG as we have clearly committed to open source wherever possible. But we really hope this is getting better in a future PMG-Update. Currently it is not really comparable to paid solutions from other vendors at all....
again - please share some logs - else it's really not possible to see if there is anything which might be improved with your current setup.

else PMG uses quite well-proven technologies for detecting spam (mostly SpamAssassin), which do work acceptably in most situations (we run it here and spam rarely passes through - but that of course might be specific to us (many open-source mailing lists - rather few uses of our email-addresses outside of work-context)

also - have you checked and implemented the recommendations from the getting started page?
https://pmg.proxmox.com/wiki/index.php/Getting_started_with_Proxmox_Mail_Gateway
 

4920441

New Member
Dec 7, 2021
7
0
1
52
Hi,

yes I read the getitng started page several times and thourouhly.

Also I put a couple of extra RBLs in the config. BTW: If I weigh the RBLs a bit, it works way better than without precedence:

Code:
zen.spamhaus.org*3 bl.mailspike.net*3 b.barracudacentral.org*2 ix.dnsbl.manitu.net

Also I learned Spam Assisn with hundreds of confirmes SPam E-Mails,

but still, I have a dozend mails which are going right trough, despite they should be able to be easiely detected

I search for some logs to present it to you.

Especiallly easy to guess E-Mail addresses like "info" or "sales" are way more prone to spam than any other individualised email addresses, maybe the spam load is way highert so thats the reason.


Cheers

4920441
 

ldsam

New Member
Feb 15, 2022
5
0
1
42
Any news on this one?
We have pretty much the same problem.

We already added a few RBL and weighted them but there is tons of very obvious spam still coming throu.

What kind of logfiles should we provide for efficient debugging?
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,690
992
163
What kind of logfiles should we provide for efficient debugging?
the mail.log - or the text for such a misclassified mail from the Tracking Center for starters
additionally maybe also share your current settings regardng DNSBLs and if you have any modifications to the default configs
 

poetry

Member
May 28, 2020
92
17
8
Same problem here. I have used a lot of my time to try and fine tune proxmox for our mailflow and still can't get detection rate to be better. You can also test your proxmox with https://emailsecuritytester.com/ and see what score you will get. Yes you can say they tune this test for their solution but why would you not implement some things into proxmox as well. It should be better at detecting malicious messages.

Some of my old posts here should update my setup I have advance it quite a bit still not good enough
https://forum.proxmox.com/threads/s...mx-filter-in-reply-to-field.80037/post-354681

The worst thing is there is no deep detection of links or files. We even purchased securiteinfo.com it helps a bit but still not where it should be. I have a lot of high custom scores defined for spam assasin again it's a never ending story of changing the weights of the system it will never be good enough. If the scores are too high there will be a lot of false positives if the scores are too low there will be too much spam. It's hard to know the quality of spam assasin score that you can increase by a lot without getting a lot of false positives.

I have already increased some of the scores for some detection's from defaults but it's a never ending story of tuning the weights of the system...

Example two malicious messages from today:

connect from un.unrepeatedshow.com[194.41.47.92]

They are now on some blocklists but they have not been before when we got the message. Having more DNSBL is not a solution so many false positives if you do that...
I am only using b.barracudacentral.org;zen.spamhaus.org;bl.mailspike.net;dnsbl.sorbs.net;bl.spamcop.net and I won't be using any more I have found that a lot of DNSBL are really poor quality.

https://www.abuseipdb.com/check/194.41.47.92
https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a81.88.48.54

The link if you click on it it's different

https://mail.oldcoinbuy.com/g00h/index.php?

Yes it's Phishing
https://www.virustotal.com/gui/url/...1810518498bea97b7227e89d1dece693603?nocache=1

spam1.png


First one:
X-SPAM-LEVEL: Spam detection results: 3
DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid
DKIM_VALID -0.5 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain
GOOG_MALWARE_DNLD 1 File download via Google - Malware?
HTML_MESSAGE 0.001 HTML included in message
HTTPS_HTTP_MISMATCH 3 -
KAM_GOOGLE_REDIR 0.5 Message contains a google URL redirector link
MIME_HTML_ONLY 0.1 Message only has text/html MIME parts
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.5 SPF: sender matches SPF record
T_SCC_BODY_TEXT_LINE -0.01 -

Second one:
connect from authsmtp04.register.it[81.88.48.54]

https://www.abuseipdb.com/check/81.88.48.54
https://mxtoolbox.com/SuperTool.aspx?action=blacklist:81.88.48.54&run=toolpage

They are not on any blocklist.

The link will download malicious file:

https://sibuceomexico.com.mx/rqcuatonuetseis/lua-oarse-outqatqttmommhuvi

https://www.virustotal.com/gui/file...17d9e434f1d2489f1cca38d74c80279734a?nocache=1

spam2.png


X-SPAM-LEVEL: Spam detection results: 0
HTML_MESSAGE 0.001 HTML included in message
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust
RCVD_IN_MSPIKE_H3 0.001 Good reputation (+3)
RCVD_IN_MSPIKE_WL 0.001 Mailspike good senders
SPF_HELO_PASS -0.001 SPF: HELO matches SPF record
SPF_PASS -0.5 SPF: sender matches SPF record
T_SCC_BODY_TEXT_LINE -0.01 -

How are you going to tune this? It's very hard or just impossible to detect this.

The servers are fresh not on any blocklist yet so good luck blocking this messages with proxmox.

Once the message is delivered it's too late the damage is already done. I don't have a way and don't want to scrub this malicious messages from all our mail systems it's not sustainable.
 
Last edited:

BJ78945

Active Member
Apr 15, 2015
75
12
28
Hi,

I've changed a lot of things in postfix and spamassassin config to fit my my needs.

First of all I bought some commercial services like Abusix, Spamhaus, securiteinfo, etc.

Spamhaus also have a spamassassin plugin which checks full hostname in the links. Abusix has some blacklists for url shorter and file hosting. But there is no plugin for spamassassin. (I will try to find a solution in near future)

Most mails will be blocked at MTA level:

Code:
smtpd_sender_restrictions =
        …
        reject_rhsbl_sender
        reject_rhsbl_helo
        reject_rhsbl_reverse_client
        permit_dnswl_client
        reject_rbl_client
        ...

It will check IP, Reverse, Hostnames, Domains etc. also in PMG WebGUI I have checked reject unknown clients, reject unknown sender, SMTP HELO Tests, SPF. Sadly the tracking center doesn't show logs from the configured reject commands above and also there is no statistics. But you can search in the /var/log/mail.info if you have to finde why some mails will not arrived. There you will be see if there was an connection which was blocked by an blacklist above.

In spamassassin I added the spamhaus, dcc, pyzor module and a lot of blacklist checks to custom.cf.

I also setup an own dnsbl for domains and full hostnames inside body part to rapidly block new mails. I wrote a software where users can report spam and greylist / blacklist the reported uris globally. But this helps only if someone reports bad mails.

I also changed some code to reject mails which goes to quarantine, so the sender server don't think the mail was correctly delivered to the user.

My servers receives around 25k mails a day. I can only speak for my mailbox. But this is good cleared. Sometimes I receives a really good phishing mail but I don't thing you can prevent it. Before I used PMG I was using an expensive and very good antispam filter which also din't blocked everything. Sometimes I received phishing there too. Some other users reports, that some mails are false positive. This hits are often because they contain spam loved domains like wetransfer or other services.

I also included checks for new registered domains and give some score to them.

Really good people hacks systems with good reputation and sends really good phishing mails to you. This is really hard to block. One person who wrote mails to me was hacked and the bad man sends mails to me in reply of existing communications. Only the links in the mails were serious so I thought it was phishing. But I also think that the most users will click on such a link. Also I saw, that sometimes complete Wikipedia articles were include in phishing mails to become better spamassassin score because of whitelisted domains in body.

I checked the link from your mail (sib...xico.com.mx) against a lot of commercial backlist. But there is no entry for this. So it seems that no spam trap and nobody else reported this domain at the moment.

Attachments are easy to scan or block but I have no good idea how to prevent users from links which were not send to a lot of other peoples. No blacklist, hashlist, etc. could know this bad link from an hacked mail address with good reputation.

I also tried with Bayes. I setup a cluster database to share the informations between the hosts. But also this will not recognize phishing based on real communication in every case.

Here I shared some things I've done (in German) https://forum.proxmox.com/threads/o...racking-center-im-cluster.103222/#post-444375 But in the last month I did much more. In future I will update the posts there.

Perhaps other users have better solution to catch new phishing domains or other ideas what to do?
 
Last edited:

poetry

Member
May 28, 2020
92
17
8
Have some examples every day no way to block them. Can't adjust much custom scores to block...

Here another example send from Microsoft servers so I can't use DNSBL as I will also block legitimate mail...

connect from mail-mw2nam08hn2217.outbound.protection.outlook.com[52.100.162.217]
https://www.abuseipdb.com/check/52.100.162.217
https://mxtoolbox.com/SuperTool.aspx?action=blacklist:52.100.162.217&run=toolpage

The link is Phishing/Malicious

https://www.virustotal.com/gui/url/...47394b5fba285da2ed6778d3b62987410e8?nocache=1




https://bit.ly/3H6xAgu
spam.png

How to block this? Already adjusted some scores as you can see but can't do much more than that...

X-SPAM-LEVEL: Spam detection results: 0
DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid
DKIM_VALID -0.5 Message has at least one valid DKIM or DK signature
HTML_MESSAGE 0.001 HTML included in message
KAM_BLANKSUBJECT 0.25 Message has a blank Subject
KAM_SHORT 0.75 Use of a URL Shortener for very short URL
RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust
RCVD_IN_MSPIKE_H2 -0.001 Average reputation (+2)
SPF_HELO_PASS -0.001 SPF: HELO matches SPF record
SPF_PASS -0.5 SPF: sender matches SPF record
TVD_SPACE_RATIO 0.001 -
T_SCC_BODY_TEXT_LINE -0.01 -
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,690
992
163
The link is Phishing/Malicious
while the link is most likely malicious - virus total only has 2 out of its many engines which consider it malicious - and this is the problem.,
for a link to be flagged as malicious you need a few pointers to those things - by that time I guess it would also be listed in uribl - which is check by SpamAssassin.

Sadly I have no solid recommendation how to find "fresh" spam/phishing links right after you first see them - unless they've been listed

Most solutions people come up with regarding those usually cause far more false positives - or break what people expect from e-mail (e.g. being able to send a mail to every person)
 

poetry

Member
May 28, 2020
92
17
8
@Stoiko Ivanov this is a very hard problem to solve I understand that. I hope there is something that can be improve with deep link detection in proxmox.

Just had another example looks like completely targeted phishing it was send to our info address. This is really bad and I have no way of blocking it efficiently. It had our logo and our email autocomplete in the phishing link. I change the link a bit to not expose our company information but it still works.

from server.solarwiz.net[160.20.145.67]
https://www.abuseipdb.com/check/160.20.145.67
https://mxtoolbox.com/SuperTool.aspx?action=blacklist:160.20.145.67&run=toolpage

It was send from IP that is only on UCEPROTECTL3 DNSBL blacklist that is really bad quality.

Can't increase any spam score as they are all valid.
X-SPAM-LEVEL: Spam detection results: 0
DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid
DKIM_VALID -0.5 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain
HTML_MESSAGE 0.001 HTML included in message
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.5 SPF: sender matches SPF record
T_SCC_BODY_TEXT_LINE -0.01 -

Return-Path: ladio@solarwiz.net

https://669502.selcdn.ru/ow327BFC55...317&e=c5cec25e50BgVtWAzsQmMcaL3zBjzlZ2TJR

Not enough detection for @Stoiko Ivanov ?

https://www.virustotal.com/gui/url/...c9f4f1bf812f7549851fd251d7dd123e987?nocache=1
phishing1.png



phishing.png
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!