[TUTORIAL] Advancing Proxmox Mail Gateway (especially Spam and Virus Detection)

Discussion in 'Mail Gateway: Installation and configuration' started by heutger, May 29, 2018.

  1. DerDanilo

    DerDanilo Member
    Proxmox VE Subscriber

    Joined:
    Jan 21, 2017
    Messages:
    248
    Likes Received:
    17
    Let's ansibleize your advancements and put them into different ansible roles, so they are available to more people without the fear of doing something wrong.
     
  2. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    Will send you PM ;-)
     
  3. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    @heutger I was wondering if you were getting
    Code:
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: yyerror(): /var/lib/clamav/antidebug_antivm.yar line 497 undefined identifier "pe"
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: yyerror(): /var/lib/clamav/antidebug_antivm.yar line 512 undefined identifier "pe"
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: yyerror(): /var/lib/clamav/antidebug_antivm.yar line 528 undefined identifier "pe"
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: yyerror(): /var/lib/clamav/antidebug_antivm.yar line 544 undefined identifier "pe"
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: yyerror(): /var/lib/clamav/antidebug_antivm.yar line 557 undefined identifier "pe"
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: yyerror(): /var/lib/clamav/antidebug_antivm.yar line 603 undefined identifier "pe"
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: yyerror(): /var/lib/clamav/antidebug_antivm.yar line 614 undefined identifier "pe"
    Nov 10 00:25:20 pmg clamd[778]: LibClamAV Error: cli_loadyara: failed to parse rules file /var/lib/clamav/antidebug_antivm.yar, error count 7
    recently check the logs and happens when clavmav scans
     
  4. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    Hi,

    as of the ClamAV update in the past, I posted here about, which did not ignore yararules failures any more, I‘m not sure, which solution you applied. One just fixed some of the failures, the other was to disable and remove yararules at all. I‘d choosen the last option to prevent continous bugfixing. As of that, I can’t confirm, my logs look fine, may be you should disable also the yararules at all. I gave a link on both solutions in my previous post about the ClamAV update.

    Regards,
    Christian
     
    killmasta93 likes this.
  5. ishan

    ishan New Member

    Joined:
    Nov 12, 2018
    Messages:
    6
    Likes Received:
    1
    Excellent thread, a lot of things to learn from it.
     
    heutger likes this.
  6. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    Just a very small add-on. I currently use the postfix header checks on my commercial test for some aggressive spammers. For sure, I believe, that once I can train bayes via the centralized spam and ham boxes, detection will get better and I can milter-reject some more mails, but sometimes I would prefer something like Scrollout F1 has, that you can run your own blacklist by catching aggressive spam. So what I mean are the typical "purchase this attendee or user list" spams. For sure, I could also block Gmail at all as most of such spam is coming from Google Mail Servers (valid signed with DKIM, coming from valid SPF servers, so all new techniques promoted by Google don't help, as if Google is then sending the spam themselves, as they seem to be lame on outgoing spam detection meanwhile propagating always new technologies and how well they perform for Google), but I'm afraid, that's no got idea. So here is my current postfix header_checks file (as adjusted from above):

    Code:
    /^From:/ INFO
    /^To:/ INFO
    /^Subject:.*medtrade.*/ REJECT
    /^Subject:.*user.*list.*/ REJECT
    /^Subject:/ INFO
     
    killmasta93 likes this.
  7. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    I still play around with additional blacklists and also wrote elsewhere in this forum on how I'm performing this, so refer to Relay Blacklist Optimization to check more out of my tests. How I'm performing:

    I add to main.cf.in additional blacklists with warn_if_reject and then reject_rbl_client or reject_rhsbl_client for domain name blacklists and then check my logs (via tracker) for some time, usually minimum a week, up to one month. That's also why I already asked for and adjust as written above the postfix to perform header checks to add the sender, subject and recipient information given in the e-mail which may differ from the technical ones, how the mail is coming in. So called ESPs use the sender for detecting bounces and therefor don't give the later shown sender (e.g. in the mail client) but use a technical address. But if checking e.g. Amazon SES addresses for being rejected legit or not, it's hard if not being able to see the real sender, Amazon SES is used for. Same for Mailchimp and many others.
     
    killmasta93 likes this.
  8. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    Quick question for the SMTP milter reject could apply for the blacklist? lets say i block a domain and if that domain sends an email instead of dropping it would reject telling them that they are blocked?
     
  9. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    No, SMTP milter only works for the SpamAssassin part, so the ruleset of PMG is not integrated here. I would prefer, if Proxmox would overthink for the next release, how they integrated their ruleset, SpamAssassin and ClamAV virus scan and replace the current content filter way by a milter way. For the time being, SMTP milter can only reject (if you like and set this level) on reaching a particular score by SpamAssassin checks and always then state to the sender, that it's content blocked by SpamAssassin. If you want to reject because of a "blacklist", you may do that manually via postfix.
     
    killmasta93 likes this.
  10. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    Thanks for the reply, dully noted. Also was implementing the EBL on msbl.org just got a cron error which was this

    Code:
    /etc/cron.hourly/sa-update:
    plugin: failed to parse plugin (from @INC): Can't locate Mail/SpamAssassin/Plugin/HashBL.pm in @INC (you may need to install the Mail::SpamAssassin::Plugin::HashBL module) (@INC contains: lib /usr/share/perl5 /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at (eval 160) line 1.
    
    plugin: failed to parse plugin (from @INC): Can't locate Mail/SpamAssassin/Plugin/HashBL.pm in @INC (you may need to install the Mail::SpamAssassin::Plugin::HashBL module) (@INC contains: lib /usr/share/perl5 /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at (eval 233) line 1.
    
    
    Thank you
     
  11. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    Looks like you don't have the plugin. If that's the issue, it seems like you didn't perform the latest updates from PMG as they brought a new SpamAssassin version, which include HashBL.pm.
     
    killmasta93 likes this.
  12. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    Thanks for the reply, your right on another box i had with that version it worked
     
  13. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    Quick question any ideas howcome on the sa-update i keep getting a cron email from this
    Code:
    channel: could not find working mirror, channel failed
    
    This is the sa-update
    Code:
    #!/bin/sh
    
    # schaal @it
    #
    # Simple script to update SpamAssassin
    
    SYSLOG_TAG=sa-update
    
    compile=0
    
    logger -d -t $SYSLOG_TAG "Start SA-Update"
    
    sa-update --nogpg
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    
    sa-update --nogpg --channel updates.spamassassin.org
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    sa-update --nogpg --channel sa.zmi.at
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    sa-update --nogpg --channel sa.schaal-it.net
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    sa-update --nogpg --channel sought.rules.yerp.org
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    sa-update --nogpg --channel spamassassin.heinlein-support.de
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    if [ $compile -eq 1 ]; then
        logger -d -t $SYSLOG_TAG "SA-Update found"
        sa-compile --quiet 2>/dev/null
        systemctl restart pmg-smtp-filter.service
    else
        logger -d -t $SYSLOG_TAG "No SA-Update found"
    fi
     
  14. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    At my updated documentation (and also mentioned some posts before) with new SA SHA256 signatures are mandantory. First also zmi failed, but has been fixed, however sought seems to be gone at all, so please remove to update for sought and it will work again.
     
    killmasta93 likes this.
  15. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    Thanks that did the trick for the 5.1 gave issue but the 5.0 was not giving any issue so i guess on the new box which had 5.1 had that issue

    Thank you again
     
  16. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    Update: So today i got a few cron error
    Code:
    invalid regexp for rule SCHAALIT_HEADER_5752: /\Part num your Hacked phone!/: Can't find Unicode property definition "a" in regex; marked by <-- HERE in m/\Pa <-- HERE rt num your Hacked phone!/
    
    config: warning: description exists for non-existent rule SCHAALIT_HEADER_5752
    
    
    This is the SA-update

    Code:
    #!/bin/sh
    
    # schaal @it
    #
    # Simple script to update SpamAssassin
    
    SYSLOG_TAG=sa-update
    
    compile=0
    
    logger -d -t $SYSLOG_TAG "Start SA-Update"
    
    sa-update --nogpg
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    
    sa-update --nogpg --channel updates.spamassassin.org
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    sa-update --nogpg --channel sa.zmi.at
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    sa-update --nogpg --channel sa.schaal-it.net
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    sa-update --nogpg --channel spamassassin.heinlein-support.de
    retval="$?"
    if [ $retval -eq 0 ]; then compile=1; fi
    
    if [ $compile -eq 1 ]; then
        logger -d -t $SYSLOG_TAG "SA-Update found"
        sa-compile --quiet 2>/dev/null
        systemctl restart pmg-smtp-filter
    else
        logger -d -t $SYSLOG_TAG "No SA-Update found"
    fi
    
     
  17. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    370
    Likes Received:
    88
    I got that too and already contacted the rule provider and he already solved the problem.
     
    bhueske and killmasta93 like this.
  18. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    Thanks for the reply, so for now havent got any cron alerts so far so good :)
     
  19. lhorace

    lhorace New Member

    Joined:
    Oct 17, 2015
    Messages:
    7
    Likes Received:
    0
    Hello,

    First, I'd like to thank @heutger for his contribution to this. However, has there been any movement on this? So folks like me could benefit from this in a more reliable way? So far, I have a ad hoc, working setup, with postfixadmin, rspamd, dovecot, ClamAV, and Sieve. Although, I will experiment with vanilla Proxmox Mail Gateway nevertheless, I'd still like to give this a shot? The Guide here, the formatting, seems to cause more problems than it solves. :\
     
  20. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    292
    Likes Received:
    9
    not sure if anyone else is getting this constant from the cron?

    Code:
    channel: no 'mirrors.updates.spamassassin.org' record found, channel failed
    channel: no 'mirrors.updates.spamassassin.org' record found, channel failed
    channel: no 'mirrors.sa.zmi.at' record found, channel failed
    channel: no 'mirrors.sa.schaal-it.net' record found, channel failed
    channel: no 'mirrors.spamassassin.heinlein-support.de' record found, channel failed
    
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice