Feature Requests

Discussion in 'Mail Gateway: Installation and configuration' started by heutger, Apr 25, 2018.

  1. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    Your solution looks fine and I test it for installation in our company (then for sure with license). However, I recognized, that some things are missing:

    - IPv6 is not fetched on installation and setting need restart?
    - No time server could be set!
    - Cert handling would be fine via UI, but is ok via command line
    - Why no own bind server shipped with installation (prevent from blocks like URIBL with common nameservers)
    - Why no firewall setup / settings to run Proxmox also "in the internet"
    - Additional VPN setup would be fine to restrict access to the system
    - Missing Pyzor and DCC (if because of license perhaps providing assistance/a script/... to install DCC)
    - Missing additional postfix restrictions against RFC ignorant spam (smtpd_data_restrictions = reject_unauth_pipelining, unknown_address_reject_code = 550, unknown_client_reject_code = 550)
    - Missing additional blacklist possibilities (reject_rhsbl_helo dbl.spamhaus.org, reject_rhsbl_sender dbl.spamhaus.org, reject_rhsbl_client dbl.spamhaus.org
    - Missing additional content list possibilities (add additional blacklists to spamassassin like Barracuda RBL (has some false positives, so better for content), RBLDNS.RU (has more false positives, same for following ones), SPFBL, s5h RBL, JunkEmailFilter with Black, White and Brown, DNSRBL, JustSpam, inps DNSBL, GBUdb Truncate, V4BL/FREE, maybe also UCEPROTECT Level 1 and BACKSCATTERER Zone)
    - Tracking Center may also show subject and especially the spam level reached, should be filterable for Status (maybe more Status options for rejected ones) and searchable for log content
    - Statistic graphs don't include spam mails and there is no clear definition in difference of rejected is Junk and content scanned maybe Spam

    So the solution looks very good with room for improvements. I believe, open sourcing the solution has the idea to find help from users, where to improve to kick all competitor products away.
     
    Nur Hidayat and DerDanilo like this.
  2. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    I suggest to contribute by your own to the product by creating an howto here or by submitting patches to devs.
     
    #2 Davide Bozzelli, Apr 26, 2018
    Last edited: Apr 26, 2018
  3. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    I'm unsure, if I did things correct, but what I did yet:

    1 vi /etc/apt/sources.list.d/pmg-enterprise.list => changing from enterprise repo to no-subscription repo for first tests

    deb ... (external links not allowed) stretch pmg-no-subscription

    2 apt-get update
    3 apt-get upgrade
    4 apt-get dist-upgrade

    to update/upgrade with the "new" repo

    5 vi /etc/pmg/pmg-api.pem
    6 vi /etc/pmg/pmg-tls.pem

    adding a valid certificate (first private key, then endpoint cert, then chain certs (between endpoint and root, but not root itself)

    7 apt-get install pyzor

    to install pyzor

    8 apt-get install gcc
    9 apt-get install make

    to have the prerequirements to install dcc

    10 cd /tmp
    11 wget ... (external links not allowed) dcc-dccproc.tar.Z
    12 tar xzvf dcc-dccproc.tar.Z
    13 cd dcc-dccproc-*
    14 ./configure
    15 make
    16 make install
    17 ln -s /var/dcc/libexec/dccifd /usr/local/bin/dccifd

    to install dcc

    18 vi /etc/mail/spamassassin/custom.cf => to activate both

    loadplugin Mail::SpamAssassin::plugin::pyzor
    use_pyzor 1
    loadplugin Mail::SpamAssassin::plugin::DCC
    use_dcc 1
    dcc_path /usr/local/bin/dccproc

    19 spamassassin -D --lint

    to test valid config

    20 echo "test" | spamassassin -D pyzor 2>&1 | less

    to test pyzor (did not found a way to test DCC yet)

    21 apt-get install bind9
    22 systemctl start bind9

    to have a local nameserver to prevent blockings at blocklists because of too much enquiries from common (caching) nameservers, after that chaning nameserver to 127.0.0.1 in the PMG GUI

    I found a bug (or somehow an unhappy setting) if changing the search domain to . (as I don't want non fqdn domains to be checked by adding search domain) that the same settings are used for postfix mydomain and myhostname resulting in illegal settings here and postfix to stop working

    23 apt-get install ufw

    for easier firewall configuration

    24 ufw enable
    25 ufw default deny incoming
    26 ufw default allow outgoing
    27 ufw allow ssh
    28 ufw allow smtp
    29 ufw allow 8006

    to set the firewall up with all required ports from outside (if only using incoming filtering, if also using outgoing filtering, port 26 is needed as well)

    30 ufw status verbose

    to check firewall config

    31 apt-get install ntp

    to install timeserver update

    32 vi /etc/ntp.conf => set a better time source to sync with

    server de.pool.ntp.org

    33 systemctl start ntp
    34 ntpq -p

    to adopt the new setting and check, if timeserver is fetching the right time source

    I delayed VPN setup until I'm sure, I will keep PMG

    I did not set the additional postfix restrictions, as main.cf seems to be written by PMG and as there is the custom.cf for spamassassin custom settings, I did not found any mentions on how to do with postfix.

    I can't do GUI changes and contribute them, as I don't do web development any more.

    For the additional lists in spamassassin, I will now first check out with running URIBL, Pyzor and DCC and will consider to add additional blacklists lateron. I will update this thread with additional custom.cf settings then and also with VPN setup, if done.

    Oh, and I whitlisted 194.25.134.0/24 (as German Telecom always fails not to get their servers blacklisted) with the following PMG adjustments:

    I removed quarantine or blocks and changed them to deliver with subject spam prefix or removing infected attachments, I adjusted level 3 because of too much false positive to level 4, I added the blocklists ix.dnsbl.manitu.net;sbl-xbl.spamhaus.org;bl.spamcop.net;psbl.surriel.com;spamrbl.imp.ch;noptr.spamrats.com;escalations.dnsbl.sorbs.net;singular.ttk.pte.hu because this set worked well for me (sbl-xbl instead of zen because of dialup-IPs and my home systems however also send mails using dialup-IPs, singular is on testing currently, checking rejects from singular currently, all the others are running well for yours, barracuda recently had false positives, so I don't use it here any more, same for GBUdb, you may check inps.de and/or intra2net.com spam statistics, but I don't agree with their counts for my setup(s)), I changed maximum message to 104857600 bytes, enabled reject unknown clients, senders and smtp helo checks, disabled greylisting and enabled spf as well as all TLS options.

    Oh, and I installed a vi version, which is usable, not this stupid version preinstalled:

    35 apt-get install vim
     
    DerDanilo likes this.
  4. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    For postfix conf the procedure to modify conf files is documented in the admin guide.
    Basically mkdir -p /etc/pmg/templates and copy here global conf files which lives in /var/lib/pmg/templates.

    Then make your own modifications and do
    Pmgconfig sync
     
  5. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    I prefer running dcc as service (dccifd) to avoid on each spamassassin scan to spawn a new dccproc.
     
  6. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    Sounds a bit complicated, but I will give it a try
     
  7. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    Sounds much better and I believe, therefore I need to create the symlink, however, I don't know on how to do that. Can you help?

    I currently try to improve clamav detection, as it looks not so good (the mails I currently try and filter are filtered afterwards by two additional antispam service providers, so I can see, what they do with "the rest" and they recently found a virus, which clamav did not detect). I try SecurityInfo and SaneSecurity and will report then.

    I just found another "bug", I will post bugs in an extra thread. It's about SPF seems to have problems with forwarded mails although the forwarding server seems to have SRS enabled.

    And I need to remove ntpd again, I just found, that a time server service is already running, it's just not adjustable via GUI but via command line, I also will report on this.
     
  8. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    Updated tasks done:

    31 vi /etc/systemd/timesyncd.conf

    [Time]
    NTP=de.pool.ntp.org

    32 systemctl restart systemd-timesyncd

    33 apt-get install vim

    vi is catastrophic, vim is a bit too much, I prefer CentOS and vim-enhanced there, but vim looks most similar

    34 mkdir .ssh
    35 vi .ssh/authorized_keys

    I prefer to use a real terminal and it's much easier to be authorized by a ssh key (for sure I won't post it here)

    36 vi /etc/profile => I usually set some aliases and I hate spaces and duplicates in history, so I add this one

    HISTSIZE=1000
    if [ "$HISTCONTROL" = "ignorespace" ] ; then
    export HISTCONTROL=ignoreboth
    else
    export HISTCONTROL=ignoredups
    fi
    export HISTSIZE HISTCONTROL

    alias "ls=ls -al"
    alias "rm=rm --preserve-root"
    alias "cd..=cd .."
    alias "dir=ls -al"

    Now I will check for clamav improvements
     
  9. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    I did your recommendation, many thanks:

    37 mkdir -p /etc/pmg/templates
    38 cp /var/lib/pmg/templates/* /etc/pmg/templates/.
    39 cd /etc/pmg/templates
    40 vi main.cf.in

    config follows below

    41 pmgconfig sync --restart 1
     
  10. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    # auto-generated by proxmox


    compatibility_level = 2

    command_directory = /usr/sbin

    daemon_directory = /usr/lib/postfix/sbin

    data_directory = /var/lib/postfix


    # appending .domain is the MUA's job.

    append_dot_mydomain = yes


    smtpd_banner = $myhostname [% pmg.mail.banner %]

    biff = no


    [% IF pmg.mail.dwarning %]

    delay_warning_time = [% pmg.mail.dwarning %]h

    [% END %]


    best_mx_transport = local

    message_size_limit = [% pmg.mail.maxsize %]

    mailbox_size_limit = [% ((pmg.mail.maxsize*2 > 51200000) ? pmg.mail.maxsize*2 : 51200000) %]


    mydomain = [% dns.domain %]

    myhostname = [% dns.hostname %].[% dns.domain %]


    parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks,permit_mx_backup_networks,qmqpd_authorized_clients,smtpd_access_maps


    alias_maps = hash:/etc/aliases

    alias_database = hash:/etc/aliases

    mydestination = localhost, $myhostname

    mynetworks = [% postfix.mynetworks %]


    relay_domains = hash:/etc/pmg/domains


    transport_maps = hash:/etc/pmg/transport


    [% IF pmg.mail.relay %]

    [% IF pmg.mail.relaynomx %]

    relay_transport = smtp:[[% pmg.mail.relay %]]:[% pmg.mail.relayport %]

    [% ELSE %]

    relay_transport = smtp:[% pmg.mail.relay %]:[% pmg.mail.relayport %]

    [% END %]

    [% END %]


    [% IF pmg.mail.smarthost %]

    default_transport = smtp:[% pmg.mail.smarthost %]

    [% END %]


    content_filter=scan:127.0.0.1:10024


    mail_name = Proxmox


    [% IF pmg.mail.helotests %]

    smtpd_helo_required = yes

    smtpd_helo_restrictions = permit_mynetworks reject_non_fqdn_helo_hostname reject_invalid_helo_hostname reject_rhsbl_helo dbl.spamhaus.org

    [% ELSE %]

    smtpd_helo_restrictions =

    [% END %]


    postscreen_access_list =

    permit_mynetworks,

    cidr:/etc/postfix/postscreen_access


    [% IF postfix.dnsbl_sites %]

    postscreen_dnsbl_sites = [% postfix.dnsbl_sites %]

    [% END %]


    postscreen_dnsbl_action = enforce

    postscreen_greet_action = enforce


    smtpd_sender_restrictions =

    permit_mynetworks

    reject_non_fqdn_sender

    check_client_access cidr:/etc/postfix/clientaccess

    check_sender_access regexp:/etc/postfix/senderaccess

    check_recipient_access regexp:/etc/postfix/rcptaccess

    [%- IF pmg.mail.rejectunknown %] reject_unknown_client_hostname[% END %]

    [%- IF pmg.mail.rejectunknownsender %] reject_unknown_sender_domain[% END %]

    reject_rhsbl_client dbl.spamhaus.org

    reject_rhsbl_sender dbl.spamhaus.org


    smtpd_recipient_restrictions =

    permit_mynetworks

    reject_unauth_destination

    reject_non_fqdn_recipient

    check_recipient_access regexp:/etc/postfix/rcptaccess

    [%- IF postfix.usepolicy %] check_sender_access regexp:/etc/postfix/senderaccess[% END %]

    [%- IF postfix.usepolicy %] check_client_access cidr:/etc/postfix/clientaccess[% END %]

    [%- IF postfix.usepolicy %] check_policy_service inet:127.0.0.1:10022[% END %]

    [%- IF pmg.mail.verifyreceivers %] reject_unknown_recipient_domain[% END %]

    [%- IF pmg.mail.verifyreceivers %] reject_unverified_recipient[% END %]


    smtpd_data_restrictions = reject_unauth_pipelining


    [% IF pmg.mail.rejectunknownsender %]

    unknown_address_reject_code = 550

    [% ELSE %]

    [% IF pmg.mail.verifyreceivers %]

    unknown_address_reject_code = 550

    [% END %]

    [% END %]


    [% IF pmg.mail.rejectunknown %]

    unknown_client_reject_code = 550

    [% END %]


    [% IF pmg.mail.verifyreceivers %]

    unverified_recipient_reject_code = [% pmg.mail.verifyreceivers %]

    [% END %]


    smtpd_client_connection_count_limit = [% pmg.mail.conn_count_limit %]

    smtpd_client_connection_rate_limit = [% pmg.mail.conn_rate_limit %]

    smtpd_client_message_rate_limit = [% pmg.mail.message_rate_limit %]


    [% IF pmg.mail.tls %]

    smtp_tls_security_level = may

    smtp_tls_policy_maps = hash:/etc/pmg/tls_policy

    smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

    smtpd_tls_security_level = may

    smtpd_tls_cert_file = /etc/pmg/pmg-tls.pem

    smtpd_tls_key_file = $smtpd_tls_cert_file

    [% IF pmg.mail.tlslog %]

    smtpd_tls_loglevel = 1

    smtp_tls_loglevel = 1

    [% END %]

    [% IF pmg.mail.tlsheader %]

    smtpd_tls_received_header = yes

    [% END %]

    [% END %]


    smtp_tls_session_cache_database = btree:/var/lib/postfix/smtp_tls_session_cache

    smtpd_tls_session_cache_database = btree:/var/lib/postfix/smtpd_tls_session_cache



    default_destination_concurrency_limit = 40

    lmtp_destination_concurrency_limit = 20

    relay_destination_concurrency_limit = 20

    smtp_destination_concurrency_limit = 20

    virtual_destination_concurrency_limit = 20


    recipient_delimiter = +
     
  11. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    For clamav just use clamav-unofficial-sig.
    Get it from github and not use the one bundled with debian as it really old.
    Be sure to subscribe freely to the various feeds as documented in the Shell script.
     
  12. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    Did you the same setup as me? Could you provide help on dcc? How can I check, if it's really running. All spam mails I checked don't have any PYZOR or DCC "evidence".

    I now worked on as suggested:

    42 apt-get install unzip

    as unzip is missing

    43 cd /tmp
    44 wget https://github.com/extremeshok/clamav-unofficial-sigs/archive/master.zip
    45 unzip master.zip

    to fetch clamav-unofficial-sigs from github

    46 cp clamav-unofficial-sigs-master/clamav-unofficial-sigs.sh /usr/local/sbin/
    47 chmod 755 /usr/local/sbin/clamav-unofficial-sigs.sh
    48 mkdir /etc/clamav-unofficial-sigs
    49 cp clamav-unofficial-sigs-master/config/* /etc/clamav-unofficial-sigs/
    50 mkdir /var/log/clamav-unofficial-sigs

    doing as described to prepare all files and folders

    51 cd /etc/clamav-unofficial-sigs
    52 cat /etc/*release*

    to find out OS version

    53 mv os.debian9.conf os.conf

    to use the OS specific configuration

    54 vi user.conf

    adjusted content follows

    55 /usr/local/sbin/clamav-unofficial-sigs.sh --install-cron
    56 /usr/local/sbin/clamav-unofficial-sigs.sh --install-logrotate
    57 /usr/local/sbin/clamav-unofficial-sigs.sh --install-man

    to install as described

    58 /usr/local/sbin/clamav-unofficial-sigs.sh

    first run as described, just found issues here again, I reported bugs at github: securiteinfo.hdb does not download and MalwarePatrol different product code (I'm unsure why, but I got 32 instead of 8 which should have been assigned) ignored (although set in config), I additionally adjusted master.conf now, but need to wait 24 hours to recheck

    59 cp /tmp/clamav-unofficial-sigs-master/systemd/* /etc/systemd/

    as described for systemd

    60 clamscan --debug 2>&1 /dev/null | grep "loaded"

    as described to verify installation, looks good beside the issues above
     
  13. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    I just uncommented and adjusted in user.conf

    malwarepatrol_receipt_code="xxxxx" => for sure, here is my receipt code set
    malwarepatrol_product_code="32" => that's the product code my URL had and with this one, I'm able to download, I could not retest as --force also forced to retry the SecuriteInfo database, which does not download and hang the script
    malwarepatrol_list="clamav_basic" # clamav_basic or clamav_ext
    malwarepatrol_free="yes"

    securiteinfo_authorisation_signature="xxxxx" => same here, I won't provide my credentials here

    user_configuration_complete="yes" => not well documented, but need to uncomment, to activate user.conf
     
  14. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    Hi

    Here my dcc.service


    [Unit]
    Description=DCC (Distributed Checksum Clearinghouses) interface daemon
    After=remote-fs.target systemd-journald-dev-log.socket

    [Service]
    Type=forking
    PermissionsStartOnly=true
    RuntimeDirectory=dcc
    ExecStart=/var/dcc/libexec/dccifd
    User=root
    Group=root
    Nice=1

    #DCC writes pid file with "-" at the beginning which confuses systemd
    #PIDFile=/run/dcc/dccifd.pid

    [Install]
    WantedBy=multi-user.target
     
  15. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    FOR DCC:

    1) edit /vardcc/dcc_conf
    2) change to DCCIFD_ENABLE=on
    3) Add to /etc/mail/spamassassin/custom.cf :
    dcc_path /usr/local/bin/dccproc
    dcc_home /var/dcc
    dcc_dccifd_path /var/dcc/dccifd
    dcc_body_max 999999
    dcc_fuz1_max 999999
    dcc_fuz2_max 999999
    use_dcc 1
    dcc_timeout 10

    Then try to run spamassasin lint ad see if sa talks correctly with dccifd started from systemd (see my previous post)
     
  16. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    I sugget to register to malwarepatrol and securiteinfo .
     
  17. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    Parallel I checked the quality of spam detection now with my adjustments and be very happy with. I'm just upset, that the last statistics are missing. I now decided for the first not to add any additional blacklists but try to run with the current setup. If spam detection will get worser, I will first just add barracudacentral as it's the most reliable database (however, if you google, you can find many complains about this list in conjunction with emailreg, looking a bit like UCEPROTECT seem to take money for unlisting with strange listing issues).

    So next and for the first last two steps will be setting up an automated backup (using the backup script for settings backup plus backing up relevant paths, which include adjustments I did and uploading them to my backup location via lftp, if that's available for Debian also (as mentioned coming from CentOS systems, I always prefer)) and setting up a VPN connection to the server, that I can close SSH and GUI from internet (as system is hosted, I can't add a firewall in front).

    If everything still works fine, I will consider a productive test in our company and will then purchase subscription therefor. I'm unsure, which one, depending on what support is really provided. E.g. if adjustment (checking) or performing is provided, maybe a greater support level would be fine.
     
  18. heutger

    heutger Active Member

    Joined:
    Apr 25, 2018
    Messages:
    257
    Likes Received:
    67
    I did and got the codes, however securiteinfo first file got downloaded well, second it hangs, malwarepatrol doesn't work at all, as I got another product code, which I entered in user.conf, but got ignored
     
  19. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    I don't know if you made massive tweaks and mods the support will assist you.
    As for rbl, i've preferred to use postscreen rbl and use the threeshold feature (see: http://rob0.nodns4.us/postscreen.html).
    i've just using the rbl feature not any other checks (After-220 tests), just using pregreet and rbl.
    The most notable feature is to use whitelist rbls to decrease points for blacklisted ip.
     
  20. Davide Bozzelli

    Joined:
    Feb 6, 2018
    Messages:
    57
    Likes Received:
    3
    I've followed EXACTLY what docs of script said and it works
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice