Feature Requests

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
Hi

Here my dcc.service


[Unit]
Description=DCC (Distributed Checksum Clearinghouses) interface daemon
After=remote-fs.target systemd-journald-dev-log.socket

[Service]
Type=forking
PermissionsStartOnly=true
RuntimeDirectory=dcc
ExecStart=/var/dcc/libexec/dccifd
User=root
Group=root
Nice=1

#DCC writes pid file with "-" at the beginning which confuses systemd
#PIDFile=/run/dcc/dccifd.pid

[Install]
WantedBy=multi-user.target
Many thanks. I'm a bit new to systemd, where do I need to store this dcc.service file? Do I need to start the dcc.service then first? My recent dccifd symlink then is completely unneccessary?
 
Feb 6, 2018
76
6
8
47
Many thanks. I'm a bit new to systemd, where do I need to store this dcc.service file? Do I need to start the dcc.service then first? My recent dccifd symlink then is completely unneccessary?
ls -l /lib/systemd/system/dcc.service
-rw-r--r-- 1 root root 404 Mar 21 18:17 /lib/systemd/system/dcc.service
systemctl enable dcc
systemctl start dcc

I wold suggest to test FIRST the service outside systemd to see if it starts correctly
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
I don't know if you made massive tweaks and mods the support will assist you.
As for rbl, i've preferred to use postscreen rbl and use the threeshold feature (see: http://rob0.nodns4.us/postscreen.html).
i've just using the rbl feature not any other checks (After-220 tests), just using pregreet and rbl.
The most notable feature is to use whitelist rbls to decrease points for blacklisted ip.
Sounds interesting, however, also needs adjustments, not available via GUI. I recently did not work with whitelists yet, although I don't believe in all the numbers, http://analyse.inps.de/?type=monthly&lang=de&service=&month=03&year=2018&sort=5 state also many false-negatives for the current only running whitelist with a big pool of IPs DNSWL.org
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
Thx, I was able to follow your explanations to change DCC to a service

--- .bash_history 2018-04-28 22:57:46.549979499 +0200
+++ .bash_history.bak 2018-04-28 22:01:20.952671659 +0200
@@ -14,10 +14,7 @@
./configure
make
make install
-vi /lib/systemd/system/dcc.service
-systemctl enable dcc
-systemctl start dcc
-vi /var/dcc/dcc_conf
+ln -s /var/dcc/libexec/dccifd /usr/local/bin/dccifd
vi /etc/mail/spamassassin/custom.cf
spamassassin -D --lint
echo "test" | spamassassin -D pyzor 2>&1 | less
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
I did two additional adjustments:

I removed singular.ttk.pte.hu from my blacklists as United Internet (1&1, GMX, web.de, ...) seems not be able to hold their servers under control and got listed. Although I whitelisted the DTAG/Deutsche Telekom, I won't do with United Internet as DTAG/Deutsche Telekom always get listed on Spamhaus or Spamcop, which from my point of opinion are industry standards, meanwhile singular.ttk.pte.hu is from a hungarian university for primary internal use, so it's better to remove this list from blocking and consider to add for tagging instead of whitelisting false-positives. I will think about adding bl.spameatingmonkey.net as many people like this list, but I will see.

I was in need to adjust /etc/clamav-unofficial-sigs/user.conf and set malwarepatrol_free="no" as otherwise my product code, which is 32 instead of 8, get ignored and the list is not working. I'm unsure, why I got another code than others, however with 8 it's not working and malwarepatrol_free="yes" seems to override malwarepatrol_product_code, although set to 32. Now everything seems to work fine, need to delete /var/lib/clamav-unofficial-sigs/configs/last-mbl-update.txt for testing.

Additional comment:

securiteinfo.hdb seems not to hang, it just take about 15 minutes, once it took 30 minutes as seen in the log file, to update the file. Strange, but that's the reason why it looked like the script hang. As the server, I'm testing Proxmox with, has 2.6 Tbit/s internet connectivity, internet connect can't be the reason. However, it's working now, that's the most important.
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
One more, I disabled Configuraion => Option => Use advanced statistic filters to be able to see statistics for incoming filtering only, otherwise would require outgoing mails also going through PMG to be able to see any statistics per user.
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
And last one for today:

As mentioned before, minimum Spam Level is 4 for me to prevent false-positives, running well, also I don't use Quarantine, I just Modify Spam Subject. I renamed this one as mentioned before. I also renamed the ones with Viruses and Dangerous Files incoming to Remove and replaced Block and Notify Admin with Removing attachment (Virus or Dangerous File).
 
Feb 6, 2018
76
6
8
47
One more, I disabled Configuraion => Option => Use advanced statistic filters to be able to see statistics for incoming filtering only, otherwise would require outgoing mails also going through PMG to be able to see any statistics per user.
Thx!! I never figure out this when experiencing the issue and nobody clarify this .
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
Work continued. As I changed DCC to a service and adjusted my "worklog", I will continue with number 64 (3 additional rows been added as stated above to change the symlink for dcc to real service setup, thanks to Davide).

I now set up OpenVPN access to the server only. So I used an OpenVPN setup script to be very pragmatic and just adjusted some values:

64 cd /tmp
65 wget https://github.com/Angristan/OpenVPN-install/archive/master.zip
66 unzip master.zip
67 cd OpenVPN-install-master
68 chmod +x openvpn-install.sh
69 ./openvpn-install.sh

I left the detected IP address (however recognized, that this IP address is used as remote address value in client config, which I changed later on to the hostname) left port and protocol as given, I chose resolvers DNS servers, as I also removed that line later on pushing DNS servers as I don't need a road warrior setup but just a server access setup, selected the fastest options for data channel, DH and RSA (as I just want to block out script kiddies etc. from accessing or trying to access SSH or GUI, but it's no rocket science done here), left the client name and let the script set up everything for me.

Welcome to the secure OpenVPN installer (github.com/Angristan/OpenVPN-install)

I need to ask you a few questions before starting the setup
You can leave the default options and just press enter if you are ok with them

I need to know the IPv4 address of the network interface you want OpenVPN listening to.
If your server is running behind a NAT, (e.g. LowEndSpirit, Scaleway) leave the IP address as it is. (local/private IP)
Otherwise, it should be your public IPv4 address.
IP address: xxx.xxx.xxx.xxx

What port do you want for OpenVPN?
Port: 1194

What protocol do you want for OpenVPN?
Unless UDP is blocked, you should not use TCP (unnecessarily slower)
Protocol [UDP/TCP]: UDP

What DNS do you want to use with the VPN?
1) Current system resolvers (from /etc/resolv.conf)
2) Cloudflare (Anycast: worldwide)
3) Quad9 (Anycast: worldwide)
4) FDN (France)
5) DNS.WATCH (Germany)
6) OpenDNS (Anycast: worldwide)
7) Google (Anycast: worldwide)
8) Yandex Basic (Russia)
9) AdGuard DNS (Russia)
DNS [1-8]: 1

See https://github.com/Angristan/OpenVPN-install#encryption to learn more about
the encryption in OpenVPN and the choices I made in this script.
Please note that all the choices proposed are secure (to a different degree)
and are still viable to date, unlike some default OpenVPN options

Choose which cipher you want to use for the data channel:
1) AES-128-CBC (fastest and sufficiently secure for everyone, recommended)
2) AES-192-CBC
3) AES-256-CBC
Alternatives to AES, use them only if you know what you're doing.
They are relatively slower but as secure as AES.
4) CAMELLIA-128-CBC
5) CAMELLIA-192-CBC
6) CAMELLIA-256-CBC
7) SEED-CBC
Cipher [1-7]: 1

Choose what size of Diffie-Hellman key you want to use:
1) 2048 bits (fastest)
2) 3072 bits (recommended, best compromise)
3) 4096 bits (most secure)
DH key size [1-3]: 1

Choose what size of RSA key you want to use:
1) 2048 bits (fastest)
2) 3072 bits (recommended, best compromise)
3) 4096 bits (most secure)
RSA key size [1-3]: 1

Finally, tell me a name for the client certificate and configuration
Please, use one word only, no special characters
Client name: client

Okay, that was all I needed. We are ready to setup your OpenVPN server now
Press any key to continue...

...

Finished!

Your client config is available at /root/client.ovpn
If you want to add more clients, you simply need to run this script another time!

70 vi /etc/openvpn/server.conf

I removed the both push lines

push "dhcp-option DNS 127.0.0.1"
push "redirect-gateway def1 bypass-dhcp"

because DHCP makes no sense and I don't need a VPN DNS for administrating my server and for sure I don't want all traffic going through the server

71 service openvpn restart

After changing the values, I restarted OpenVPN server

72 vi /root/client.ovpn

I changed the IP address to the hostname and removed the line

setenv opt block-outside-dns

for the same reason as above (this would prevent my local DNS server settings from working).

Then I downloaded the client.ovpn to my systems, added it to their config and tested the VPN connection.

73 ufw delete allow 8006
74 ufw allow from 10.8.0.2 to any port 8006
75 ufw delete allow ssh
76 ufw allow from 10.8.0.2 to any port ssh

I removed the old firewall rules allowing full internet access to port 8006 (first tested with GUI to prevent locking myself out from ssh before I have proofed concept) and ssh (after successful testing).

Now my last topic is to set up a backup script.
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
I did it, for now, I'm finished with the setup, next step is to set another system up and test it for the company.

77 apt-get install lftp
78 mkdir /backup
79 mkdir /backup/node
80 mkdir /root/scripts
81 vi /root/scripts/backup.sh

#!/bin/bash

echo [`date +'%d.%m.%Y %H:%M'`] Starte Backup ...

echo [`date +'%d.%m.%Y %H:%M'`] Exportiere aktuelle Konfiguration [1/4] ...
rm /var/lib/pmg/backup/*
pmgbackup backup

# Zu sicherndes Verzeichnisse, z.B. /etc /root /var
sourcevrz='/etc /root /usr/local /var/lib/pmg/backup /var/log'

# Backupverzeichnis, z.B. /backup/node
backupvrz=/backup/node

# Variablendefinition
datum=$(date +'%Y%m%d')
dateiname=$backupvrz/backup$datum.tar

# -----------------------------------------------------
function f_delFiles()
# -----------------------------------------------------
# $1 Backupverzeichnis
{
loeschdatum=$(date --date='7 days ago' +'%Y%m%d')
rm $1/backup$loeschdatum.tar
}

echo [`date +'%d.%m.%Y %H:%M'`] Lösche Files in $backupvrz, die älter als 7 Tage sind [2/4] ...
f_delFiles $backupvrz

echo [`date +'%d.%m.%Y %H:%M'`] Sichere $sourcevrz nach $dateiname [3/4] ...
tar Pcf $dateiname $sourcevrz

echo [`date +'%d.%m.%Y %H:%M'`] Synchronisiere mit Onlinespeicher [4/4] ...
bash /root/scripts/upload.sh

echo [`date +'%d.%m.%Y %H:%M'`] Fertig!

82 vi /root/scripts/upload.sh

#!/bin/bash
HOST='xxxx'
USER='xxxx'
PASS='xxxx'
TARGETFOLDER='/xxxx'
SOURCEFOLDER='/backup'

lftp -e "
open $HOST
user $USER $PASS
lcd $SOURCEFOLDER
mirror --reverse --delete --use-cache --verbose $SOURCEFOLDER $TARGETFOLDER
bye
"

83 chmod +x /root/scripts/backup.sh
84 chmod +x /root/scripts/upload.sh

85 /root/scripts/backup.sh

Testing the script.

86 vi /etc/crontab

0 2 * * * root /root/scripts/backup.sh 2>&1 | /usr/bin/mail -r xxxx@xxxx -s "backup script output" xxxx@xxxx
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
I'm not yet completely happy with spam detection (for commercial test), so I did some additional adjustments:

I increased required spam score to 6 because of false-positives
I changed for virus and dangerous attachments also to add a header VIRUS and DANGEROUS as usually that would also be somehow spam mails and to inform the receipient, that there are changes to the message

I added Barracudacentral with 1.4 points as blacklist in SpamAssassin custom.cf

I added additional rules

I added and enabled GeoIP filter (also for bayes)

I added fromreplyto plugin

# apt-get install re2c
# cd /etc/cron.hourly && wget sa.schaal-it.net/sa-update && chown root.root sa-update && chmod 755 sa-update
# vi sa-update

#!/bin/sh

# schaal @it
#
# Simple script to update SpamAssassin

SYSLOG_TAG=sa-update

compile=0

logger -d -t $SYSLOG_TAG "Start SA-Update"

sa-update --nogpg
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi


sa-update --nogpg --channel updates.spamassassin.org
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sa.zmi.at
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sa.schaal-it.net
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sought.rules.yerp.org
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel spamassassin.heinlein-support.de
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

if [ $compile -eq 1 ]; then
logger -d -t $SYSLOG_TAG "SA-Update found"
sa-compile
systemctl restart pmg-smtp-filter.service
else
logger -d -t $SYSLOG_TAG "No SA-Update found"
fi

# ./sa-update
# cd /tmp
# wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
# wget http://geolite.maxmind.com/download/geoip/database/GeoIPv6.dat.gz
# gunzip GeoIP.dat.gz
# gunzip GeoIPv6.dat.gz
# mkdir /usr/share/GeoIP
# mv GeoIP.dat /usr/share/GeoIP/
# mv GeoIPv6.dat /usr/share/GeoIP/
# vi /etc/mail/spamassassin/custom.cf

loadplugin Mail::SpamAssassin::plugin::pyzor
use_pyzor 1

loadplugin Mail::SpamAssassin::plugin::DCC
dcc_path /usr/local/bin/dccproc
dcc_home /var/dcc
dcc_dccifd_path /var/dcc/dccifd
dcc_body_max 999999
dcc_fuz1_max 999999
dcc_fuz2_max 999999
use_dcc 1
dcc_timeout 10

loadplugin Mail::SpamAssassin::plugin::RelayCountry
add_header all Relay-Country _RELAYCOUNTRY_
header RELAYCOUNTRY_BAD X-Relay-Countries =~ /(CN|RU|UA|RO|VN)/
describe RELAYCOUNTRY_BAD Relayed through spammy country at some point
score RELAYCOUNTRY_BAD 2.0
header RELAYCOUNTRY_GOOD X-Relay-Countries =~ /^(DE|AT|CH)/
describe RELAYCOUNTRY_GOOD First untrusted GW is DE, AT or CH
score RELAYCOUNTRY_GOOD -0.5

header RCVD_IN_BRBL eval:check_rbl('brbl-lastexternal', 'b.barracudacentral.org.', '127.0.0.2')
describe RCVD_IN_BRBL Received via a relay in Barracuda RBL
tflags RCVD_IN_BRBL net
score RCVD_IN_BRBL 1.4

# wget https://github.com/extremeshok/spamassassin-extremeshok_fromreplyto/archive/master.zip
# unzip master.zip
# mkdir /etc/mail/spamassassin/plugins/
# cp spamassassin-extremeshok_fromreplyto-master/plugins/* /etc/mail/spamassassin/plugins/
# cp spamassassin-extremeshok_fromreplyto-master/01_extremeshok_fromreplyto.cf /etc/mail/spamassassin/
# systemctl restart pmg-smtp-filter

Next step will be to add http://artinvoice.hu/spams/ for getting bayes performing better.
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
One more thing: I adjusted my blacklists, I changed sbl-xbl.spamhaus.org to zen.spamhaus.org and moved manitu to the end because of too much false-positives in the past, I will check, how much hits occur to this list and may consider to remove it or I will change my setup to postscreen with weights and tresholds, then I may add barracuda at blocking level and meanwhile all other lists will then have a weight of * 2, manitu and barracuda will have * 1 and minimum treshold will be 2 (so both or any other need to have the server listed to blacklist it). Not yet such happy with this decision, so I will wait a while, I also contacted heise as well as manitu, that their blacklist is currently running into getting less usable, it recently had false-positive quotes like spamhaus or spamcop (or sth. in between), but currently it's more like GBUdb, WPBL or one of the others, which more often have/had false-positives.
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
Update done as announced:

# vi /etc/cron.daily/sa-learn
# chmod +x /etc/cron.daily/sa-learn
# /etc/cron.daily/sa-learn

#!/bin/bash
spamfile=spam--`date --date='1 day ago' '+%Y-%m-%d'`.gz
spamfile_unpacked=spam--`date --date='1 day ago' '+%Y-%m-%d'`
wget http://artinvoice.hu/spams/$spamfile
gunzip $spamfile
sa-learn --mbox --spam --progress $spamfile_unpacked
systemctl restart pmg-smtp-filter
rm -rf $spamfile $spamfile_unpacked
exit 0

So I need to adjust the date, as cron.daily is running in the morning, so I want to add the spams of the last day and not the few ones of the current day. Also the spamassassin restart did not fit, so I also need to adjust that one. Maybe I will need to adjust tomorrow some additional /dev/nulls to get the cronjob "quiet", but I first want to see, if it's really working, so I will first keep it as is.
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
I now also /dev/null wget and sa-learn, however, the download and import works and today I had the first time bayes_* tags in my messages. I need to say, I use one PMG for my private mails, so it's only my family and me, about 7 mailboxes to deliver mails to, so in a productive environment it would be faster to get spams and hams, but in my setup, I have many hams autolearned but nearly no spam and the filter seems need to have some ratio to start bayes-tagging. The commercial PMG test environment I just set up to send all the mails through of all domains, which are not the primary one, we usually use to talk to customers, so the mail flow is really really low. Also learning the spams now didn't help to get bayes-tagging starting.

I also checked occurence of barracuda in spamassassin vs. nix spam false-positives and will first not set the threshold thing. I will try to get in touch with heise to report about the increased false-positive rate.
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
Short update: /dev/null doesn't really work, I now use --quiet, although should not be available on some commands, it seems to work. However, I still get some response, so I test no 2>/dev/null for the errors (some KAM rules seem to have zero score). I will report, once then it's completely working.

I would be happy, if anyone can assist on how to test Pyzor and DCC. None of my mails yet got tagged with Pyzor or DCC, so I'm afraid, it doesn't work.

I also would be happy, if I could reject (hard reject) mails with a special spam score (e.g. >10).
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
Scripts are "quiet" now ;-)


Here how I changed them:


#!/bin/bash
spamfile=spam--`date --date='1 day ago' '+%Y-%m-%d'`.gz
spamfile_unpacked=spam--`date --date='1 day ago' '+%Y-%m-%d'`
wget --quiet http://artinvoice.hu/spams/$spamfile
gunzip $spamfile
sa-learn --mbox --spam --quiet $spamfile_unpacked
systemctl restart pmg-smtp-filter
rm -rf $spamfile $spamfile_unpacked
exit 0


#!/bin/sh

# schaal @it
#
# Simple script to update SpamAssassin

SYSLOG_TAG=sa-update

compile=0

logger -d -t $SYSLOG_TAG "Start SA-Update"

sa-update --nogpg
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi


sa-update --nogpg --channel updates.spamassassin.org
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sa.zmi.at
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sa.schaal-it.net
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel sought.rules.yerp.org
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

sa-update --nogpg --channel spamassassin.heinlein-support.de
retval="$?"
if [ $retval -eq 0 ]; then compile=1; fi

if [ $compile -eq 1 ]; then
logger -d -t $SYSLOG_TAG "SA-Update found"
sa-compile --quiet 2>/dev/null
systemctl restart pmg-smtp-filter.service
else
logger -d -t $SYSLOG_TAG "No SA-Update found"
fi
 

heutger

Active Member
Apr 25, 2018
727
193
43
Fulda, Hessen, Germany
www.heutger.net
I now tried for commercial use and now start to need to do some adjustments.

1. I learned, that sa-learn form "foreign" archives let start bayesian tagging fast, BUT not valid as "my" spam mails differ from "their" spam mails, so I get scores of 0%, 40%, 60% etc. which are far away from reality. So I performed

rm /etc/cron.daily/sa-learn
sa-learn --clear

and now start a new bayes database. Too sad :-(

2. I found a link here in the forum I followed to add mail subjects to the logs:

#cd /etc/pmg/templates
#vi main.cf.in

adding header_checks = regexp:/etc/postfix/header_checks

#vi /etc/postfix/header_checks

with content /^subject:/ INFO

#pmgconfig sync --restart 1

However, this does only work on valid mails, not on rejected ones (where I would like to see the headers). I checked for smtpd_delay_reject, but it's already enabled and just delay until RCPT TO and not until the message content. So I still search for a solution to add subjects to the mail log also on rejected mails.

3. I'm sad, that so many server operators are not able to set up their servers. I was required to disable reject unknown clients, reject unknown senders as well as SPF and to increase spam tagging score to 9 on the commercial test instance.

Also Manitu NiX Spam has now too much false positives, after getpocket, facebook, pinterest, I know saw Linkedin and Unitymedia as well. It's now really too much too accept that, so I will change now to postscreen with treshold. I'm also sure, that Pyzor and DCC don't work, so I will change the setup now. I will report on this adjustments lateron.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!