Suggestions and bugs 5.0

[hansm]

Proxmox Mail Gateway 5.0 looks great! Good work.

During my tests I found a few things I would like to share/ask.

I'm using PMG for incoming mail filtering only. I've setup a domains MX record to the PMG and added the domain to "Relay Domains" and setup a Transport to the destination host/mailserver.
1) PMG sees all mail as outgoing. I need all mail filter rules to apply to outgoing instead of incoming. In the statistics I only see traffic and details for outgoing but it really is all incoming email.

2) At Mail proxy Options I set DNSBL Sites, that works but I would also like to add the whitelist "list.dnswl.org" with negative scores like this: zen.spamhaus.org*2 bl.spamcop.net*1 b.barracudacentral.org*1 ix.dnsbl.manitu.net*1 psbl.surriel.com list.dnswl.org=127.0.[0..255].0*-2 list.dnswl.org=127.0.[0..255].1*-3 list.dnswl.org=127.0.[0..255].[2..3]*-4
But the field doesn't allow me to do so:

Parameter verification failed. (400)
dnsbl_sites: invalid format - value 'list.dnswl.org=127.0.[0..255].0*-2' does not look like a valid dnsbl entry

3) TLS for Mail Proxy is disabled by default. Shouldn't this be enabled by default?

4) Greylisting can be enabled/disabled globally only. I don't want to use it by default because of the delay in incoming emails (only unknown triplet, I know, but still). But I would like to be able to enable it for some domains. When we add a domain to Relay Domains an option to enable greylisting or not would be really nice.

5) We plan on using many domains of different customers. But they need to be able to view there quarantined email/spam. There's a user role Quarantine Manager but that user can see all quarantined email. Is there some generic solution for this? Or should I change the subject of spam messages to mark them as SPAM: and setup rules on the destination server to move those emails to a spam folder?

 

[tom]

Proxmox Mail Gateway 5.0 looks great! Good work.

During my tests I found a few things I would like to share/ask.

I'm using PMG for incoming mail filtering only. I've setup a domains MX record to the PMG and added the domain to "Relay Domains" and setup a Transport to the destination host/mailserver.
1) PMG sees all mail as outgoing. I need all mail filter rules to apply to outgoing instead of incoming. In the statistics I only see traffic and details for outgoing but it really is all incoming email.

The Mail Gateway uses different ports to decide if the email (smtp) traffic is incoming or outgoing. In your case, you are sending the incoming email traffic to the outgoing port.
=> Just set the ports right. For more details you can check the existing docs about setting up a Proxmox Mail Gateway 4, there is no change here.

2) At Mail proxy Options I set DNSBL Sites, that works but I would also like to add the whitelist "list.dnswl.org" with negative scores like this: zen.spamhaus.org*2 bl.spamcop.net*1 b.barracudacentral.org*1 ix.dnsbl.manitu.net*1 psbl.surriel.com list.dnswl.org=127.0.[0..255].0*-2 list.dnswl.org=127.0.[0..255].1*-3 list.dnswl.org=127.0.[0..255].[2..3]*-4
But the field doesn't allow me to do so:

Parameter verification failed. (400)
dnsbl_sites: invalid format - value 'list.dnswl.org=127.0.[0..255].0*-2' does not look like a valid dnsbl entry

I never tried adding whitelists here. Please note, these whitelists are also queried by the spamassassin rules, so I see no need to add this here.

3) TLS for Mail Proxy is disabled by default. Shouldn't this be enabled by default?

I will pass this idea to the devs.

4) Greylisting can be enabled/disabled globally only. I don't want to use it by default because of the delay in incoming emails (only unknown triplet, I know, but still). But I would like to be able to enable it for some domains. When we add a domain to Relay Domains an option to enable greylisting or not would be really nice.

You cannot enable for some domains, but you can disable it for some.

5) We plan on using many domains of different customers. But they need to be able to view there quarantined email/spam. There's a user role Quarantine Manager but that user can see all quarantined email. Is there some generic solution for this? Or should I change the subject of spam messages to mark them as SPAM: and setup rules on the destination server to move those emails to a spam folder?

Every admin user can see all quarantined emails, currently there is no such limited quarantine admin user. So in your case I suggest you just mark the emails as spam.

Many thanks for the feedback!

 

[hansm]

Thank you for your answers!

The Mail Gateway uses different ports to decide if the email (smtp) traffic is incoming or outgoing. In your case, you are sending the incoming email traffic to the outgoing port.
=> Just set the ports right. For more details you can check the existing docs about setting up a Proxmox Mail Gateway 4, there is no change here.

Thanks! Now it works but I must say it isn't really clear what values to use here, I read the deployment guide and understand the need of setting is but you talk about incoming and outgoing and in PMG it's called internal and external SMTP port. Besides that, I didn't change these settings after installing the ISO. So, defaults seem incorrect?

I never tried adding whitelists here. Please note, these whitelists are also queried by the spamassassin rules, so I see no need to add this here.

I disagree. Using DNSBL/RBL at the mail proxy level (Postscreen is the front line) is far more efficient than letting everything in and let SA do it's very CPU intensive task to determine spam. A DNS lookup is very lightweight and helps reducing system load and handling much more emails on the system. Now imagine we only use blacklists at the front and someones IP is accidently listed on Spamhaus, this email will be blocked directly but the reputation of this IP can be very good and therefore it can be listed at DNSWL.org. So, let Spamhaus set a score of 2 for an IP listed in Spamhaus but let DNSWL extract 2 from the DNSBL score so the result is 0 and email will not be blocked. See http://rob0.nodns4.us/postscreen.html for some example Postscreen configuration, maybe it can help. That site says:

DNSWL.org itself has trust levels. I score their trust level "none" as a -2; "low" as -3, and "medium" or "high" as -4.

You cannot enable for some domains, but you can disable it for some.

How can I do that? I can whitelist or blacklist domains but how do I exclude a recipient domain from greylisting it's incoming email?

 

[tom]

Thank you for your answers!


Thanks! Now it works but I must say it isn't really clear what values to use here, I read the deployment guide and understand the need of setting is but you talk about incoming and outgoing and in PMG it's called internal and external SMTP port. Besides that, I didn't change these settings after installing the ISO. So, defaults seem incorrect?

Depends on your needs, you just have to configure your network and firewall that all fits together, just a needed configuration.

I disagree. Using DNSBL/RBL at the mail proxy level (Postscreen is the front line) is far more efficient than letting everything in and let SA do it's very CPU intensive task to determine spam. A DNS lookup is very lightweight and helps reducing system load and handling much more emails on the system.

Seems you did not understand my answer. DNSBL are used and very effective. You asked about whitelists.

Now imagine we only use blacklists at the front and someones IP is accidently listed on Spamhaus, this email will be blocked directly but the reputation of this IP can be very good and therefore it can be listed at DNSWL.org. So, let Spamhaus set a score of 2 for an IP listed in Spamhaus but let DNSWL extract 2 from the DNSBL score so the result is 0 and email will not be blocked. See http://rob0.nodns4.us/postscreen.html for some example Postscreen configuration, maybe it can help. That site says:

If an IP is blacklisted on Spamhaus, you really want these blocked as there is a reason for the listing.

How can I do that? I can whitelist or blacklist domains but how do I exclude a recipient domain from greylisting it's incoming email?

See "Configuration/Mail Proxy: Whitelist"

 

[hansm]

Depends on your needs, you just have to configure your network and firewall that all fits together, just a needed configuration.

Seems fair. I corrected our settings and everything is working fine now.

Seems you did not understand my answer. DNSBL are used and very effective. You asked about whitelists.

Yes, I asked about DNS based whitelists. But you said:

Please note, these whitelists are also queried by the spamassassin rules, so I see no need to add this here.

DNS based blocklists are also checked by SA but it helps to use them earlier in the process. SA uses the same logic as I mentioned, let RBL's add points and WL extract points. I've never tested this but many sites advises to use this strategy to prevent false negatives from blacklists.
I'm not saying that I'm a big fan of this, I really don't know but would like to test with it but PMG doesn't let me add it. I would really appreciate it if I have the possibility to add it and test with it. More information about DNSBL and WL in postscreen: https://blog.schaal-24.de/mail/postscreen-im-kampf-gegen-spam/?lang=en http://rob0.nodns4.us/postscreen.html https://gist.github.com/stevejenkins/5489071 etc.
Important parameter can be "postscreen_dnsbl_threshold", would be nice if we can set it from web interface.
It's just a suggestion. Not a matter of convincing eachother ;-)

See "Configuration/Mail Proxy: Whitelist"

If I whitelist a recipient domain greylisting isn't applied, I understand that. But aren't there many other checks which are bypassed because of whitelisting? Eg. DNS BL, SPF, SpamAssassin, etc.?

New suggestions:
6) Have you considered Rspamd as alternative to SpamAssassin? It can do the same job with less CPU and memory resources in less time. Please see https://rspamd.com/misc/2016/03/03/rspamd-performance.html and https://rspamd.com/comparison.html

7) I'm used to have the subject of emails in the mail logs, it's missing in PMG's Postfix logs. It would be nice to have it logged because users get lots of email a day sometime and call about missing an email, the subject will help find the right email. It can be done as instructed here https://sumeshprabhu86.wordpress.com/2013/09/25/how-to-add-subject-in-postfix-mail-logs/

8) I'm receiving daily emails from PMG. In the Configuration->Options I've set "Send daily reports" to No, but I still get the emails on the "Administrator Email". Our users (recipient email addresses) also receive a daily email with quarantined email and a link to view it. Can I disable this? I want to do everything as transparent as possible.

 

[tom]

...
DNS based blocklists are also checked by SA but it helps to use them earlier in the process.

that's exactly what we do on the Mail Gateway, I do not understand why you suggest this as its already like this. You just tried to add a complex entry there, maybe you just added something wrong, I cannot verify this here.

If I whitelist a recipient domain greylisting isn't applied, I understand that. But aren't there many other checks which are bypassed because of whitelisting? Eg. DNS BL, SPF, SpamAssassin, etc.?

All SMTP checks are whitelisted. Spamassassin checks will be done as these are later. In normal operations, whitelisting on the SMTP level is not needed, as a correctly configured email domain/sender will not blocked anyways.

New suggestions:
6) Have you considered Rspamd as alternative to SpamAssassin? It can do the same job with less CPU and memory resources in less time. Please see https://rspamd.com/misc/2016/03/03/rspamd-performance.html and https://rspamd.com/comparison.html

We always analyse other available tools and we also know rspamd. but we use spamassassin in the Mail Gateway.

7) I'm used to have the subject of emails in the mail logs, it's missing in PMG's Postfix logs. It would be nice to have it logged because users get lots of email a day sometime and call about missing an email, the subject will help find the right email. It can be done as instructed here https://sumeshprabhu86.wordpress.com/2013/09/25/how-to-add-subject-in-postfix-mail-logs/

I never tried this, I will discuss with devs.

8) I'm receiving daily emails from PMG. In the Configuration->Options I've set "Send daily reports" to No, but I still get the emails on the "Administrator Email". Our users (recipient email addresses) also receive a daily email with quarantined email and a link to view it. Can I disable this? I want to do everything as transparent as possible.

You talk about admin reports for the daily quarantine spam report? you can disable only spam reports. See "Configuration/Spam Detector/Report Style.

Please do not continue asking multiple complex questions in one post, this is hard to follow for others. Better is only one topic per thread.

 

[hansm]

that's exactly what we do on the Mail Gateway, I do not understand why you suggest this as its already like this. You just tried to add a complex entry there, maybe you just added something wrong, I cannot verify this here.

We have a misunderstanding, forget this. The only thing I should have asked is if you can make the field Configuration->Mail Proxy->Options: DNSBL Sites less strict in checking, I would like to add the following entry: zen.spamhaus.org*2 bl.spamcop.net b.barracudacentral.org psbl.surriel.com list.dnswl.org=127.0.[0..255].[2..3]*-4
The field doesn't allow it because of the [] I suppose. I know it's a complex entry but it's allowed to do in main.cf directly, why does PMG block it?

All SMTP checks are whitelisted. Spamassassin checks will be done as these are later. In normal operations, whitelisting on the SMTP level is not needed, as a correctly configured email domain/sender will not blocked anyways.

Correct. I don't want to whitelist domains, you suggested it because I asked about excluding domains from greylisting. I like the greylisting concept for spam fighting purposes but many customers nowadays rely on direct email deliveries, think about webshop accounts with "Forgot password" option, customers want this email within seconds to complete their purchase. Greylisting prevents direct deliveries in many cases. The email will eventually pass if the sender correctly retries sending after a few minutes but there is a delay in mail delivery. I understand greylisting completely. What I asked for is an option to enable/disable greylisting per domain. It's important for us but if you don't see a larger use case for it I totally understand that you're not going to develop this.

You talk about admin reports for the daily quarantine spam report? you can disable only spam reports. See "Configuration/Spam Detector/Report Style.

Found it! Thank you.

Also thanks for the other responses, appreciated. I'll ask other questions (when they come up) in a new topic.

 

[tom]

We have a misunderstanding, forget this. The only thing I should have asked is if you can make the field Configuration->Mail Proxy->Options: DNSBL Sites less strict in checking, I would like to add the following entry: zen.spamhaus.org*2 bl.spamcop.net b.barracudacentral.org psbl.surriel.com list.dnswl.org=127.0.[0..255].[2..3]*-4
The field doesn't allow it because of the [] I suppose. I know it's a complex entry but it's allowed to do in main.cf directly, why does PMG block it?

Please file a request via https://bugzilla.proxmox.com

Correct. I don't want to whitelist domains, you suggested it because I asked about excluding domains from greylisting. I like the greylisting concept for spam fighting purposes but many customers nowadays rely on direct email deliveries, think about webshop accounts with "Forgot password" option, customers want this email within seconds to complete their purchase. Greylisting prevents direct deliveries in many cases. The email will eventually pass if the sender correctly retries sending after a few minutes but there is a delay in mail delivery. I understand greylisting completely. What I asked for is an option to enable/disable greylisting per domain. It's important for us but if you don't see a larger use case for it I totally understand that you're not going to develop this.

I forgot to mention, that a correctly configured email sender will not get greylisted. If the sender has a valid SPF record, no greylisting will apply. And you can disable greylisting for some domains, if you know that you need these emails immediately and the sender is unwilling to configure their DNS correctly.

 

[matze1]

Hi,

I forgot to mention, that a correctly configured email sender will not get greylisted. If the sender has a valid SPF record, no greylisting will apply.

Sorry to highjack this thread, but i noticed greylisting for mails from gmx.de to me.
Incoming GMX-IP ist listed in their SPF-Records and i have configured "Use SPF=no" in pmg because i don't want mails get rejected if their SPF record isn't valid. (Nevertheless later SA scores based on its own spf check)

Does "will not get greylisted if the sender has a valid SPF-Record" only apply with "Use SPF=yes"?

Regards,

Matze

 

[dietmar]

Does "will not get greylisted if the sender has a valid SPF-Record" only apply with "Use SPF=yes"?

yes.

 

[heutger]

You could adjust main.cf.in by yourself (read the manual on how to customize the templates) and then add dns relay whitelists (maybe this word would help the support better or don't we all speak german here: genauso wie es relay blacklists dns-basierend für rbl-checks gibt, gibt es auch entsprechend eine handvoll whitelists) with negative score, the corresponding treshold (if you like) and sync the templates.

However, I don't recommend to use whitelists, look at e.g. http://analyse.inps.de/?type=monthly&lang=de&service=&month=04&year=2018&sort=5, the whitelists with very few entries have very few false positives, but the lists, which should help well, primary list.dnswl.org e.g. had a failure rate of 3% in April, which I believe is too high to e.g. decrease the value of a reputative blocklist like the ones, you're using. Maybe against sorbs or others, I won't use.

 

[heutger]

Regarding greylisting, I'm experimenting with running rspamd side by side with PMG, rspamd allows greylisting to be invoked just on a particular score. It's another approach as SPF is also currently not working for my environment and to prevent, every mail gets greylisted or needs an entry in the whitelist.

 

[ned]

Hi,
Can someone point me to the right direction as I can not find a way to enable greylisting.
Below is the only thing that does appear in the logs and is related to greylisting.

tail -n 100 /var/log/mail.log | grep greylist
Aug 24 16:32:19 xxxx pmgpolicy[352]: starting policy database maintainance (greylist, rbl)

I disabled spf but greylisting is still off, see below

    greylist 1
    spf 0

Version is Mail Gateway 5.0-76