[SOLVED] Finding the reason for "adding disclaimer failed"?

[poetry]

Hello,
We added a new rule to our server so if the spam score is equal or higher then 2 it will add disclaimer as noted below. It is working on most messages but I just noticed that on message it failed to work properly. Can you give some advice how to find the reason why it failed to add the disclaimer and how can we make it more reliable?

Anonymized log:

Mar 25 21:47:40 ServerName postfix/postscreen[12249]: CONNECT from [SenderIP]:47846 to [ReciverIP]:25
Mar 25 21:47:46 ServerName postfix/postscreen[12249]: PASS NEW [SenderIP]:47846
Mar 25 21:47:46 ServerName postfix/smtpd[16919]: connect from mail.sender.hostname[SenderIP]
Mar 25 21:47:46 ServerName postfix/smtpd[16919]: Anonymous TLS connection established from mail.sender.hostname[SenderIP]: TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)
Mar 25 21:47:47 ServerName pmgpolicy[16209]: SPF says pass
Mar 25 21:47:47 ServerName postfix/smtpd[16919]: NOQUEUE: client=mail.sender.hostname[SenderIP]
Mar 25 21:47:47 ServerName pmg-smtp-filter[17079]: 2021/03/25-21:47:47 CONNECT TCP Peer: "[127.0.0.1]:35368" Local: "[127.0.0.1]:10024"
Mar 25 21:47:47 ServerName pmg-smtp-filter[17079]: 1E0171605CF6F34A1A8: new mail message-id=<MessageID@mail.sender.hostname>
Mar 25 21:47:48 ServerName pmg-smtp-filter[17079]: 1E0171605CF6F34A1A8: SA score=2/5 time=0.785 bayes=0.00 autolearn=no autolearn_force=no hits=BAYES_00(-1.9),FREEMAIL_FORGED_REPLYTO(4),FREEMAIL_REPLYTO(1),KAM_DMARC_STATUS(0.01),SPF_HELO_NONE(0.001),SPF_PASS(-0.5)
Mar 25 21:47:48 ServerName pmg-smtp-filter[17079]: 1E0171605CF6F34A1A8: adding disclaimer failed (rule: SenderWarning)
Mar 25 21:47:48 ServerName postfix/smtpd[17020]: connect from localhost.localdomain[127.0.0.1]
Mar 25 21:47:48 ServerName postfix/smtpd[17020]: 236351E0763: client=localhost.localdomain[127.0.0.1], orig_client=mail.sender.hostname[SenderIP]
Mar 25 21:47:48 ServerName postfix/cleanup[17021]: 236351E0763: message-id=<MessageID@mail.sender.hostname>
Mar 25 21:47:48 ServerName postfix/qmgr[12208]: 236351E0763: from=<fromaddress>, size=3871, nrcpt=1 (queue active)
Mar 25 21:47:48 ServerName postfix/smtpd[17020]: disconnect from localhost.localdomain[127.0.0.1] ehlo=1 xforward=1 mail=1 rcpt=1 data=1 commands=5
Mar 25 21:47:48 ServerName pmg-smtp-filter[17079]: 1E0171605CF6F34A1A8: accept mail to <sendtoaddress> (236351E0763) (rule: default-accept)
Mar 25 21:47:48 ServerName pmg-smtp-filter[17079]: 1E0171605CF6F34A1A8: processing time: 0.849 seconds (0.785, 0.034, 0)
Mar 25 21:47:48 ServerName postfix/smtpd[16919]: proxy-accept: END-OF-MESSAGE: 250 2.5.0 OK (1E0171605CF6F34A1A8); from=<fromaddress> to=<sendtoaddress> proto=ESMTP helo=<mail.sender.hostname>
Mar 25 21:47:48 ServerName postfix/smtpd[16919]: disconnect from mail.sender.hostname[SenderIP] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7
Mar 25 21:47:48 ServerName postfix/smtp[17023]: 236351E0763: to=<sendtoaddress>, relay=RelayIP[RelayIP]:25, delay=0.12, delays=0.01/0/0.01/0.1, dsn=2.6.0, status=sent (250 2.6.0 Ok, message saved <Message-ID: <MessageID@mail.sender.hostname>>)
Mar 25 21:47:48 ServerName postfix/qmgr[12208]: 236351E0763: removed

Here is how the elements are configured:


If I am looking at the message in question here are some details I can share about the message:
Content-Type: text/plain; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Description: Mail message body

Is it possible the content type is not compatible with our disclaimer message?

Thank you for your help!

 

[Stoiko Ivanov]

Hmm -this usually happens if you use non-ascii characters in the disclaimer text (html-encode them instead) .
but sind I don't see the issue here - could you please paste the disclaimer text in code-tags?
I'll try to reproduce the issue

 

[poetry]

Hmm -this usually happens if you use non-ascii characters in the disclaimer text (html-encode them instead) .
but sind I don't see the issue here - could you please paste the disclaimer text in code-tags?
I'll try to reproduce the issue

Here is the disclaimer text.

<!DOCTYPE html>
<html lang="en">
<head>
<title>Warning</title>
</head>
<body>
<div style='border:solid #0000ff 2.0pt;padding:1.0pt 1.0pt 1.0pt 1.0pt'>
<p style='background:#ff0000'>
<span style='font-weight:bold;font-size:12.0pt;color:white'> &nbsp;Sender for this message: __SENDER__
<br>
&nbsp;- We will never ask you for your password to access your mail.
<br>
&nbsp;- Do not open attachments or links unless you identify the sender or you are sure that the content you received is secure.
</span>
</div>
</body>

I did modify the disclaimer yesterday so it's fully html valid (I used this https://validator.w3.org/nu/#textarea) but that did not help. I see that it failed to add the disclaimer properly on 13 messages today. I am also seeing this on some messages it will try to add disclaimer twice:
Mar 26 10:45:31 pmg-smtp-filter[21163]: 1E1642605DAD3A233F0: adding disclaimer failed (rule: SenderWarning)
Mar 26 10:45:31 pmg-smtp-filter[21163]: 1E1642605DAD3A233F0: added disclaimer (rule: SenderWarning)

Why would it try to add disclaimer twice? I think it has something to do with the format of the message. If it's plain text email then it cannot add html type disclaimer only plain text one. Maybe I am wrong.

The main problem I have is that it's not adding disclaimer properly on all messages. We need this to work reliably otherwise it's not useful and could be dangerous for receivers of email once they get used to getting warnings on dangerous emails. If there is no disclaimer they might think the message is safe but it's not.

Also before the message I can see that it will add this:

--

It's not part of the disclaimer code but it looks something proxmox is adding?

Thank you for your help @Stoiko Ivanov I can always count on you!

 

[poetry]

@Stoiko Ivanov I have done some more testing and made a small modification to the disclaimer text
Here is the latest version:

<!DOCTYPE html>
<html lang="en">
<head>
<title>*Warning*</title>
</head>
<body>
<div style='border:solid #0000ff 2.0pt;padding:1.0pt 1.0pt 1.0pt 1.0pt'>
<p style='background:#ff0000'>
<span style='font-size:12.0pt;color:white'><b>&nbsp;Sender for this message: __SENDER__
<br>
&nbsp;- We will never ask you for your password to access mail.
<br>
&nbsp;- Do not open attachments or links unless you identify the sender or you are sure that the content you received is safe.
</b></span>
</div>
</body>

I have tested sending test email from gmail that is send as HTML and it's added fine you can see in the logs two entries:

Mar 27 14:35:33 pmg-smtp-filter[26704]: 1E1516605F34A499133: added disclaimer (rule: SenderWarning)
Mar 27 14:35:33 pmg-smtp-filter[26704]: 1E1516605F34A499133: added disclaimer (rule: SenderWarning)

In the message you will get the fully formatted HTML warning (red background and white text):
--
Sender for this message: __SENDER__
- We will never ask you for your password to access mail.
- Do not open attachments or links unless you identify the sender or you are sure that the content you received is safe.

If you send the email as plain text then there is only 1 line in the logs:

Mar 27 14:34:06 MX1 pmg-smtp-filter[26687]: 1E1516605F344D560D0: added disclaimer (rule: SenderWarning)

In the message you will get the plain text warning: (can't do bold text as it's plain text)
--
*Warning* Sender for this message: __SENDER__
- We will never ask you for your password to access mail.
- Do not open attachments or links unless you identify the sender or you are sure that the content you received is safe.


I managed to find some information about the messages that failed to add disclaimer:
This might help you replicate the problem. I am guessing that html is not supported with this content type?
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: base64

Content-type: text/plain; charset=us-ascii
Content-Transfer-Encoding: base64

MIME-Version: 1.0
Content-Type: multipart/mixed;

MIME_BASE64_TEXT 1.741 Message text disguised using base64 encoding

Thank you for your help!

 

[poetry]

@Stoiko Ivanov as our policy has changed now we have to add disclaimer to every messages I see a lot more failures to add disclaimer. I have send your the logs via email. Hope you can reproduce the issue. I wish we could have very low or close to 0 failure rate to add disclaimer. Some malicious senders will change their content/format type to avoid detection or adding of disclaimers. I will try to send you more if I can replicate and catch them in the quarantine. It's hard to do that if someone has an idea how to catch every failure like copy the message to quarantine and still deliver the message let me know. I cannot replicate the logs if I deliver from quarantine.

EDIT: Still failing after html-encoding all html characters that could be problematic so it must be something else...

 

[Stoiko Ivanov]

I have send your the logs via email.

Please keep everything in the forum as far as possible - If you need to share confidential data and don't want to anonymize it - I would recommend getting a Subscription of level Basic or above and contacting our enterprise support.
Else keeping everything in the forum has the huge advantage that the whole community can see how the issue looks like and profit from the solution (without us here having the additional work of summing up what's going on apart from the forum)

That being said - my guess from a quick look at 2 of the mails: They use 'us-ascii' as character set - your disclaimer contains characters not representable in that charset - thus PMG does not add the disclaimer (to not modify the original mail) - using HTML entities does not change that.

If you really need to add a disclaimer to every single mail - stick with ascii characters

I hope this helps!

 

[poetry]

your disclaimer contains characters not representable in that charset - thus PMG does not add the disclaimer (to not modify the original mail) - using HTML entities does not change that.

If you really need to add a disclaimer to every single mail - stick with ascii characters

Will test as you said but I don't think it's going to help. I had disclaimer before with only ascii characters and it was still failing on some messages.
If I test my html code or just the disclaimer via https://onlineasciitools.com/validate-ascii I get


Will report after a few days of running it with the changes to ascii characters only.
I am only using characters noted in the image below this should be OK.

 

[poetry]

@Stoiko Ivanov I really do not understand because now it's seems to be working fine. Previous version of the disclaimer was failing even if it did not had any non ascii characters.
I did remove some code from disclaimer like <div> so that might helped...
I agree that we should share as much as we can on the forum I will keep that in mind for the future.

Here is the code for someone if they want to reuse the disclaimer

<title>***Warning***</title><br><p style="background-color: #ffdf00; font-family: Merriweather, Georgia, Arial, Helvetica; border: 8px solid #ffdf00; font-size: 11pt; color: #000000; font-weight: bold;"><u>Email from</u>: <a href="#" style="text-decoration: none; font-weight: normal; font-size: 12pt; font-style: italic;">__SENDER__</a> | <i>Don't reply</i> and <i>do not open</i> <u>attachments</u> and <u>links</u> if you do not recognize the specified sender or if you are not convinced that the received content is safe.</p>

You can use https://html.onlineviewer.net/ to visualize the code is super helpful.

The only thing that is bothering me right now that it's adding a br only on html messages. And no it's not because of this <title>***Warning***</title><br> because I have tested and run it without this code and it's the same.

I am guessing the macro __SENDER__ is adding a <br> because I see the same happening on

Notification I have set for virus senders adding <br> for no reason

Code for this (no br anywhere)

Your message to __RECEIVERS__ was blocked by the mail filtering system. If applicable resend your message following the solution below.

Summary:
Message subject: __SUBJECT__
Receiver mail address: __RECEIVERS__
Sender mail address: __SENDER__
Sender IP: __SENDER_IP__
Message ID: __MSGID__
Cause:

 

[max046]

Good day!
@Stoiko Ivanov, @poetry I encountered the same problem, I did not set the diclaimer to the same form as stated here, but I continue to experience this problem. disclaimer added every other time. How to fix it?

<title>***Warning***</title><br>
<p style="background-color: #FF0000; font-family: Merriweather, Georgia, Arial, Helvetica; border: 8px solid #FF0000; font-size: 11pt; color: #000000; font-weight: bold;"><u>CAUTION: Это электронное письмо было отправлено за пределами организации отправителем </u>: <a href="#" style="text-decoration: none; font-weight: normal; font-size: 12pt; font-style: italic;">__SENDER__</a> | <i>Не переходите по ссылкам, не сообщайте пароль</i> и <i>не открывайте вложения, если не узнаете отправителя и сообщите коллегам из ИТ на <u>XXX@domain.com</u> </i> <i>Помните,</i> что контент может быть небезопасен. </p>
​

 

[Stoiko Ivanov]

The disclaimer only gets added if the mail comes with a compatible encoding set - things that might help is writing all non-ascii characters as html-escapes (this includes all Cyrillic letters) - I hope this helps!

 

[max046]

@Stoiko Ivanov I noticed that this problem is if the letter contains Content-Transfer-Encoding: 7bit, is it possible to fix this if you try to create a filter that will replace the Content-Transfer-Encoding of the incoming letter with base64?

 

[Stoiko Ivanov]

s it possible to fix this if you try to create a filter that will replace the Content-Transfer-Encoding of the incoming letter with base64?

no currently it's not possible as PMG will not change the encoding of the incoming mail

 

[max046]

The disclaimer only gets added if the mail comes with a compatible encoding set - things that might help is writing all non-ascii characters as html-escapes (this includes all Cyrillic letters) - I hope this helps!

@Stoiko Ivanov can you help me perform html-escapes for my disclaimer?

 

[max046]

Have a nice day, everyone. Executed disclamer in the form:

<title>***Warning***</title><br>
<p style="background-color: #FF0000; font-family: Merriweather, Georgia, Arial, Helvetica; border: 8px solid #FF0000; font-size: 11pt; color: #000000; font-weight: bold;">
    <u>CAUTION: &#1069;&#1090;&#1086; &#1101;&#1083;&#1077;&#1082;&#1090;&#1088;&#1086;&#1085;&#1085;&#1086;&#1077; &#1087;&#1080;&#1089;&#1100;&#1084;&#1086; &#1073;&#1099;&#1083;&#1086; &#1086;&#1090;&#1087;&#1088;&#1072;&#1074;&#1083;&#1077;&#1085;&#1086; &#1079;&#1072; &#1087;&#1088;&#1077;&#1076;&#1077;&#1083;&#1072;&#1084;&#1080; &#1086;&#1088;&#1075;&#1072;&#1085;&#1080;&#1079;&#1072;&#1094;&#1080;&#1080; &#1086;&#1090;&#1087;&#1088;&#1072;&#1074;&#1080;&#1090;&#1077;&#1083;&#1077;&#1084;</u> :
    <a href="#" style="text-decoration: none; font-weight: normal; font-size: 12pt; font-style: italic;">__SENDER__</a> |
    <i>&#1053;&#1077; &#1087;&#1077;&#1088;&#1077;&#1093;&#1086;&#1076;&#1080;&#1090;&#1077; &#1087;&#1086; &#1089;&#1089;&#1099;&#1083;&#1082;&#1072;&#1084;, &#1085;&#1077; &#1089;&#1086;&#1086;&#1073;&#1097;&#1072;&#1081;&#1090;&#1077; &#1087;&#1072;&#1088;&#1086;&#1083;&#1100;</i> &#1080;
    <i>&#1085;&#1077; &#1086;&#1090;&#1082;&#1088;&#1099;&#1074;&#1072;&#1081;&#1090;&#1077; &#1074;&#1083;&#1086;&#1078;&#1077;&#1085;&#1080;&#1103;, &#1077;&#1089;&#1083;&#1080; &#1085;&#1077; &#1091;&#1079;&#1085;&#1072;&#1077;&#1090;&#1077; &#1086;&#1090;&#1087;&#1088;&#1072;&#1074;&#1080;&#1090;&#1077;&#1083;&#1103;.</i>
    <i>&#1055;&#1086;&#1084;&#1085;&#1080;&#1090;&#1077;,</i> &#1095;&#1090;&#1086; &#1082;&#1086;&#1085;&#1090;&#1077;&#1085;&#1090; &#1084;&#1086;&#1078;&#1077;&#1090; &#1073;&#1099;&#1090;&#1100; &#1085;&#1077;&#1073;&#1077;&#1079;&#1086;&#1087;&#1072;&#1089;&#1077;&#1085;.
</p>

and this helped solve the problem with adding a disclaimer to each incoming letter

 

[max046]

Have a nice day, everyone. Executed disclamer in the form:

<title>***Warning***</title><br>
<p style="background-color: #FF0000; font-family: Merriweather, Georgia, Arial, Helvetica; border: 8px solid #FF0000; font-size: 11pt; color: #000000; font-weight: bold;">
    <u>CAUTION: &#1069;&#1090;&#1086; &#1101;&#1083;&#1077;&#1082;&#1090;&#1088;&#1086;&#1085;&#1085;&#1086;&#1077; &#1087;&#1080;&#1089;&#1100;&#1084;&#1086; &#1073;&#1099;&#1083;&#1086; &#1086;&#1090;&#1087;&#1088;&#1072;&#1074;&#1083;&#1077;&#1085;&#1086; &#1079;&#1072; &#1087;&#1088;&#1077;&#1076;&#1077;&#1083;&#1072;&#1084;&#1080; &#1086;&#1088;&#1075;&#1072;&#1085;&#1080;&#1079;&#1072;&#1094;&#1080;&#1080; &#1086;&#1090;&#1087;&#1088;&#1072;&#1074;&#1080;&#1090;&#1077;&#1083;&#1077;&#1084;</u> :
    <a href="#" style="text-decoration: none; font-weight: normal; font-size: 12pt; font-style: italic;">__SENDER__</a> |
    <i>&#1053;&#1077; &#1087;&#1077;&#1088;&#1077;&#1093;&#1086;&#1076;&#1080;&#1090;&#1077; &#1087;&#1086; &#1089;&#1089;&#1099;&#1083;&#1082;&#1072;&#1084;, &#1085;&#1077; &#1089;&#1086;&#1086;&#1073;&#1097;&#1072;&#1081;&#1090;&#1077; &#1087;&#1072;&#1088;&#1086;&#1083;&#1100;</i> &#1080;
    <i>&#1085;&#1077; &#1086;&#1090;&#1082;&#1088;&#1099;&#1074;&#1072;&#1081;&#1090;&#1077; &#1074;&#1083;&#1086;&#1078;&#1077;&#1085;&#1080;&#1103;, &#1077;&#1089;&#1083;&#1080; &#1085;&#1077; &#1091;&#1079;&#1085;&#1072;&#1077;&#1090;&#1077; &#1086;&#1090;&#1087;&#1088;&#1072;&#1074;&#1080;&#1090;&#1077;&#1083;&#1103;.</i>
    <i>&#1055;&#1086;&#1084;&#1085;&#1080;&#1090;&#1077;,</i> &#1095;&#1090;&#1086; &#1082;&#1086;&#1085;&#1090;&#1077;&#1085;&#1090; &#1084;&#1086;&#1078;&#1077;&#1090; &#1073;&#1099;&#1090;&#1100; &#1085;&#1077;&#1073;&#1077;&#1079;&#1086;&#1087;&#1072;&#1089;&#1077;&#1085;.
</p>

and this helped solve the problem with adding a disclaimer to each incoming letter

@Stoiko Ivanov it didn't really help. Moreover, we cannot understand the dependence. For example, there is a letter like:

Content-Type: text/html;
    charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

in one case the disclaimer is added, in the other it is not. attached 2 files with detailed headers. I don't understand what could be wrong
Please help me understand what the problem is.