SSL wants a read first

[Hunduster]

Hello everyone,

I am running two PMGs in a cluster and have been having problems receiving emails with node2 for some time.

The test from internet.nl claims that the PMG does not offer STARTTLS and does not test any further. The test from checktls.com, on the other hand, shows the following error:

[000.000]		Trying TLS on *******[1.2.3.4:25] (20)
[000.099]		Server answered
[000.211]	<‑‑	220 nice.smtp.banner
[000.211]		We are allowed to connect
[000.212]	‑‑>	EHLO www12-azure.checktls.com
[000.307]	<‑‑	250-nice.smtp.banner 250-PIPELINING 250-SIZE 25000000 250-ETRN 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-SMTPUTF8 250 CHUNKING
[000.308]		We can use this server
[000.308]		TLS is an option on this server
[000.308]	‑‑>	STARTTLS
[000.403]	<‑‑	220 2.0.0 Ready to start TLS
[000.403]		STARTTLS command works on this server
[004.407]		Cannot convert to SSL (reason: SSL wants a read first)
[004.408]		Note: This same test with Format set to "Debug" may show more
[004.408]	‑‑>	MAIL FROM:<test@checktls.com>
[004.408]		Read failed (reason: did not read)
[004.408]	‑‑>	QUIT
[004.409]		Read failed (reason: did not read)

I have gone through the confis of both servers several times and can't see any difference, especially as the two sync anyway.

Unfortunately, I can't do anything with the error message.

 

[Stoiko Ivanov]

Not quite sure what is going on there exactly - but I ran the test from checktls - for a domain of mine - and it seems they are not following all SMTP-standards too much:

500 5.5.2 Error: bad syntaxMAIL FROM:test@checktls.com

maybe their tests can't pass postscreen?

what does mxtoolbox say for your domain? (I think they have a starttls-check too)

 

[Hunduster]

Hello Stoiko,

Thank you very much for your reply.

As described, I have two nodes running. Node1 does not produce a single error message during the test. Neither from internet.nl nor from checktls.com

MXtoolbox on the other hand is fine with both nodes, here node2:



I have just carried out another test at ssl-tools.net. Somehow there really seems to be an issue with the certificate (Let's Encrypt), node2:



I have already completely deleted the certificates once and recreated them. The ACME Challenge also runs without errors.

It really looks as if PMG will not roll out the certificate in this case. All test pages that test TLS report a timeout during the connection. Only MXtoolbox does not report an error.

Connections always get through in any case, I can see this on the upstream firewall.

 

[Hunduster]

I just saw in the Postfix Livelog that whenever a server connects to node2 I get the following error:

"ssl_accept error from"

If that helps with the diganosis.

Mxtoolbox still goes through, but doesn't seem to do TLS or check it.

 

[Stoiko Ivanov]

the postfix warning from dnsblog with the changes in main.cf might give a hint - do you maybe have modified the configuration template?

 

[Hunduster]

I also get this message on node1.

Both nodes use the exact same config.
The reason is the change in the template:

# auto-generated by proxmox

smtp_tls_protocols = !SSLv2, !SSLv3,!TLSv1,!TLSv1.1
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
lmtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
lmtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# Disable VRFY for Postfix

disable_vrfy_command =yes

smtp_tls_ciphers = high
smtp_tls_mandatory_ciphers = high
smtpd_tls_ciphers = high
smtpd_tls_mandatory_ciphers = high
tls_high_cipherlist = ECDHE-ECDSA-AES256-GCM-SHA384:TLS_AES_256_GCM_SHA384:ECDHE-ECDSA-CHACHA20-POLY1305: TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES256-GCM-S>

tls_preempt_cipherlist = yes

smtpd_tls_eecdh_grade = ultra
tls_ssl_options = NO_COMPRESSION
tls_ssl_options = NO_RENEGOTIATION

I have certainly checked and compared the configurations of both nodes 10 times now
Node1 runs perfectly.

I still suspect that the certificate is at fault here, or that it is not being used at all.

 

[Stoiko Ivanov]

I have just carried out another test at ssl-tools.net. Somehow there really seems to be an issue with the certificate (Let's Encrypt), node2:

that test has tls as supported - the only "issue" is that DANE is not deployed (DANE is a not too widely used standard and has nothing to do with basic STARTTLS functionality)

Try connecting to the node from the public internet with openssl - if tls works this should work too:
`openssl s_client -connect <ip.of.your.pmg>:25 -starttls smtp`

share the output - maybe this gives a hint.

 

[Stoiko Ivanov]

I also get this message on node1.

yes - having the 2 options one below the other does not seem to work:
https://www.postfix.org/postconf.5.html#tls_ssl_options

 

[Hunduster]

yes - having the 2 options one below the other does not seem to work:
https://www.postfix.org/postconf.5.html#tls_ssl_options

Thanks for the hint! I will correct it.

that test has tls as supported - the only "issue" is that DANE is not deployed (DANE is a not too widely used standard and has nothing to do with basic STARTTLS functionality)

Try connecting to the node from the public internet with openssl - if tls works this should work too:
`openssl s_client -connect <ip.of.your.pmg>:25 -starttls smtp`

share the output - maybe this gives a hint.

I have implemented Dane. In this case, it wasn't even tested because the test already stops at Cert.

With OpsenSSL I get the full cert chain from Let's Encrypt displayed on Node1.
With Node2, nothing happens after CONNECTED.

If I omit the parameter -starttls, it connects immediately:

CONNECTED(00000003)
20D0778EFFFF0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:354:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 318 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

 

[Stoiko Ivanov]

If I omit the parameter -starttls, it connects immediately:

That's expected - but it fails because there is no SSL-listener there, but a plain-text SMTP session (you need starttls to speak TLS over SMTP with starttls)

is the node which does not work maybe behind some kind of firewall which might interfere with TLS traffic?

How does the openssl command look if you connect from a trusted IP on the internal port?

 

[Hunduster]

Hello Stoiko,

it is quite unpleasant for me but the error was indeed due to the firewall

Yesterday evening, before your post, I had opened a TLS connection from Node1 to Node2 on port 25 and the certificate was rolled out without any problems.

I run both PMGs behind an OPNsense cluster and have also checked everything here more than 3 times and tested all the rules and security mechanisms. Now, towards the end, I simply swiveled the firewall node and lo and behold...it works. Now that I have restarted the original master node, it also works with it. I suspect some kind of error in the session table or something else.

I'm sorry that I didn't think of this earlier and that I've kept you busy with it now

 

[Stoiko Ivanov]

No need to apologize! - Has happened to most of us here (me certainly) and such issues are always very hard to find!

Glad you found the cause - and thanks for reporting back!

 

[Hunduster]

It's always the little things that make a big difference! I have now been able to find out exactly what the problem was: MTU.

With our old firewall, I had set up an MTU of 1412 on the Vodafone connection. I had stupidly adopted this with OPNsense.
Now that I have set the MTU back to 1500, it is stable on all firewall nodes