SSL networking issues

[mervin]

Hi there,

We are experiencing a strange issue with SSL connections, we randomly see one of the following SSL errors when we try to access a certain remote endpoint from inside an LXC container with a Tomcat application installed:

Error: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
Error: error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error
Error: error:04091068:rsa routines:int_rsa_verify:bad signature

The funny thing is that when we try to access to same endpoint on the Proxmox host itself this does not occur. Even more strange on the second Proxmox machine this also does not happen from inside the containers. We have been pulling our hair out for a week now, we have checked everything from firewalls, switches, cables, port settings, MTU etc, etc. The Proxmox versions and kernels are also the same on both machines.

Is there something else on the Proxmox side we can check? We believe this has to be caused by Proxmox due to the fact that it does not happen on the host itself.

These are the Proxmox package versions we are using:

proxmox-ve: 6.4-1 (running kernel: 5.4.203-1-pve)
pve-manager: 6.4-15 (running version: 6.4-15/af7986e6)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.195-1-pve: 5.4.195-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+deb10u1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.14-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1

Thank you in advance!

 

[jduekGuy]

Hi Marvin, is this error constantly? Or does it occur randomly?

Since one week we have some similar (random) errors. We did exactly the same as you but we are unable to trace the error.

Pending: write EPROTO 10E08095FFFF0000:error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:623

I think this error came after the last Proxmox updates.

We are running on the following versions:

proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-7

 

[mervin]

@jduekGuy it happens randomly in our case. Our Proxmox versions differ (we are still on 6.4) but it would be interesting to see if it is indeed related to recent updates.

 

[jduekGuy]

@jduekGuy it happens randomly in our case. Our Proxmox versions differ (we are still on 6.4) but it would be interesting to see if it is indeed related to recent updates.

Check! And what kind of NICs do you have?

 

[mervin]

@jduekGuy the server we are using is an HP server with NX3031 Multifunction 1/10-Gigabit Server Adapter ports in a bonded setup across two switches. I've checked the updates from the previous months, the only thing that could be related are openssl updates, however these are standard Debian security updates and are installed on both our servers on the 26th of December not only the one with the issue.

 

[mervin]

For the time being we have cloned some containers to another Proxmox host, which does not have this issue.
We are still searching for the root cause, it's a very strange issue.