Proxmox VE 9 – Sustained TLS Transfers Fail With
Summary
After moving this machine from Proxmox VE 8 to Proxmox VE 9, sustained TLS/HTTPS transfers became unreliable.Large HTTPS transfers fail at random offsets with errors such as:
Decryption has failed.
and:
OpenSSL SSL_read: OpenSSL/3.5.7:
error:0A000119:SSL routines::decryption failed or bad record mac
The problem affects:
- outbound HTTPS downloads from the PVE host;
- inbound HTTPS ISO uploads through the PVE web interface.
The same physical machine/network worked correctly under PVE 8. The problem appeared after moving to PVE 9.
Environment
Fresh Proxmox VE 9.2.2 install.Kernels tested:
7.0.2-6-pve FAIL
7.0.14-15-pve FAIL
6.14.11-9-pve FAIL
The older 6.14 kernel was explicitly installed and booted for comparison.
NICs tested
Originally used the motherboard Realtek NIC.Because Realtek issues were suspected, I replaced it with an Intel dual-port server NIC using:
driver: igb
Speed: 1000Mb/s
Duplex: Full
Link detected: yes
The same problem occurs with both NICs.
Intel NIC counters remain clean:
rx_crc_errors: 0
rx_errors: 0
tx_errors: 0
tx_dropped: 0
rx_frame_errors: 0
rx_fifo_errors: 0
tx_fifo_errors: 0
rx_queue_*_drops: 0
rx_queue_*_csum_err: 0
tx_timeout_count: 0
wget / GnuTLS reproduction
Directly on the PVE host:wget -4 \
https://releases.ubuntu.com/26.04.1/ubuntu-26.04.1-live-server-amd64.iso \
-O /dev/null
Fails at random offsets, e.g.:
Read error at byte 141655877/2927861760
(Decryption has failed.)
Read error at byte 255287687/2927861760
(Decryption has failed.)
Read error at byte 264687877/2927861760
(Decryption has failed.)
Retries may progress further, then fail again at another random offset.
curl / OpenSSL reproduction
Using curl backed by OpenSSL 3.5.7:curl -4 -L --http1.1 \
https://releases.ubuntu.com/26.04.1/ubuntu-26.04.1-live-server-amd64.iso \
-o /dev/null
Example:
168M transferred
curl: (56) OpenSSL SSL_read:
OpenSSL/3.5.7:
error:0A000119:
SSL routines::decryption failed or bad record mac
Another run failed after only ~64 MB.
Therefore this is not specific to wget/GnuTLS:
wget / GnuTLS FAIL
curl / OpenSSL FAIL
TLS versions
TLS 1.2:curl -4 -L --http1.1 \
--tlsv1.2 --tls-max 1.2 \
https://releases.ubuntu.com/26.04.1/ubuntu-26.04.1-live-server-amd64.iso \
-o /dev/null
Result:
OpenSSL SSL_read:
error:1C800066:
Provider routines::cipher operation failed
TLS 1.3:
curl -4 -L --http1.1 \
--tlsv1.3 --tls-max 1.3 \
https://releases.ubuntu.com/26.04.1/ubuntu-26.04.1-live-server-amd64.iso \
-o /dev/null
Result:
SSL routines::decryption failed or bad record mac
ChaCha20-Poly1305 was also tested instead of the normal AES-GCM path and also failed.
Therefore this is not specific to TLS 1.2, TLS 1.3, or AES-GCM.
PVE GUI upload failure
Uploading an ISO from a Windows PC through the PVE web interface also fails with:Error '0' occurred while receiving the document.
So both practical directions are affected:
Internet -> PVE HTTPS download FAIL
PC -> PVE HTTPS GUI upload FAIL
An nginx reverse proxy was also tested as an alternative TLS termination point in front of pveproxy. ISO uploads still failed.
Plain TCP/HTTP works perfectly
A 1 GB random file was created on PVE:dd if=/dev/urandom of=/tmp/net-test.bin bs=1M count=1024
sha256sum /tmp/net-test.bin
Hash:
72e10b9df9bd4ad4bfd99467c4d3705501068629aa5d7fa49413b453ba4d579a
PVE -> Windows
Served via plain HTTP:python3 -m http.server 8080 --bind 0.0.0.0
Windows downloaded the complete 1 GB at ~39 MB/s.
Windows SHA256:
72E10B9DF9BD4AD4BFD99467C4D3705501068629AA5D7FA49413B453BA4D579A
Exact match.
Windows -> PVE
The same file was then served from Windows and downloaded by PVE:curl http://192.168.40.216:8081/net-test.bin \
-o /tmp/from-windows.bin
Completed at ~44 MB/s.
SHA256 on PVE:
72e10b9df9bd4ad4bfd99467c4d3705501068629aa5d7fa49413b453ba4d579a
Exact match.
Therefore:
PVE -> Windows plain TCP/HTTP PASS
Windows -> PVE plain TCP/HTTP PASS
SHA256 integrity both ways PASS
Local TLS works perfectly
A 2 GB file was served locally with OpenSSL:truncate -s 2G /tmp/tls-test.bin
openssl s_server \
-accept 8443 \
-cert tls-test.crt \
-key tls-test.key \
-WWW \
-quiet
Then:
curl -k https://127.0.0.1:8443/tls-test.bin -o /dev/null
This completed successfully.
Repeated three more times.
Approximate speeds:
1974 MB/s PASS
1366 MB/s PASS
1911 MB/s PASS
1935 MB/s PASS
So approximately 8 GB of loopback TLS traffic passed with no authentication/decryption errors.
This gives the unusual combination:
Plain network traffic PASS
TLS over loopback PASS
External TLS traffic FAIL
RAM testing
Installed RAM:2 x 8 GB DDR4-3200
Tested:
- XMP disabled;
- each DIMM individually;
- known-good RAM;
- failure unchanged.
MTU and offloads
MTU was tested at both 1420 and normal 1500.No difference.
Relevant Intel NIC offloads were disabled:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
hw-tc-offload: off
HTTPS still failed immediately afterward.
Behaviour matrix
| Test | Result |
|---|---|
| PVE 8 on same machine historically | PASS |
| PVE 9 HTTPS download | FAIL |
| wget / GnuTLS | FAIL |
| curl / OpenSSL | FAIL |
| TLS 1.2 | FAIL |
| TLS 1.3 | FAIL |
| ChaCha20 alternative | FAIL |
| PVE GUI ISO upload | FAIL |
| nginx front-end upload | FAIL |
| PVE -> Windows plain HTTP 1 GB | PASS |
| SHA256 PVE -> Windows | EXACT |
| Windows -> PVE plain HTTP 1 GB | PASS |
| SHA256 Windows -> PVE | EXACT |
| Local TLS ~8 GB total | PASS |
| Realtek NIC | FAIL |
| Intel igb NIC | FAIL |
| Kernel 7.0.2-6 | FAIL |
| Kernel 7.0.14-15 | FAIL |
| Kernel 6.14.11-9 | FAIL |
| Known-good RAM | FAIL |
| XMP disabled | FAIL |
| Common NIC offloads disabled | FAIL |
Current working theory
This does not behave like ordinary Ethernet corruption.Plain TCP can move at least 1 GB in either direction with exact SHA256 integrity.
TLS can also process multiple gigabytes correctly over loopback.
The failure only appears when sustained TLS traffic crosses the real network path on PVE 9.
Because the same hardware previously worked correctly under PVE 8, I suspect a PVE 9 / Debian 13 Trixie interaction involving networking, buffer/DMA handling, kernel/userspace interaction, or another low-level platform compatibility issue.
I have not identified the exact root cause.
Impact
Affected:- PVE ISO downloads;
- large HTTPS downloads from host;
- ISO uploads through PVE GUI;
- potentially other sustained encrypted transfers/backups.
Request
Has anyone reproduced this on PVE 9, or is there a known PVE 9/Trixie issue where sustained external TLS traffic produces random bad record mac / decryption failures while:plain TCP both directions PASS
TLS over loopback PASS
external TLS FAIL
I can provide targeted lspci, CPU/motherboard info, package versions, packet captures or kernel traces if useful.