Backup for a single LXC failing - HTTP/2.0 connection failed

[Stockdoodle]

Running Proxmox 9.2.4 with PBS running as an LXC. All my VMs/Containers back up successfully except for one container. It fails every time, but not at the same place (sometimes it runs 30 minutes, sometimes nearly an hour). It looks like a network error, but the backup server's a container on the Proxmox server. The only difference I can think of is that this particular container has a folder w/ about 2800 files, some of which are tiny and some multiple GBs.

End of PBS task log.

2026-07-11T13:19:42-05:00: successfully added chunk 30a46a20644df1c3c3e286b24ba2c9e1d7c81d26156ee04b01944398967ec917 to dynamic index 2 (offset 610242651881, size 2788116)
2026-07-11T13:19:42-05:00: successfully added chunk 2d848a1e566be0da05742650468361a1641db669ce74190560728586a53fce69 to dynamic index 2 (offset 610245439997, size 3294929)
2026-07-11T13:19:42-05:00: POST /dynamic_chunk
2026-07-11T13:19:42-05:00: upload_chunk done: 11826856 bytes, c7cdf6bea026f52bebd50c3ce65548a84de773b0479d1f29d4ff8e0f820f5934
2026-07-11T13:19:42-05:00: POST /dynamic_chunk
2026-07-11T13:19:42-05:00: backup failed: connection error
2026-07-11T13:19:42-05:00: removing failed backup
2026-07-11T13:19:42-05:00: removing backup snapshot "/mnt/backup/ct/102/2026-07-11T17:13:00Z"
2026-07-11T13:19:42-05:00: TASK ERROR: connection error: error:0A000119:SSL routines:tls_get_more_records:decryption failed or bad record mac:../ssl/record/methods/tls_common.c:873:, error:0A000139:SSL routines::record layer failure:../ssl/record/rec_layer_s3.c:698:
2026-07-11T13:19:42-05:00: POST /dynamic_chunk: 400 Bad Request: error reading a body from connection

PVE task log.

INFO: starting new backup job: vzdump 102 --all 0 --prune-backups 'keep-monthly=1,keep-weekly=1' --fleecing 0 --mode snapshot --notification-mode notification-system --storage pbs --notes-template '{{guestname}}' --node proxmox
INFO: Starting Backup of VM 102 (lxc)
INFO: Backup started at 2026-07-11 12:13:00
INFO: status = running
INFO: CT Name: Stash
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
freeze guest filesystem
thaw guest filesystem
INFO: creating Proxmox Backup Server archive 'ct/102/2026-07-11T17:13:00Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=encrypt --keyfd=17 pct.conf:/var/tmp/vzdumptmp715875_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 102 --backup-time 1783789980 --entries-max 1048576 --repository root@pam@pbs.stockdoodle.net:bu_store
INFO: Starting backup: ct/102/2026-07-11T17:13:00Z
INFO: Client name: proxmox
INFO: Starting backup protocol: Sat Jul 11 12:13:01 2026
INFO: Using encryption key from file descriptor..
INFO: Encryption key fingerprint: f3:ce:d7:87:60:25:50:f5
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp715875_102/etc/vzdump/pct.conf' to 'root@pam@pbs.stockdoodle.net:8007:bu_store' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.stockdoodle.net:8007:bu_store' as root.pxar.didx
INFO: processed 1.982 GiB in 1m, uploaded 1.979 GiB
INFO: processed 3.599 GiB in 2m, uploaded 3.599 GiB
INFO: processed 12.291 GiB in 3m, uploaded 12.033 GiB
INFO: processed 21.614 GiB in 4m, uploaded 21.396 GiB
INFO: processed 30.884 GiB in 5m, uploaded 30.632 GiB
INFO: processed 39.819 GiB in 6m, uploaded 39.575 GiB
INFO: processed 49.425 GiB in 7m, uploaded 49.176 GiB
INFO: processed 58.955 GiB in 8m, uploaded 58.719 GiB
INFO: processed 67.997 GiB in 9m, uploaded 67.771 GiB
INFO: processed 76.872 GiB in 10m, uploaded 76.644 GiB
INFO: processed 86.553 GiB in 11m, uploaded 86.291 GiB
INFO: processed 96.092 GiB in 12m, uploaded 95.837 GiB
INFO: processed 105.12 GiB in 13m, uploaded 104.888 GiB
INFO: processed 114.167 GiB in 14m, uploaded 113.911 GiB
INFO: processed 123.136 GiB in 15m, uploaded 122.911 GiB
INFO: processed 131.844 GiB in 16m, uploaded 131.811 GiB
INFO: processed 141.001 GiB in 17m, uploaded 140.759 GiB
INFO: processed 149.813 GiB in 18m, uploaded 149.596 GiB
INFO: processed 158.655 GiB in 19m, uploaded 158.446 GiB
INFO: processed 168.562 GiB in 20m, uploaded 166.791 GiB
INFO: processed 177.745 GiB in 21m, uploaded 175.8 GiB
INFO: processed 186.523 GiB in 22m, uploaded 184.528 GiB
INFO: processed 195.345 GiB in 23m, uploaded 193.404 GiB
INFO: processed 204.318 GiB in 24m, uploaded 202.339 GiB
INFO: processed 213.38 GiB in 25m, uploaded 211.447 GiB
INFO: processed 222.025 GiB in 26m, uploaded 220.149 GiB
INFO: processed 230.831 GiB in 27m, uploaded 228.858 GiB
INFO: processed 239.77 GiB in 28m, uploaded 237.814 GiB
INFO: processed 248.499 GiB in 29m, uploaded 246.542 GiB
INFO: processed 257.274 GiB in 30m, uploaded 255.334 GiB
INFO: processed 265.948 GiB in 31m, uploaded 264.024 GiB
INFO: processed 274.595 GiB in 32m, uploaded 272.669 GiB
INFO: processed 283.217 GiB in 33m, uploaded 281.256 GiB
INFO: processed 291.827 GiB in 34m, uploaded 289.835 GiB
INFO: processed 300.503 GiB in 35m, uploaded 298.524 GiB
INFO: processed 309.365 GiB in 36m, uploaded 307.274 GiB
INFO: processed 317.953 GiB in 37m, uploaded 315.833 GiB
INFO: processed 326.811 GiB in 38m, uploaded 324.727 GiB
INFO: processed 335.784 GiB in 39m, uploaded 333.743 GiB
INFO: processed 344.847 GiB in 40m, uploaded 342.791 GiB
INFO: processed 353.513 GiB in 41m, uploaded 351.41 GiB
INFO: processed 362.329 GiB in 42m, uploaded 360.217 GiB
INFO: processed 371.445 GiB in 43m, uploaded 369.36 GiB
INFO: processed 380.513 GiB in 44m, uploaded 378.455 GiB
INFO: processed 389.382 GiB in 45m, uploaded 387.317 GiB
INFO: processed 398.059 GiB in 46m, uploaded 396.151 GiB
INFO: processed 407.135 GiB in 47m, uploaded 404.996 GiB
INFO: processed 414.988 GiB in 48m, uploaded 412.678 GiB
INFO: processed 422.633 GiB in 49m, uploaded 420.267 GiB
INFO: processed 430.795 GiB in 50m, uploaded 428.427 GiB
INFO: processed 437.817 GiB in 51m, uploaded 435.438 GiB
INFO: processed 444.412 GiB in 52m, uploaded 442.02 GiB
INFO: processed 451.679 GiB in 53m, uploaded 449.307 GiB
INFO: processed 459.317 GiB in 54m, uploaded 457.062 GiB
INFO: processed 467.535 GiB in 55m, uploaded 465.198 GiB
INFO: processed 475.956 GiB in 56m, uploaded 473.726 GiB
INFO: processed 484.42 GiB in 57m, uploaded 482.066 GiB
INFO: processed 492.865 GiB in 58m, uploaded 490.534 GiB
INFO: processed 501.142 GiB in 59m, uploaded 498.756 GiB
INFO: processed 509.654 GiB in 1h, uploaded 507.268 GiB
INFO: processed 518.222 GiB in 1h 1m, uploaded 515.849 GiB
INFO: processed 527.123 GiB in 1h 2m, uploaded 524.783 GiB
INFO: processed 536.257 GiB in 1h 3m, uploaded 533.912 GiB
INFO: processed 545.372 GiB in 1h 4m, uploaded 542.997 GiB
INFO: processed 553.661 GiB in 1h 5m, uploaded 551.323 GiB
INFO: processed 562.639 GiB in 1h 6m, uploaded 560.345 GiB
INFO: HTTP/2.0 connection failed
INFO: unclosed encoder dropped
INFO: closed encoder dropped with state
INFO: unfinished encoder state dropped
INFO: finished encoder state with errors
INFO: catalog upload error - channel closed
INFO: Error: error:0A0003FC:SSL routines:ssl3_read_bytes:ssl/tls alert bad record mac:../ssl/record/rec_layer_s3.c:918:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 102 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=encrypt' '--keyfd=17' pct.conf:/var/tmp/vzdumptmp715875_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 102 --backup-time 1783789980 --entries-max 1048576 --repository root@pam@pbs.stockdoodle.net:bu_store' failed: exit code 255
INFO: Failed at 2026-07-11 13:19:44
INFO: Backup job finished with errors
INFO: notified via target `gmail`
TASK ERROR: job errors

 

[g.fiore]

The "bad record" error is almost always TSO/GSO/checksum offload misbehaving under certain traffic patterns — the NIC hardware corrupts bytes on the wire, the receiver's TLS MAC fails, connection dies.

What NIC does your PVE host have? `lspci | grep -i eth` and `ethtool -k <nic> | grep -E "tcp-seg|generic-seg|generic-recv"` will tell us. Common culprits are Realtek and Intel e1000e.

If it's one of those, the fix is usually: ethtool -K <nic> tso off gso off gro off

To test right away, no reboot needed. If backups pass with that applied, make it persistent in /etc/network/interfaces with a post-up.

One more question: does the PBS container talk to PVE via its LAN IP, or via a hostname that resolves outside? If the traffic goes out to the router and hairpins back, the physical NIC is involved both ways and offload bugs get amplified.

 

[Stockdoodle]

It's a Realtek card, and I set the PBS storage in PVE to use a hostname. DNS server also lives on the same PVE hardware. I'm turning offloading off and re-running the backup. I'll report back. Thanks for the help.

 

[Stockdoodle]

The "bad record" error is almost always TSO/GSO/checksum offload misbehaving under certain traffic patterns — the NIC hardware corrupts bytes on the wire, the receiver's TLS MAC fails, connection dies.

What NIC does your PVE host have? `lspci | grep -i eth` and `ethtool -k <nic> | grep -E "tcp-seg|generic-seg|generic-recv"` will tell us. Common culprits are Realtek and Intel e1000e.

If it's one of those, the fix is usually: ethtool -K <nic> tso off gso off gro off

To test right away, no reboot needed. If backups pass with that applied, make it persistent in /etc/network/interfaces with a post-up.

One more question: does the PBS container talk to PVE via its LAN IP, or via a hostname that resolves outside? If the traffic goes out to the router and hairpins back, the physical NIC is involved both ways and offload bugs get amplified.

Didn't immediately fix things. The server it's running on has two NICs, the other one being a 1G Intel NIC. I swapped to the Intel one and backups started working. An automatic GC run caused one to fail, but it ran successfully after the GC was finished.

Thanks again for your help.