Backups Failing on all containers/VMs: "HTTP/2.0 connection failed"

alexdelprete

Member
Jan 5, 2022
44
7
13
51
Rome, Italy
I tried solving this for 3 days straight, but I surrendered. Read all threads/posts on the forum and tried all suggestions but I didn't solve anything.

Here's some contextual info: PBS runs on a dedicated Intel NUC. I have 2 PVE nodes running on separate physical servers, all three are on the same switch. I have a homelab with a lot of applications/services, no issues regarding networking, firewall etc., everything's working as expected.

log of the error (same for every lxc/vm):

Code:
INFO: starting new backup job: vzdump --mailnotification always --all 1 --mailto alessandro.delprete+pbslocal@gmail.com --mode snapshot --quiet 1 --notes-template '{{guestname}}' --storage pbs-local --compress zstd
INFO: skip external VMs: 102, 103, 104, 105, 106, 108, 200, 301
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2022-07-25 05:00:05
INFO: status = running
INFO: CT Name: plex
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nas') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/100/2022-07-25T03:00:05Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 100 --backup-time 1658718005 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/100/2022-07-25T03:00:05Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:07 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_100/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: HTTP/2.0 connection failed
INFO: catalog upload error - channel closed
INFO: Error: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 100 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1005202_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 100 --backup-time 1658718005 --repository root@pam@pbs.axel.dom:pvecluster' failed: exit code 255
INFO: Failed at 2022-07-25 05:00:09
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2022-07-25 05:00:09
INFO: status = running
INFO: CT Name: meshcentral
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/101/2022-07-25T03:00:09Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_101/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1658718009 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/101/2022-07-25T03:00:09Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:09 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_101/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: root.pxar: had to backup 1.04 GiB of 1.042 GiB (compressed 444.231 MiB) in 9.76s
INFO: root.pxar: average backup speed: 109.117 MiB/s
INFO: root.pxar: backup was done incrementally, reused 2.046 MiB (0.2%)
INFO: Uploaded backup catalog (640.848 KiB)
INFO: Duration: 9.92s
INFO: End Time: Mon Jul 25 05:00:19 2022
INFO: adding notes to backup
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 101 (00:00:11)
INFO: Backup finished at 2022-07-25 05:00:20
INFO: Starting Backup of VM 107 (lxc)
INFO: Backup started at 2022-07-25 05:00:20
INFO: status = running
INFO: CT Name: docker2
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nas') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/107/2022-07-25T03:00:20Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_107/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 107 --backup-time 1658718020 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/107/2022-07-25T03:00:20Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:25 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_107/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: HTTP/2.0 connection failed
INFO: catalog upload error - channel closed
INFO: Error: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 107 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1005202_107/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 107 --backup-time 1658718020 --repository root@pam@pbs.axel.dom:pvecluster' failed: exit code 255
INFO: Failed at 2022-07-25 05:00:42
INFO: Starting Backup of VM 109 (lxc)
INFO: Backup started at 2022-07-25 05:00:42
INFO: status = running
INFO: CT Name: jellyfin
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nas') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/109/2022-07-25T03:00:42Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_109/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 109 --backup-time 1658718042 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/109/2022-07-25T03:00:42Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:43 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_109/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: HTTP/2.0 connection failed
INFO: catalog upload error - channel closed
INFO: Error: pipelined request failed: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 109 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1005202_109/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 109 --backup-time 1658718042 --repository root@pam@pbs.axel.dom:pvecluster' failed: exit code 255
INFO: Failed at 2022-07-25 05:03:19
INFO: Backup job finished with errors
TASK ERROR: job errors

proxmox-backup-manager report
Code:
= FILES =

$ cat '/etc/hostname'
pbs


$ cat '/etc/hosts'
127.0.0.1 localhost.localdomain localhost
10.1.10.12 pbs.axel.dom pbs

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts


$ cat '/etc/network/interfaces'
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
    address 10.1.10.12/23
    gateway 10.1.10.1


$ cat '/etc/proxmox-backup/datastore.cfg'
datastore: pvecluster
    comment
    gc-schedule daily
    keep-daily 7
    keep-last 1
    keep-monthly 1
    keep-weekly 1
    path /backups/pvecluster
    prune-schedule daily


$ cat '/etc/proxmox-backup/user.cfg'
user: root@pam
    email alessandro.delprete@gmail.com

user: alexdelprete@pbs
    email alessandro.delprete+pbs@gmail.com
    firstname Alessandro
    lastname Del Prete


$ cat '/etc/proxmox-backup/acl.cfg'
acl:1:/:alexdelprete@pbs:Admin


$ cat '/etc/proxmox-backup/remote.cfg'


$ cat '/etc/proxmox-backup/sync.cfg'
# file does not exist

$ cat '/etc/proxmox-backup/verification.cfg'
# file does not exist

$ cat '/etc/proxmox-backup/tape.cfg'
# file does not exist

$ cat '/etc/proxmox-backup/media-pool.cfg'
# file does not exist

$ cat '/etc/proxmox-backup/traffic-control.cfg'
# file does not exist
= COMMANDS =

$ `proxmox-backup-manager versions --verbose`
proxmox-backup             2.2-1        running kernel: 5.15.39-1-pve
proxmox-backup-server      2.2.5-1      running version: 2.2.5
pve-kernel-5.15            7.2-6
pve-kernel-helper          7.2-6
pve-kernel-5.13            7.1-9
pve-kernel-5.15.39-1-pve   5.15.39-1
pve-kernel-5.15.35-2-pve   5.15.35-5
pve-kernel-5.13.19-6-pve   5.13.19-15
pve-kernel-5.13.19-1-pve   5.13.19-3
ifupdown2                  3.1.0-1+pmx3
libjs-extjs                7.0.0-1
proxmox-backup-docs        2.2.5-1
proxmox-backup-client      2.2.5-1
proxmox-mini-journalreader 1.3-1
proxmox-widget-toolkit     3.5.1
pve-xtermjs                4.16.0-1
smartmontools              7.2-pve3
zfsutils-linux             2.1.4-pve1


$ `proxmox-backup-manager subscription get`
status   notfound
message  There is no subscription key
serverid 2D7B0570208358864494A2F5ED909942
url      https://www.proxmox.com/en/proxmox-backup-server/pricing


$ `df -h`
Filesystem            Size  Used Avail Use% Mounted on
udev                  3.8G     0  3.8G   0% /dev
tmpfs                 783M  864K  783M   1% /run
/dev/mapper/pbs-root  930G   12G  919G   2% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2        511M  336K  511M   1% /boot/efi
tmpfs                 783M     0  783M   0% /run/user/0


$ `lsblk --ascii`
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1      259:0    0 953.9G  0 disk
|-nvme0n1p1  259:1    0  1007K  0 part
|-nvme0n1p2  259:2    0   512M  0 part /boot/efi
`-nvme0n1p3  259:3    0 953.4G  0 part
  |-pbs-swap 253:0    0     7G  0 lvm  [SWAP]
  `-pbs-root 253:1    0 930.4G  0 lvm  /


$ `ls -l /dev/disk/by-id /dev/disk/by-path`
/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-name-pbs-root -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-name-pbs-swap -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-uuid-LVM-IDO3gFZsQII0u5zBak3NleFWW85YAvxy5bjBOAvaka2jP2q6Xq4LLqcKWXAuK0K9 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-uuid-LVM-IDO3gFZsQII0u5zBak3NleFWW85YAvxyneqsJ0q80lE8kdEZR1tLgIQZR7XUhU0i -> ../../dm-1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 lvm-pv-uuid-ED2kmG-XEGx-0YFW-eqes-P4A8-8Yck-nOwvJe -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 13 Jul 20 09:20 nvme-eui.6479a751a0c006af -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-eui.6479a751a0c006af-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-eui.6479a751a0c006af-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-eui.6479a751a0c006af-part3 -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 13 Jul 20 09:20 nvme-Sabrent_798707171E7400006255 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-Sabrent_798707171E7400006255-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-Sabrent_798707171E7400006255-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-Sabrent_798707171E7400006255-part3 -> ../../nvme0n1p3

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root 13 Jul 20 09:20 pci-0000:3a:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 pci-0000:3a:00.0-nvme-1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 pci-0000:3a:00.0-nvme-1-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Jul 20 09:20 pci-0000:3a:00.0-nvme-1-part3 -> ../../nvme0n1p3


$ `zpool status`


$ `zfs list`

= FUNCTIONS =

$ Datastores
pvecluster
pveversion -v
Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-2-pve: 5.15.35-5
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
 
Last edited:
I just found out this regarding the specific SSL error 140943FC:

  • SSL Error error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
    Occurs only with pre-shared key authentication. There is a mismatch between the pre-shared key of the TLS server and the client. The specified key may be different, or only one of them has it specified.

Hope someone can point me in the right direction. Thanks.
 
Where do you see this error? I can't find it in the logs you posted.
It seems the http connection gets terminated at some point. Please look at the syslog from around that time, is there anything unusual?
 
Hi Matthias,

I replaced the log on the OP with one of this morning so you can see the detail of the error. This is a backup of node 2 (pve2) which runs VMs 100, 101, 107, 109. They all failed with that SSL error, except for VM 101, which seemed to have worked.

Thanks for the support.
 
The only thing I can think of is regenerating your keys, i.e pvecm updatecerts --force and removing and adding the datastore on PVE.
No idea if that fixes the problem, though.
 
Hi @Matthias. Any suggestion regarding this problem?

Thanks.

one other strange thing is that I have that backup scheduled, that gives errors, and another backup, the remote one, on Tuxis, and the remote one is perfect, never an error, and I backup everything like the local one with errors. does it give you some more info about what could the issue be?
 
Last edited:
The only thing I can think of is regenerating your keys, i.e pvecm updatecerts --force and removing and adding the datastore on PVE.
No idea if that fixes the problem, though.

Hi Matthias. I regenerated the keys as instructed. Removed datastore and backup job, recreated datastore and job, ran it manually and same issue. :(

Since backups to Tuxis datastore are working fine, I thought the issue was the PBS installation, so I decided to reinstall PBS from scratch on the physical server (an Intel NUC), recreated datastore, job, etc. And when I started the job:

Code:
INFO: starting new backup job: vzdump 110 --storage pbs-local --notes-template '{{guestname}}' --mailto xxxx@yyy.com --all 0 --node pve2 --mailnotification always --mode snapshot
INFO: Starting Backup of VM 110 (lxc)
INFO: Backup started at 2022-10-12 14:59:17
INFO: status = running
INFO: CT Name: emqx
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/110/2022-10-12T12:59:17Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2470528_110/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 110 --backup-time 1665579557 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/110/2022-10-12T12:59:17Z
INFO: Client name: pve2
INFO: Starting backup protocol: Wed Oct 12 14:59:17 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp2470528_110/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: HTTP/2.0 connection failed
INFO: catalog upload error - channel closed
INFO: Error: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 110 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp2470528_110/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 110 --backup-time 1665579557 --repository root@pam@pbs.axel.dom:pvecluster' failed: exit code 255
INFO: Failed at 2022-10-12 14:59:20
INFO: Backup job finished with errors
TASK ERROR: job errors

And this is the log on PBS of that job:

Code:
2022-10-12T14:59:17+02:00: starting new backup on datastore 'pvecluster': "ct/110/2022-10-12T12:59:17Z"
2022-10-12T14:59:17+02:00: protocol upgrade done
2022-10-12T14:59:17+02:00: GET /previous_backup_time
2022-10-12T14:59:17+02:00: POST /blob
2022-10-12T14:59:17+02:00: add blob "/backups/pvecluster/ct/110/2022-10-12T12:59:17Z/pct.conf.blob" (350 bytes, comp: 350)
2022-10-12T14:59:17+02:00: POST /dynamic_index
2022-10-12T14:59:17+02:00: POST /dynamic_index
2022-10-12T14:59:17+02:00: created new dynamic index 1 ("ct/110/2022-10-12T12:59:17Z/root.pxar.didx")
2022-10-12T14:59:17+02:00: created new dynamic index 2 ("ct/110/2022-10-12T12:59:17Z/catalog.pcat1.didx")
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: upload_chunk done: 2276076 bytes, a46f0916f9ca3f0638d003e82a7071bc567cfb0393c2cb7a5f8e8b53a5a1f968
2022-10-12T14:59:17+02:00: upload_chunk done: 1556855 bytes, cfeaf28eed91936fc67409f677e9fba63fea109d3a8161e02e6fd8a19e942910
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: upload_chunk done: 2652241 bytes, eb7bac515d3de6e7c257bed4ed9cca1d4a0c1f8b26758ad0dea364f4036aba46
2022-10-12T14:59:17+02:00: upload_chunk done: 6594857 bytes, ffa60fc94d98c4bfc5405c071efc164346f148a841a00d1ada65bf98ff0151f4
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: upload_chunk done: 5077129 bytes, 713b85e39a05c451bb51c0ac527cfe4e3e2f554d86597e1a27a8822ab6da2de9
2022-10-12T14:59:17+02:00: upload_chunk done: 3298708 bytes, 0dc63c39e7e8cee748e781c8b0ff29462a9b373fa8b008b56638b64abdbc28ae
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: upload_chunk done: 2120558 bytes, 0f6d2478f0459aaa200c80764209dabf6063f42d4af7b1bfd01e3e726caa2a73
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: upload_chunk done: 1955106 bytes, c5163ea48b7a267183bd6544e6f01d4679ac0722e827f404f4e3b72ca54b9f93
2022-10-12T14:59:17+02:00: upload_chunk done: 7805691 bytes, daa60ca8b125e8bad217ab994ae43e4390e2d0fb76df57417882ef4295357a89
2022-10-12T14:59:17+02:00: upload_chunk done: 1843193 bytes, 54eba5260248618e1bfff5335b51bfacd9ee31d2fb32c6c34a174278ccb11422
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: upload_chunk done: 3456804 bytes, 8972bef19d4ad2074a9731f46b28049f4bc332b723481a6c31eba87b8d42e31a
2022-10-12T14:59:17+02:00: POST /dynamic_chunk
2022-10-12T14:59:17+02:00: backup failed: connection error: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:
2022-10-12T14:59:17+02:00: removing failed backup
2022-10-12T14:59:17+02:00: POST /dynamic_chunk: 400 Bad Request: error reading a body from connection: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:
2022-10-12T14:59:17+02:00: TASK ERROR: connection error: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:
2022-10-12T14:59:17+02:00: POST /dynamic_chunk: 400 Bad Request: backup already marked as finished.
2022-10-12T14:59:17+02:00: POST /dynamic_chunk: 400 Bad Request: backup already marked as finished.

It doesn't look like a certificate issue, because it starts transmitting chunks...then it stops with the SSL error.

I'm so frustrated...don't know what else to try...:(
 
Last edited:
Are you absolutely certain the hardware is not to blame? Please try connecting the PBS directly to the PVE, no switch or anything in between, and run a backup. If it still doesn't work, use a different cable and try again.

Other than that, I don't know what to do either :(
 
This Intel NUC was previously a PVE node, but I converted it to PBS. Never had any hw issue.

I switched cables, changed switch (I have 2 switches). No tx/rx errors on the switch ports. Tried a CAT6 (high quality) cable.

That's why I'm frustrated...don't know what else I can try. The only thing I didn't try is connecting it directly to the PVE server, because I have only 1 port on the PVE node, and that would bring it offline.

Is there any debug level setting I can configure somewhere to have a more meaningful and verbose log?
 
@Matthias. just to let you know that I tried again today, all PVE nodes updated and PBS too, and it finally all backups completed without errors. :)

Didn't change anything at hw level, I don't know which of the updates of last month did the trick, but it worked. Here's the current versions of all PVE/PBS components:

Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.60-2-pve: 5.15.60-2
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-4
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-5
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

proxmox-backup                2.2-1        running kernel: 5.15.64-1-pve
proxmox-backup-server         2.2.7-1      running version: 2.2.7
pve-kernel-5.15               7.2-13
pve-kernel-helper             7.2-13
pve-kernel-5.15.64-1-pve      5.15.64-1
pve-kernel-5.15.60-2-pve      5.15.60-2
pve-kernel-5.15.35-1-pve      5.15.35-3
ifupdown2                     3.1.0-1+pmx3
libjs-extjs                   7.0.0-1
proxmox-backup-docs           2.2.7-1
proxmox-backup-client         2.2.7-1
proxmox-mini-journalreader    1.2-1
proxmox-offline-mirror-helper 0.5.0-1
proxmox-widget-toolkit        3.5.1
pve-xtermjs                   4.16.0-1
smartmontools                 7.2-pve3
zfsutils-linux                2.1.6-pve1
 
Huh. That's interesting, but certainly good to hear!
Now that it works, please mark the thread as solved (edit it, there's a dropdown near the title) and let's hope it stays that way :)
 
@Matthias. would be interesting to know which update addressed the issue. Where can I see a detail of the updates of proxmox backup client (maybe also the server)?

I can't see a dropdown near the title nor the first post. Where is it? Maybe in the threads view?
 
Last edited:
I'm afraid the details won't help you, but you can take a look with apt changelog proxmox-backup-server.

Our git repository https://git.proxmox.com/ contains all code, so you could also have a look at that, probably best after you have identified the update that caused the issue.

However, the change might not necessarily come from the proxmox-backup-(server|client) package.
 
See also https://forum.proxmox.com/threads/pbs-sync-failed-each-time.113921/#post-492989, I had a similar problem with an Intel NIC and disabled TSO (TCP Segmentation Offload).

You were right, it was the segment offload bug of the e1000 drivers. I used Thomas recommendation to disable it completely: https://forum.proxmox.com/threads/e1000e-reset-adapter-unexpectedly.87769/post-384609

Bash:
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
    address 10.1.10.12/23
    gateway 10.1.10.1
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

I used this for 3 days and backups always complete without a single error, never happened before. :)

Thanks a lot for the heads-up @jenssen99.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!