I tried solving this for 3 days straight, but I surrendered. Read all threads/posts on the forum and tried all suggestions but I didn't solve anything.
Here's some contextual info: PBS runs on a dedicated Intel NUC. I have 2 PVE nodes running on separate physical servers, all three are on the same switch. I have a homelab with a lot of applications/services, no issues regarding networking, firewall etc., everything's working as expected.
log of the error (same for every lxc/vm):
proxmox-backup-manager report
pveversion -v
Here's some contextual info: PBS runs on a dedicated Intel NUC. I have 2 PVE nodes running on separate physical servers, all three are on the same switch. I have a homelab with a lot of applications/services, no issues regarding networking, firewall etc., everything's working as expected.
log of the error (same for every lxc/vm):
Code:
INFO: starting new backup job: vzdump --mailnotification always --all 1 --mailto alessandro.delprete+pbslocal@gmail.com --mode snapshot --quiet 1 --notes-template '{{guestname}}' --storage pbs-local --compress zstd
INFO: skip external VMs: 102, 103, 104, 105, 106, 108, 200, 301
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2022-07-25 05:00:05
INFO: status = running
INFO: CT Name: plex
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nas') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/100/2022-07-25T03:00:05Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 100 --backup-time 1658718005 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/100/2022-07-25T03:00:05Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:07 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_100/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: HTTP/2.0 connection failed
INFO: catalog upload error - channel closed
INFO: Error: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 100 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1005202_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 100 --backup-time 1658718005 --repository root@pam@pbs.axel.dom:pvecluster' failed: exit code 255
INFO: Failed at 2022-07-25 05:00:09
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2022-07-25 05:00:09
INFO: status = running
INFO: CT Name: meshcentral
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/101/2022-07-25T03:00:09Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_101/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1658718009 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/101/2022-07-25T03:00:09Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:09 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_101/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: root.pxar: had to backup 1.04 GiB of 1.042 GiB (compressed 444.231 MiB) in 9.76s
INFO: root.pxar: average backup speed: 109.117 MiB/s
INFO: root.pxar: backup was done incrementally, reused 2.046 MiB (0.2%)
INFO: Uploaded backup catalog (640.848 KiB)
INFO: Duration: 9.92s
INFO: End Time: Mon Jul 25 05:00:19 2022
INFO: adding notes to backup
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 101 (00:00:11)
INFO: Backup finished at 2022-07-25 05:00:20
INFO: Starting Backup of VM 107 (lxc)
INFO: Backup started at 2022-07-25 05:00:20
INFO: status = running
INFO: CT Name: docker2
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nas') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/107/2022-07-25T03:00:20Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_107/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 107 --backup-time 1658718020 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/107/2022-07-25T03:00:20Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:25 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_107/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: HTTP/2.0 connection failed
INFO: catalog upload error - channel closed
INFO: Error: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 107 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1005202_107/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 107 --backup-time 1658718020 --repository root@pam@pbs.axel.dom:pvecluster' failed: exit code 255
INFO: Failed at 2022-07-25 05:00:42
INFO: Starting Backup of VM 109 (lxc)
INFO: Backup started at 2022-07-25 05:00:42
INFO: status = running
INFO: CT Name: jellyfin
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/nas') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/109/2022-07-25T03:00:42Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1005202_109/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 109 --backup-time 1658718042 --repository root@pam@pbs.axel.dom:pvecluster
INFO: Starting backup: ct/109/2022-07-25T03:00:42Z
INFO: Client name: pve2
INFO: Starting backup protocol: Mon Jul 25 05:00:43 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp1005202_109/etc/vzdump/pct.conf' to 'root@pam@pbs.axel.dom:8007:pvecluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs.axel.dom:8007:pvecluster' as root.pxar.didx
INFO: HTTP/2.0 connection failed
INFO: catalog upload error - channel closed
INFO: Error: pipelined request failed: error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1543:SSL alert number 20
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 109 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1005202_109/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 109 --backup-time 1658718042 --repository root@pam@pbs.axel.dom:pvecluster' failed: exit code 255
INFO: Failed at 2022-07-25 05:03:19
INFO: Backup job finished with errors
TASK ERROR: job errorsproxmox-backup-manager report
Code:
= FILES =
$ cat '/etc/hostname'
pbs
$ cat '/etc/hosts'
127.0.0.1 localhost.localdomain localhost
10.1.10.12 pbs.axel.dom pbs
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
$ cat '/etc/network/interfaces'
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet static
address 10.1.10.12/23
gateway 10.1.10.1
$ cat '/etc/proxmox-backup/datastore.cfg'
datastore: pvecluster
comment
gc-schedule daily
keep-daily 7
keep-last 1
keep-monthly 1
keep-weekly 1
path /backups/pvecluster
prune-schedule daily
$ cat '/etc/proxmox-backup/user.cfg'
user: root@pam
email alessandro.delprete@gmail.com
user: alexdelprete@pbs
email alessandro.delprete+pbs@gmail.com
firstname Alessandro
lastname Del Prete
$ cat '/etc/proxmox-backup/acl.cfg'
acl:1:/:alexdelprete@pbs:Admin
$ cat '/etc/proxmox-backup/remote.cfg'
$ cat '/etc/proxmox-backup/sync.cfg'
# file does not exist
$ cat '/etc/proxmox-backup/verification.cfg'
# file does not exist
$ cat '/etc/proxmox-backup/tape.cfg'
# file does not exist
$ cat '/etc/proxmox-backup/media-pool.cfg'
# file does not exist
$ cat '/etc/proxmox-backup/traffic-control.cfg'
# file does not exist
= COMMANDS =
$ `proxmox-backup-manager versions --verbose`
proxmox-backup 2.2-1 running kernel: 5.15.39-1-pve
proxmox-backup-server 2.2.5-1 running version: 2.2.5
pve-kernel-5.15 7.2-6
pve-kernel-helper 7.2-6
pve-kernel-5.13 7.1-9
pve-kernel-5.15.39-1-pve 5.15.39-1
pve-kernel-5.15.35-2-pve 5.15.35-5
pve-kernel-5.13.19-6-pve 5.13.19-15
pve-kernel-5.13.19-1-pve 5.13.19-3
ifupdown2 3.1.0-1+pmx3
libjs-extjs 7.0.0-1
proxmox-backup-docs 2.2.5-1
proxmox-backup-client 2.2.5-1
proxmox-mini-journalreader 1.3-1
proxmox-widget-toolkit 3.5.1
pve-xtermjs 4.16.0-1
smartmontools 7.2-pve3
zfsutils-linux 2.1.4-pve1
$ `proxmox-backup-manager subscription get`
status notfound
message There is no subscription key
serverid 2D7B0570208358864494A2F5ED909942
url https://www.proxmox.com/en/proxmox-backup-server/pricing
$ `df -h`
Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 783M 864K 783M 1% /run
/dev/mapper/pbs-root 930G 12G 919G 2% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/nvme0n1p2 511M 336K 511M 1% /boot/efi
tmpfs 783M 0 783M 0% /run/user/0
$ `lsblk --ascii`
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 953.9G 0 disk
|-nvme0n1p1 259:1 0 1007K 0 part
|-nvme0n1p2 259:2 0 512M 0 part /boot/efi
`-nvme0n1p3 259:3 0 953.4G 0 part
|-pbs-swap 253:0 0 7G 0 lvm [SWAP]
`-pbs-root 253:1 0 930.4G 0 lvm /
$ `ls -l /dev/disk/by-id /dev/disk/by-path`
/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-name-pbs-root -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-name-pbs-swap -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-uuid-LVM-IDO3gFZsQII0u5zBak3NleFWW85YAvxy5bjBOAvaka2jP2q6Xq4LLqcKWXAuK0K9 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 20 09:20 dm-uuid-LVM-IDO3gFZsQII0u5zBak3NleFWW85YAvxyneqsJ0q80lE8kdEZR1tLgIQZR7XUhU0i -> ../../dm-1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 lvm-pv-uuid-ED2kmG-XEGx-0YFW-eqes-P4A8-8Yck-nOwvJe -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 13 Jul 20 09:20 nvme-eui.6479a751a0c006af -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-eui.6479a751a0c006af-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-eui.6479a751a0c006af-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-eui.6479a751a0c006af-part3 -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 13 Jul 20 09:20 nvme-Sabrent_798707171E7400006255 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-Sabrent_798707171E7400006255-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-Sabrent_798707171E7400006255-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Jul 20 09:20 nvme-Sabrent_798707171E7400006255-part3 -> ../../nvme0n1p3
/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root 13 Jul 20 09:20 pci-0000:3a:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 pci-0000:3a:00.0-nvme-1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Jul 20 09:20 pci-0000:3a:00.0-nvme-1-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Jul 20 09:20 pci-0000:3a:00.0-nvme-1-part3 -> ../../nvme0n1p3
$ `zpool status`
$ `zfs list`
= FUNCTIONS =
$ Datastores
pvecluster
Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-2-pve: 5.15.35-5
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
Last edited:
