Problem with restoring backups from PBS (Proxmox Backup Server).

S

Slavickox

New Member
Oct 14, 2024
18
0
1
First of all, I’d like to mention that I’m new here, and I’m not sure if this is the right section for this type of question.
So, I have a problem with restoring a backup from PBS. Below are error code: one from the UI and one from the CLI.
Code: 
recovering backed-up configuration from 'PBS:backup/ct/201/2024-10-12T19:00:43Z'
  Logical volume "vm-201-disk-0" created.
Creating filesystem with 1048576 4k blocks and 262144 inodes
Filesystem UUID: 0c547c1a-c448-4613-a423-c03002b06de0
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736
restoring 'PBS:backup/ct/201/2024-10-12T19:00:43Z' now..
HTTP/2.0 connection failed
Error: error extracting archive - encountered unexpected error during extraction: error at entry "gravity.db": failed to extract file: failed to copy file contents: error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:622:
  Logical volume "vm-201-disk-0" successfully removed.
TASK ERROR: unable to restore CT 201 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/201/2024-10-12T19:00:43Z root.pxar /var/lib/lxc/201/rootfs --allow-existing-dirs --repository root@pam@192.168.1.199:Backup' failed: exit code 255

Code: 
root@Proxmox:~# lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/201/2024-10-12T19:00:43Z root.pxar /var/lib/lxc/201/rootfs --allow-existing-dirs --repository root@pam@192.168.1.199:Backup
Password for "root@pam": **********
fingerprint: eb:8c:0d:e6:84:a8:5c:16:1e:23:0e:9b:e1:cb:f6:3b:9d:e1:dc:bc:28:6f:9a:84:43:b1:33:1d:9e:29:6d:05
Are you sure you want to continue connecting? (y/n): y
Permission denied (os error 13)
fingerprint: eb:8c:0d:e6:84:a8:5c:16:1e:23:0e:9b:e1:cb:f6:3b:9d:e1:dc:bc:28:6f:9a:84:43:b1:33:1d:9e:29:6d:05
Are you sure you want to continue connecting? (y/n): y
Permission denied (os error 13)
storing login ticket failed: $XDG_RUNTIME_DIR (`/run/user/0`) must be accessible by the current user (error: Permission denied (os error 13))
Error: error extracting archive - encountered unexpected error during extraction: error at entry "bin": failed to extract symlink: EACCES: Permission denied

All backups are verified and are fine. I also checked older versions, and it's the same. I spent several hours looking for answers but found nothing. Please help me quickly.
 
Last edited:
Check the system journal on the PBS side for errors around the time of the connection failure. Did you already double check the fingerprint as configured for the PBS storage on the PVE side. Did you recently regenerate the PBS certificates or switch from self signed to acme generated certificates?
 
Chris said:
Check the system journal on the PBS side for errors around the time of the connection failure. Did you already double check the fingerprint as configured for the PBS storage on the PVE side. Did you recently regenerate the PBS certificates or switch from self signed to acme generated certificates?
Click to expand...
This is the log from PBS. I checked the fingerprint; I regenerated the certificate three times and checked the fingerprint each time, twice.
This error also occurred before regenerating the certificate.

Code: 
Oct 14 12:29:19 pbs proxmox-backup-proxy[695]: starting new backup reader datastore 'Backup': "/"
Oct 14 12:29:19 pbs proxmox-backup-proxy[695]: protocol upgrade done
Oct 14 12:29:20 pbs proxmox-backup-proxy[695]: GET /download
Oct 14 12:29:20 pbs proxmox-backup-proxy[695]: download "/ct/201/2024-10-12T19:00:43Z/index.json.blob"
Oct 14 12:29:20 pbs proxmox-backup-proxy[695]: GET /download
Oct 14 12:29:20 pbs proxmox-backup-proxy[695]: download "/ct/201/2024-10-12T19:00:43Z/pct.conf.blob"
Oct 14 12:29:20 pbs proxmox-backup-proxy[695]: reader finished successfully
Oct 14 12:29:20 pbs proxmox-backup-proxy[695]: TASK OK
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: starting new backup reader datastore 'Backup': "/"
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: protocol upgrade done
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: GET /download
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: download "/ct/201/2024-10-12T19:00:43Z/index.json.blob"
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: GET /download
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: download "/ct/201/2024-10-12T19:00:43Z/pct.conf.blob"
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: reader finished successfully
Oct 14 12:29:23 pbs proxmox-backup-proxy[695]: TASK OK
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: starting new backup reader datastore 'Backup': "/"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: protocol upgrade done
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /download
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download "/ct/201/2024-10-12T19:00:43Z/index.json.blob"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /download
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download "/ct/201/2024-10-12T19:00:43Z/root.pxar.didx"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: register chunks in 'root.pxar.didx' as downloadable.
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /chunk
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download chunk "/.chunks/395b/395bc11a4bcddb8290c53f85565ed13008a14db95b86fd8bb4095f8406c5082e"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /chunk
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download chunk "/.chunks/bfaa/bfaa417024285ec97c1b5f5e77c73a6d39e6741a4cc7878ec4d4081c194a58a8"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /chunk
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download chunk "/.chunks/8059/80597666740bc1153a2a8a27b66dac8918277514ad1c5de1507b98dfd22d6b66"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /chunk
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download chunk "/.chunks/91e5/91e571c94e58e91aff6d99c84e57f7a6fd0581a7e12bdd9668c20493e2515ab5"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /chunk
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download chunk "/.chunks/c87a/c87a81a78bb0d4fc8647497c7db511359ea0d3500f945b19555ee544ea2cca43"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /chunk
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download chunk "/.chunks/8c87/8c87c797ce47e58966d39710722fe20063e05535f2a9d32e23dc9d6151153de3"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: GET /chunk
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: download chunk "/.chunks/8e4b/8e4b77efa737b0e32b84b3e4eb6a84f8e7e92719e4f07ad6b4066a79fe7ed42a"
Oct 14 12:29:24 pbs proxmox-backup-proxy[695]: TASK ERROR: connection error: error:0A0003FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1605:SSL alert number 20
Oct 14 12:40:27 pbs proxmox-backup-proxy[695]: write rrd data back to disk
Oct 14 12:40:27 pbs proxmox-backup-proxy[695]: starting rrd data sync
Oct 14 12:40:28 pbs proxmox-backup-proxy[695]: rrd journal successfully committed (25 files in 1.095 seconds)
 
Last edited:
And I can't use ACME at the moment because my DNS isn't working right now since I need to restore it from a backup.
 
Is the snapshot you are trying to restore verified on the PBS side? Please try to force a verification, might be that a chunk is corrupt and cannot be read.

Please provide also the output of pveversion -v from the PVE host and proxmox-backup-manager version --verbose from the PBS host.
 
pveversion -v :
Code: 
proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.4-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.0-1
proxmox-backup-file-restore: 3.2.0-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.1
pve-cluster: 8.0.6
pve-container: 5.0.10
pve-docs: 8.2.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.5
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2

proxmox-backup-manager version --verbose:
Code: 
roxmox-backup                    3.0.1        running kernel: 6.8.4-2-pve
proxmox-backup-server             3.2.2-1      running version: 3.2.2     
proxmox-kernel-helper             8.1.0                                   
proxmox-kernel-6.8                6.8.4-2                                 
proxmox-kernel-6.8.4-2-pve-signed 6.8.4-2                                 
ifupdown2                         3.2.0-1+pmx8                           
libjs-extjs                       7.0.0-4                                 
proxmox-backup-docs               3.2.2-1                                 
proxmox-backup-client             3.2.2-1                                 
proxmox-mail-forward              0.2.3                                   
proxmox-mini-journalreader        1.4.0                                   
proxmox-offline-mirror-helper     0.6.6                                   
proxmox-widget-toolkit            4.2.3                                   
pve-xtermjs                       5.3.0-3                                 
smartmontools                     7.3-pve1                               
zfsutils-linux                    2.2.3-pve2

The verification is fine on all 20 containers and 2 VMs that I have.
 
Last edited:
Slavickox said:
The verification is fine on all 20 containers and 2 VMs that I have.
Click to expand...
When was this last verified? Please force a verification for that particular snapshot. Does this issue affect restores for other guests as well?
 
Chris said:
When was this last verified? Please force a verification for that particular snapshot. Does this issue affect restores for other guests as well?
Click to expand...
Today, there was a verification, and each container/VM gets this message."
 
Hi, I think I found what is causing this error:
Output of proxmox-backup-manager cert info | grep Fingerprint
Code: 
root@pbs:~# proxmox-backup-manager cert info | grep Fingerprint
Fingerprint (sha256): 3c:98:8f:5a:00:f9:20:2e:19:cb:00:ae:17:bb:6c:eb:c4:49:4e:89:e9:f2:a7:42:14:a4:17:5e:d1:f5:1d:fe

Screenshot of trying to add with this key:
1728974637077.png

And the web ui of Backup Server:

1728974700149.png

What i can do to fix this?
 
Slavickox said:
Hi, I think I found what is causing this error:
Output of proxmox-backup-manager cert info | grep Fingerprint
Code: 
root@pbs:~# proxmox-backup-manager cert info | grep Fingerprint
Fingerprint (sha256): 3c:98:8f:5a:00:f9:20:2e:19:cb:00:ae:17:bb:6c:eb:c4:49:4e:89:e9:f2:a7:42:14:a4:17:5e:d1:f5:1d:fe

Screenshot of trying to add with this key:
View attachment 76299

And the web ui of Backup Server:

View attachment 76300

What i can do to fix this?
Click to expand...
Are you sure you are connecting to the correct host? Maybe you have duplicate IPs?

You might want to restart the api and proxy by running systemctl restart proxmox-backup-proxy.service proxmox-backup.service on the PBS side. Then check the TLS connection from the PVE host to the PBS by running openssl s_client -connect 192.168.1.199:8007 on the PVE host.
 
Chris said:
Are you sure you are connecting to the correct host? Maybe you have duplicate IPs?

You might want to restart the api and proxy by running systemctl restart proxmox-backup-proxy.service proxmox-backup.service on the PBS side. Then check the TLS connection from the PVE host to the PBS by running openssl s_client -connect 192.168.1.199:8007 on the PVE host.
Click to expand...
I even changed the IP now, and the fingerprint discrepancy stopped, but I still have the same problem.
The command you provided outputs the certificate that is set on the PBS.
 
Is it possible to specify the command so that the backup restoration can be done without a certificate?
 
Slavickox said:
Is it possible to specify the command so that the backup restoration can be done without a certificate?
Click to expand...
No, that is not possible. The restore is only possible via an encrypted http/2 connection.

Slavickox said:
I even changed the IP now, and the fingerprint discrepancy stopped, but I still have the same problem.
The command you provided outputs the certificate that is set on the PBS.
Click to expand...
Then you still have the fingerprint misconfiguration on the PVE side I guess? I fail to see what else might be wrong. What is the output of cat /etc/pve/storage.cfg on the PVE host?
 
Chris said:
No, that is not possible. The restore is only possible via an encrypted http/2 connection.


Then you still have the fingerprint misconfiguration on the PVE side I guess? I fail to see what else might be wrong. What is the output of cat /etc/pve/storage.cfg on the PVE host?
Click to expand...
After changing the IP, I also changed the certificate.

PBS certificate for now :

Code: 
proxy.pem
58:fe:b4:6a:77:c7:7d:51:f4:3c:f9:fb:80:72:02:8c:98:25:f3:5d:12:3b:c9:3a:38:4b:26:d9:0d:c1:4d:8f
O = Proxmox Backup Server, OU = B6E9439D-7878-438B-9722-10EEE764398C, CN = pbs.1.1.1.1
O = Proxmox Backup Server, OU = B6E9439D-7878-438B-9722-10EEE764398C, CN = pbs.1.1.1.1
rsaEncryption
4096
2024-10-15 09:13:43
3024-02-16 08:13:43
localhost
pbs
pbs.1.1.1.1

Output of cat /etc/pve/storage.cfg
Code: 
root@Proxmox:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

pbs: backu
        datastore Backup
        server 192.168.1.197
        content backup
        fingerprint 58:fe:b4:6a:77:c7:7d:51:f4:3c:f9:fb:80:72:02:8c:98:25:f3:5d:12:3b:c9:3a:38:4b:26:d9:0d:c1:4d:8f
        prune-backups keep-all=1
        username root@pam
 
How many hops away is the PVE from the PBS? Do you get the same error always on the same point? Can you restore on a independent PVE host or do you get the same error?
 
You must log in or register to reply here.
Top Bottom