File level recovery does not work

F

fladnar

New Member
Feb 28, 2024
6
0
1
Hi all :)
I'm new to proxmox. I've searched online for the solution to this, but nothing works.
I'm testing a latest version of proxmox with the pbs role added.
Backups are created ok, and i can recover the whole vm with no problems, but when i try to start an individual file recovery, i get this error:

proxmox-file-restore failed: Error: Starting VM failed. See output above for more information. (500)

Any clues?. I'm testing on a 16 core xeon with 48g of ram and there is only a vm running with 4 cores / 16g. The backups are on a zfs mirror pool.

Thanks in advance :-)
 
what does the log file under `/var/log/proxmox-backup/file-restore/qemu.log` say ?
 
It's empty :-?

root@proxmox:~# cat /var/log/proxmox-backup/file-restore/qemu.log
[2024-02-28T17:52:21+01:00] PBS file restore VM log
root@proxmox:~#
 
is that the node where you connect to the webui ? (not necessarily the one where you try to live-restore)
 
can you post your 'pveversion -v' and 'proxmox-backup-manager versions --verbose' ?
 
Of course :-)
dcsapak said:
can you post your 'pveversion -v' and 'proxmox-backup-manager versions --verbose' ?
Click to expand...
Of course :)

Linux proxmox 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Feb 29 13:26:28 CET 2024 on pts/0
root@proxmox:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve2

---------------


proxmox-backup unknown running kernel: 6.5.13-1-pve
proxmox-backup-server 3.1.4-1 running version: 3.1.4
proxmox-kernel-helper 8.1.0
proxmox-kernel-6.5.13-1-pve-signed 6.5.13-1
proxmox-kernel-6.5 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed 6.5.11-8
ifupdown2 3.2.0-1+pmx8
libjs-extjs 7.0.0-4
proxmox-backup-docs 3.1.4-1
proxmox-backup-client 3.1.4-1
proxmox-mail-forward 0.2.3
proxmox-mini-journalreader 1.4.0
proxmox-offline-mirror-helper 0.6.5
proxmox-widget-toolkit 4.1.3
pve-xtermjs 5.3.0-3
smartmontools 7.3-pve1
zfsutils-linux 2.2.2-pve2
root@proxmox:~#
 
mhmm seems fine to me...

could you try to start a new file-restore and check the log file again? there should be at least some log lines there

also could you then check the syslog/journal from that time? maybe we can see more there
 
dcsapak said:
mhmm seems fine to me...

could you try to start a new file-restore and check the log file again? there should be at least some log lines there

also could you then check the syslog/journal from that time? maybe we can see more there
Click to expand...

root@proxmox:~# cat /var/log/proxmox-backup/file-restore/qemu.log
[2024-03-05T08:57:12+01:00] PBS file restore VM log

syslog:
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-virtio0.img.fidx"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-virtio0.img.fidx' as downloadable.
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/6440/6440a9471f8a40a85a19d7fdf80078b0684972ab0045e049d9eaf2a2c8001973"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-efidisk0.img.fidx"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-efidisk0.img.fidx' as downloadable.
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/7c12/7c1242e52d56df964dd94fe13d50831dbbf3787ad434f1633c23434b3c315045"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-virtio0.img.fidx"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-virtio0.img.fidx' as downloadable.
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/6440/6440a9471f8a40a85a19d7fdf80078b0684972ab0045e049d9eaf2a2c8001973"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-efidisk0.img.fidx"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-efidisk0.img.fidx' as downloadable.
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/7c12/7c1242e52d56df964dd94fe13d50831dbbf3787ad434f1633c23434b3c315045"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 09:00:35 proxmox chronyd[1266]: Selected source 162.159.200.1 (2.debian.pool.ntp.org)
 
besides these errors:

fladnar said:
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Click to expand...

the only thing out of the ordinary is that the qemu.log file is empty...

can you check if the network connection is working correctly to localhost, e.g. that the hostname/pbs storage/etc. is properly configured?
 
dcsapak said:
besides these errors:



the only thing out of the ordinary is that the qemu.log file is empty...

can you check if the network connection is working correctly to localhost, e.g. that the hostname/pbs storage/etc. is properly configured?
Click to expand...
It seems all rigthly configured.
Bear in mind the whole vm recovery works flawlessly, it's the file level the one that fails.
 
mhmm... what you could do is to start the file-restore manually with
Code: 
proxmox-file-restore list <snapshot> <path>
e.g. start with '/' as path, and navigate through the backup with the output you get, e.g. if it says the drive is 'drive-scsi0.img.fidx' then you have follow up with '/drive-scsi0.img.fidx' as a path and so on

note that you have to export PBS_FINGERPRINT, PBS_PASSWORD and (optionally) PBS_REPOSITORY for it to work properly

if you run into any difficulty just ask
 
Having the same issue here. Ran the following command:

Code: 
proxmox-file-restore list vm/101/2024-05-02T07:42:01Z drive-efidisk0.img.fidx --repository root@pam@localhost:pbs_backup

Response:

Code: 
PBS connect failed: command error: error trying to connect: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889:
   
Error: Starting VM failed. See output above for more information.

Restoring from the UI works fine, it's just the file level restore on VMs that show the above error, both on the UI and command line. Just to note I'm running PBS on the PVE host.
 
Last edited:
the response indicates that the tls certificate could not be verified. if you have not installed a 'valid' certificate (e.g. via upload or letsencrypt) you need to pass the fingerprint to the command via the environment variable 'PBS_FINGERPRINT'
like for the proxmox-backup-client, see https://pbs.proxmox.com/docs/backup-client.html#environment-variables
 
After exporting the PBS fingerprint as an environment variable this is the error I get:

Code: 
proxmox-file-restore list vm/103/2024-05-13T19:22:08Z drive-efidisk0.img.fidx --repository root@pam@localhost:pbs_backup
qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=10,disable-legacy=on: vhost-vsock: failed to open vhost device: No such file or directory

Error: Starting VM failed. See output above for more information.

I've seen other posts with a similar issue, but most say a reboot of the host fixes it - no luck for me so far.
 
dcsapak said:
the response indicates that the tls certificate could not be verified. if you have not installed a 'valid' certificate (e.g. via upload or letsencrypt) you need to pass the fingerprint to the command via the environment variable 'PBS_FINGERPRINT'
like for the proxmox-backup-client, see https://pbs.proxmox.com/docs/backup-client.html#environment-variables
Click to expand...

Any ideas by chance after getting the error I posted above? The file definitely exists. Just to reiterate I'm using PBS install on the Proxmox host, so it's the same IP.

This user on reddit had the same issue as I did, but a host reboot fixed it for them. I'm thinking of wiping the PBS install and trying again.
 
sorry, seems i missed your previous response.

can you check if there is a log in /var/log/proxmox-backup/file-restore/qemu.log ?
(that should contain the log for the restore vm that gets started)
it's possible there isn't any though

could you check if there is a file '/dev/vhost-vsock' ?
if not, please try to execute
Code: 
modprobe vhost_vsock

and try the file restore again
 
dcsapak said:
sorry, seems i missed your previous response.

can you check if there is a log in /var/log/proxmox-backup/file-restore/qemu.log ?
(that should contain the log for the restore vm that gets started)
it's possible there isn't any though

could you check if there is a file '/dev/vhost-vsock' ?
if not, please try to execute
Code: 
modprobe vhost_vsock

and try the file restore again
Click to expand...
The log only showed 'PBS file restore VM log'.

'/dev/vhost-vsock' did not exist, but after running the command modprobe vhost_vsock file restore works now! Thanks so much.
 
Last edited:
glances said:
The log only showed 'PBS file restore VM log'.

'/dev/vhost-vsock' did not exist, but after running the command modprobe vhost_vsock file restore works now! Thanks so much.
Click to expand...
Hi @glances. We are having the same symptoms as you with the latest version of PBS 3.3.1 in combination with PVE 8.3.1. File recovery is not a functionality we are use often so I don't know in which version stopped working, but full image restore works definitely.

Non of the solutions worked for us, nor the reboot of the host nor running `modprobe vhost_vsock`. The log file '/var/log/proxmox-backup/file-restore/qemu.log' only shows one line:
'[2024-12-16T11:31:46+00:00] PBS file restore VM log'.

Do you have any other ideas @dcsapak ?

Thanks in advance
 
You must log in or register to reply here.
Top Bottom