File level recovery does not work

F

fladnar

New Member
Feb 28, 2024
6
0
1
Hi all :)
I'm new to proxmox. I've searched online for the solution to this, but nothing works.
I'm testing a latest version of proxmox with the pbs role added.
Backups are created ok, and i can recover the whole vm with no problems, but when i try to start an individual file recovery, i get this error:

proxmox-file-restore failed: Error: Starting VM failed. See output above for more information. (500)

Any clues?. I'm testing on a 16 core xeon with 48g of ram and there is only a vm running with 4 cores / 16g. The backups are on a zfs mirror pool.

Thanks in advance :-)
 
what does the log file under `/var/log/proxmox-backup/file-restore/qemu.log` say ?
 
It's empty :-?

root@proxmox:~# cat /var/log/proxmox-backup/file-restore/qemu.log
[2024-02-28T17:52:21+01:00] PBS file restore VM log
root@proxmox:~#
 
is that the node where you connect to the webui ? (not necessarily the one where you try to live-restore)
 
can you post your 'pveversion -v' and 'proxmox-backup-manager versions --verbose' ?
 
Of course :-)
dcsapak said:
can you post your 'pveversion -v' and 'proxmox-backup-manager versions --verbose' ?
Click to expand...
Of course :)

Linux proxmox 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Feb 29 13:26:28 CET 2024 on pts/0
root@proxmox:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve2

---------------


proxmox-backup unknown running kernel: 6.5.13-1-pve
proxmox-backup-server 3.1.4-1 running version: 3.1.4
proxmox-kernel-helper 8.1.0
proxmox-kernel-6.5.13-1-pve-signed 6.5.13-1
proxmox-kernel-6.5 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed 6.5.11-8
ifupdown2 3.2.0-1+pmx8
libjs-extjs 7.0.0-4
proxmox-backup-docs 3.1.4-1
proxmox-backup-client 3.1.4-1
proxmox-mail-forward 0.2.3
proxmox-mini-journalreader 1.4.0
proxmox-offline-mirror-helper 0.6.5
proxmox-widget-toolkit 4.1.3
pve-xtermjs 5.3.0-3
smartmontools 7.3-pve1
zfsutils-linux 2.2.2-pve2
root@proxmox:~#
 
mhmm seems fine to me...

could you try to start a new file-restore and check the log file again? there should be at least some log lines there

also could you then check the syslog/journal from that time? maybe we can see more there
 
dcsapak said:
mhmm seems fine to me...

could you try to start a new file-restore and check the log file again? there should be at least some log lines there

also could you then check the syslog/journal from that time? maybe we can see more there
Click to expand...

root@proxmox:~# cat /var/log/proxmox-backup/file-restore/qemu.log
[2024-03-05T08:57:12+01:00] PBS file restore VM log

syslog:
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-virtio0.img.fidx"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-virtio0.img.fidx' as downloadable.
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/6440/6440a9471f8a40a85a19d7fdf80078b0684972ab0045e049d9eaf2a2c8001973"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-efidisk0.img.fidx"
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-efidisk0.img.fidx' as downloadable.
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:11 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/7c12/7c1242e52d56df964dd94fe13d50831dbbf3787ad434f1633c23434b3c315045"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-virtio0.img.fidx"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-virtio0.img.fidx' as downloadable.
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/6440/6440a9471f8a40a85a19d7fdf80078b0684972ab0045e049d9eaf2a2c8001973"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: protocol upgrade done
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/index.json.blob"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /download
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download "/mnt/datastore/backup/vm/100/2024-03-05T07:54:37Z/drive-efidisk0.img.fidx"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: register chunks in 'drive-efidisk0.img.fidx' as downloadable.
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: GET /chunk
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: download chunk "/mnt/datastore/backup/.chunks/7c12/7c1242e52d56df964dd94fe13d50831dbbf3787ad434f1633c23434b3c315045"
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Mar 05 09:00:35 proxmox chronyd[1266]: Selected source 162.159.200.1 (2.debian.pool.ntp.org)
 
besides these errors:

fladnar said:
Mar 05 08:57:12 proxmox proxmox-backup-proxy[1223]: TASK ERROR: connection error: not connected
Click to expand...

the only thing out of the ordinary is that the qemu.log file is empty...

can you check if the network connection is working correctly to localhost, e.g. that the hostname/pbs storage/etc. is properly configured?
 
dcsapak said:
besides these errors:



the only thing out of the ordinary is that the qemu.log file is empty...

can you check if the network connection is working correctly to localhost, e.g. that the hostname/pbs storage/etc. is properly configured?
Click to expand...
It seems all rigthly configured.
Bear in mind the whole vm recovery works flawlessly, it's the file level the one that fails.
 
mhmm... what you could do is to start the file-restore manually with
Code: 
proxmox-file-restore list <snapshot> <path>
e.g. start with '/' as path, and navigate through the backup with the output you get, e.g. if it says the drive is 'drive-scsi0.img.fidx' then you have follow up with '/drive-scsi0.img.fidx' as a path and so on

note that you have to export PBS_FINGERPRINT, PBS_PASSWORD and (optionally) PBS_REPOSITORY for it to work properly

if you run into any difficulty just ask
 
Having the same issue here. Ran the following command:

Code: 
proxmox-file-restore list vm/101/2024-05-02T07:42:01Z drive-efidisk0.img.fidx --repository root@pam@localhost:pbs_backup

Response:

Code: 
PBS connect failed: command error: error trying to connect: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889:
   
Error: Starting VM failed. See output above for more information.

Restoring from the UI works fine, it's just the file level restore on VMs that show the above error, both on the UI and command line. Just to note I'm running PBS on the PVE host.
 
Last edited:
the response indicates that the tls certificate could not be verified. if you have not installed a 'valid' certificate (e.g. via upload or letsencrypt) you need to pass the fingerprint to the command via the environment variable 'PBS_FINGERPRINT'
like for the proxmox-backup-client, see https://pbs.proxmox.com/docs/backup-client.html#environment-variables
 
After exporting the PBS fingerprint as an environment variable this is the error I get:

Code: 
proxmox-file-restore list vm/103/2024-05-13T19:22:08Z drive-efidisk0.img.fidx --repository root@pam@localhost:pbs_backup
qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=10,disable-legacy=on: vhost-vsock: failed to open vhost device: No such file or directory

Error: Starting VM failed. See output above for more information.

I've seen other posts with a similar issue, but most say a reboot of the host fixes it - no luck for me so far.
 
dcsapak said:
the response indicates that the tls certificate could not be verified. if you have not installed a 'valid' certificate (e.g. via upload or letsencrypt) you need to pass the fingerprint to the command via the environment variable 'PBS_FINGERPRINT'
like for the proxmox-backup-client, see https://pbs.proxmox.com/docs/backup-client.html#environment-variables
Click to expand...

Any ideas by chance after getting the error I posted above? The file definitely exists. Just to reiterate I'm using PBS install on the Proxmox host, so it's the same IP.

This user on reddit had the same issue as I did, but a host reboot fixed it for them. I'm thinking of wiping the PBS install and trying again.
 
sorry, seems i missed your previous response.

can you check if there is a log in /var/log/proxmox-backup/file-restore/qemu.log ?
(that should contain the log for the restore vm that gets started)
it's possible there isn't any though

could you check if there is a file '/dev/vhost-vsock' ?
if not, please try to execute
Code: 
modprobe vhost_vsock

and try the file restore again
 
dcsapak said:
sorry, seems i missed your previous response.

can you check if there is a log in /var/log/proxmox-backup/file-restore/qemu.log ?
(that should contain the log for the restore vm that gets started)
it's possible there isn't any though

could you check if there is a file '/dev/vhost-vsock' ?
if not, please try to execute
Code: 
modprobe vhost_vsock

and try the file restore again
Click to expand...
The log only showed 'PBS file restore VM log'.

'/dev/vhost-vsock' did not exist, but after running the command modprobe vhost_vsock file restore works now! Thanks so much.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom