[SOLVED] Restoring Backups from AWS, datastore not showing content

[sysadminbb]

Hello everyone, first post in the forum so bare with me!

I have PBS 3.0-3 that takes backups of 3 PVE nodes we have in a cluster. PBS is a software raidz2 configuration (40TB) and and does backups daily.

I have 2 datastores, one called "Backups" and one called "Recovery", I use "Recovery" for when we pull backups back down from AWS Glacier.

I have two scripts that interact with AWS-CLI, one that is for uploading to glacier, and one for emergencies when we need to pull data back down. The script compares the hash of the backups and validates before extracting to my "Recovery" datastore.

My problem is that when the data is extracted to the Recovery datastore, it does not show up under 'Content", the ct and vm folders are there with the data, but nothing displayed in my recovery datastore.

Should I not be pulling back down to a different datastore? Should it just pull back to the datastore that it was originally in before being sent to glacier? Let me know what I am missing or if any more details are needed.

Thanks,

 

[dcsapak]

Should I not be pulling back down to a different datastore? Should it just pull back to the datastore that it was originally in before being sent to glacier?

no that shouldn't be an issue

does it create the files/folders with the correct permission? can you post the output of 'ls -lhR /path/to/recovery' ?
(just to make sure, you do also backup/restore the .chunks folder ? that's where the actual data is)

also do you use a non root@pam user? does the user have the correct permissions to see the backups?

 

[sysadminbb]

no that shouldn't be an issue

does it create the files/folders with the correct permission? can you post the output of 'ls -lhR /path/to/recovery' ?
(just to make sure, you do also backup/restore the .chunks folder ? that's where the actual data is)

also do you use a non root@pam user? does the user have the correct permissions to see the backups?

Thanks for the reply, I do use root@pam user, and here is my output, I do see the .chunks files as well, this restoral is essentially pulling down the whole backup directory so nothing should be missed.

root@pbs:~# ls -lhR .backuppulls/
.backuppulls/:
total 1.0K
drwxr-xr-x 3 root root 3 Oct  5 06:54 ct
drwxr-xr-x 3 root root 3 Oct  5 06:55 vm

.backuppulls/ct:
total 512
drwxr-xr-x 3 root root 4 Oct 10 09:36 110

.backuppulls/ct/110:
total 9.0K
drwxr-xr-x 2 root root 8 Oct  5 06:55 2023-10-05T10:54:33Z
-rw-r--r-- 1 root root 9 Oct  5 06:54 owner

'.backuppulls/ct/110/2023-10-05T10:54:33Z':
total 63K
-rw-r--r-- 1 root root 4.1K Oct  5 06:55 catalog.pcat1.didx
-rw-r--r-- 1 root root  800 Oct  5 06:55 client.log.blob
-rw-r--r-- 1 root root   42 Oct  5 06:54 fw.conf.blob
-rw-r--r-- 1 root root  513 Oct  5 06:55 index.json.blob
-rw-r--r-- 1 root root  335 Oct  5 06:54 pct.conf.blob
-rw-r--r-- 1 root root  41K Oct  5 06:55 root.pxar.didx

.backuppulls/vm:
total 512
drwxr-xr-x 3 root root 4 Oct  5 06:55 121

.backuppulls/vm/121:
total 9.0K
drwxr-xr-x 2 root root 6 Oct  5 07:03 2023-10-05T10:55:19Z
-rw-r--r-- 1 root root 9 Oct  5 06:55 owner

'.backuppulls/vm/121/2023-10-05T10:55:19Z':
total 426K
-rw-r--r-- 1 root root 2.1K Oct  5 07:03 client.log.blob
-rw-r--r-- 1 root root 404K Oct  5 07:03 drive-scsi0.img.fidx
-rw-r--r-- 1 root root  416 Oct  5 07:03 index.json.blob
-rw-r--r-- 1 root root  342 Oct  5 06:55 qemu-server.conf.blob
root@pbs:~/.backuppulls# ls -a
.  ..  .chunks  ct  .gc-status  .lock  vm
root@pbs:~/.backuppulls#

Thanks again.

 

[dcsapak]

ok, so you restore the files with the owner user:group 'root:root' but it needs to be 'backup:backup' since the daemon that read from that runs as the backup user/group
(not sure if that is the source of your problem, but it could very well be)

 

[sysadminbb]

ok, so you restore the files with the owner user:group 'root:root' but it needs to be 'backup:backup' since the daemon that read from that runs as the backup user/group
(not sure if that is the source of your problem, but it could very well be)

I went ahead and and ran the following:

chown -R backup:backup .backuppulls/

ran ls -lhR and it is now owned by backups user, what would next steps be? I reloaded the datastore from content section and it's not loading ct/vm data.

Thanks,

 

[dcsapak]

can you post your datastore config (/etc/proxmox-backup/datastore.cfg)
and the output of:

proxmox-backup-client snapshot list --repository localhost:<datastore-name>

?

 

[sysadminbb]

can you post your datastore config (/etc/proxmox-backup/datastore.cfg)
and the output of:

proxmox-backup-client snapshot list --repository localhost:<datastore-name>

?

datastore: PCBs comment gc-schedule daily path /backups prune-schedule daily verify-new true datastore: Backups comment gc-schedule daily path /mnt/datastore/backups prune-schedule daily datastore: Recovery comment gc-schedule daily path /.backuppulls prune-schedule daily verify-new true

The first time I ran

proxmox-backup-client snapshot list --repository localhost:Recovery
Password for "root@pam":**
fingerprint: **
Are you sure you want to continue connecting? (y/n): y
Error: channel closed

The second time it just listed nothing, my other datastores will list the data

root@pbs:/etc/proxmox-backup# proxmox-backup-client snapshot list --repository localhost:Recovery
root@pbs:/etc/proxmox-backup#

Thanks,

 

[sysadminbb]

Would this datastore being located in a hidden folder be causing any issues?

 

[dcsapak]

root@pbs:~/.backuppulls#

path /.backuppulls

your path is wrong

'~/.backuppulls' is in the homedir of the root user, which is /root so the whole path is '/root/.backuppulls'
but you configured '/.backuppulls'

having it in the root home folder will not work because that is not readable by the backup user at all

so you have to create the '/.backuppulls' folder and adapt your restore script to restore it there

 

[sysadminbb]

your path is wrong

'~/.backuppulls' is in the homedir of the root user, which is /root so the whole path is '/root/.backuppulls'
but you configured '/.backuppulls'

having it in the root home folder will not work because that is not readable by the backup user at all

so you have to create the '/.backuppulls' folder and adapt your restore script to restore it there

This worked. I changed the recovery datastore to /mnt/datastore/recovery and updated restore script to that folder.

recovered backups are showing correctly in content section in GUI now.

Thanks!