[SOLVED] Missing all backups after upgrade from Backup Server 2.1 to 3.0-2

[TheStaticTurtle]

Hello,

I just upgraded from PBS 2.1 to 3.0 and all backups in the UI disappeared. ALL the files are still present in the datastore tho. I suspect this is related to the new namespace feature.


I then edited my config in the PVE cluster, set a namespace and manually did a backup which worked fine, and I could see the new backup in the list. I then moved the ct/ and vm/ folders that were in the root of the datastore to the ns/<new namespace/ folder but even after a reboot they are nowhere to be seen in the UI.

Is there any way to force a complete re-scan of the datastore so that the backups repopulate in the UI?

Thanks,

 

[Dunuin]

Did you hit Ctrl+F5 in the PBS webUI after the upgrade, so you are not trying to use the cached PBS2 webUI on a PBS3 in case it is just a cosmetic thing?

Usually the backups shouldn't be missing after upgrading an existing server from PBS 2 to 3.

And you don't have to use namespaces. Backups snapshots in the "root" should be fine and backwards compatible.

Make sure not to run a prune or GC whwn moving stuff around.

 

[TheStaticTurtle]

No I am using the right webUI, the version on the top is correct + I can see/enter the namespace

Also, if I do a "verify all", on the namsepace, it finds all backups (despite the UI only showing the one that I manually backed-up) but reports 0 snapshots:

Example:

2023-09-08T17:15:56+02:00: verify group Gringotts:ct/100 (0 snapshots)
2023-09-08T17:15:56+02:00: verify group Gringotts:ct/101 (0 snapshots)
2023-09-08T17:15:56+02:00: verify group Gringotts:ct/102 (0 snapshots)
..... all the other backups .....
2023-09-08T17:15:56+02:00: verify group Gringotts:ct/122 (1 snapshots)

(ct/122 is the one I backed-up manually)

 

[VictorSTS]

In the upgrade instructions [0] it's stated explicitly that you must be using the latest 2.4 version before upgrading to PBS3. Still, a fast read of the roadmap [1] did not yield an evident / breaking change thay may explain this.

Please post /etc/proxmox-backup/datastores.cfg and a ls -la of the path indicated in the path of each datastore in the datastores.cfg file.

Also, the verify log you posted shows that there's no snapshots for your old backups. Either the directories are empty for some reason or the permissions are wrong. Check that there's content in ns/[namespace]/ct/id/ and that every file and directory is owned by user:group backup:backup.


[0] https://pbs.proxmox.com/wiki/index.php/Upgrade_from_2_to_3#Prerequisites
[1] https://pbs.proxmox.com/wiki/index.php/Roadmap#Roadmap

 

[TheStaticTurtle]

Hi,

In the upgrade instructions [0] it's stated explicitly that you must be using the latest 2.4 version before upgrading to PBS3

Of course, I did. The only thing is that I did not check the backups after the upgrade to the latest 2.X version maybe the actual issue happened during this step of the upgrade.

Here is the output of the command you requested:

root@pbs:~# cat /etc/proxmox-backup/datastore.cfg
datastore: Gringotts
        comment
        gc-schedule daily
        path /nas/backup

root@pbs:~# ls -la /nas/backup
total 11
drwxrwxrwx 2 backup backup    0 Sep  8 18:32 .
drwxr-xr-x 3 root   root   4096 Apr  5  2022 ..
drwxrwxrwx 2 backup backup    0 Apr  5  2022 .chunks
drwxrwxrwx 2 backup backup    0 Sep  8 18:32 ct
-rwxrwxrwx 1 backup backup  325 Sep  8 02:24 .gc-status
-rwxrwxrwx 1 backup backup    0 Apr  5  2022 .lock
drwxrwxrwx 2 backup backup    0 Sep  8 18:30 ns
drwxrwxrwx 2 backup backup    0 Sep  8 18:32 vm

Content of one of the vms snapshots (truncated the output)

root@pbs:~# ls -la /nas/backup/ns/Internal/vm/106/*
-rwxrwxrwx 1 backup backup 9 Jan  5  2023 /nas/backup/ns/Internal/vm/106/owner

/nas/backup/ns/Internal/vm/106/2022-04-24T030844Z:
total 237
drwxrwxrwx 2 backup backup      0 Apr 24  2022 .
drwxrwxrwx 2 backup backup      0 Jul  8 00:00 ..
-rwxrwxrwx 1 backup backup   1023 Apr 24  2022 client.log.blob
-rwxrwxrwx 1 backup backup   4128 Apr 24  2022 drive-efidisk0.img.fidx
-rwxrwxrwx 1 backup backup 528384 Apr 24  2022 drive-sata0.img.fidx
-rwxrwxrwx 1 backup backup   4128 Apr 24  2022 drive-tpmstate0-backup.img.fidx
-rwxrwxrwx 1 backup backup    517 Apr 24  2022 index.json.blob
-rwxrwxrwx 1 backup backup    496 Apr 24  2022 qemu-server.conf.blob

I also tested the proxmox-backup-debug recover index command, and it can recover files

root@pbs:~# proxmox-backup-debug recover index /nas/backup/ns/Internal/vm/106/2023-07-07T030010Z/drive-efidisk0.img.fidx /nas/backup/.chunks
root@pbs:~# ls -lah drive-efidisk0.img
-rw-r--r-- 1 root root 528K Sep  8 23:29 drive-efidisk0.img

I just noticed something weird:


All the old snapshots have the : in their names replaced by the  (Unicode U+F03A)
Renaming the snapshot did work and now is being shown in the UI. Verifying the snapshot also worked.

I could write a small script to rename every folder, but I would like to understand why it happened in the first place

 

[Dunuin]

I just noticed something weird:

All the old snapshots have the : in their names replaced by the  (Unicode U+F03A)
Renaming the snapshot did work and now is being shown in the UI. Verifying the snapshot also worked.

I could write a small script to rename every folder, but I would like to understand why it happened in the first place

I've seen this before. But I can't find the old thread discussing this

Edit: Got it!
You are probably using a Synology too to store your datastore?
Here the same happend after a upgrade from PBS1 to 2 but no solution why this happend: https://forum.proxmox.com/threads/ver-2-cannot-see-ver-1-backups.111877/

 

[Neobin]

I've seen this before. But I can't find the old thread discussing this

Edit: Got it!
You are probably using a Synology too to store your datastore?
Here the same happend after a upgrade from PBS1 to 2 but no solution why this happend: https://forum.proxmox.com/threads/ver-2-cannot-see-ver-1-backups.111877/

Same here with a Synology:
https://forum.proxmox.com/threads/pbs-import-datastore.127917/post-559815

 

[TheStaticTurtle]

Ok,

Wrote the script, if it helps anyone, here it is (maybe you'll need to adjust the rename a bit):

import os

def snapshot(snapshot_old_name):
        snapshot_new_name = snapshot_old_name.encode('utf-8').replace(b"\xef\x80\xa2", b":").decode("utf8")
        print(snapshot_old_name, snapshot_new_name)
        os.rename(snapshot_old_name, snapshot_new_name)

def namespace(path):
        for root, dirs, files in os.walk(path+"/ct", topdown = False):
                if root != (path + "/ct"):
                        for name in dirs:
                                snapshot(os.path.join(root, name))
        for root, dirs, files in os.walk(path+"/vm", topdown = False):
                if root != (path + "/vm"):
                        for name in dirs:
                                snapshot(os.path.join(root, name))

namespace("/nas/backup/ns/Internal")

I started a verify on the whole namespace so far so good. What's weird is that the folder name is correct in the smb mount inside the pbs but stil wrong on the nas (running truenas scale so zfs) / other mounts. Maybe it's an issue with the smb client version

Will mark as solved if everything checks out later

 

[TheStaticTurtle]

Hi,

Everything checked out and verified OK, I'll mark this solved as the issue is probably because I use non-recommended configurations (HDD on a SMB mount)

If I find out more, I'll update the thread.
Thanks everyone