Metadata Detection Mode Not Working

[tdashmike]

I made a backup a few days ago using the command:

nohup proxmox-backup-client backup nas.pxar:/storage/nas --repository 192.168.1.7:backup --ns PROXMOX --change-detection-mode=metadata &

I'm running the same command again right now but it doesn't seem to be using the previous catalog.

Starting backup: [PROXMOX]:host/PROXMOX/2025-01-17T17:18:57Z
Client name: PROXMOX
Starting backup protocol: Fri Jan 17 12:18:57 2025
Downloading previous manifest (Fri Jan 17 12:18:23 2025)
Upload directory '/storage/nas' to '192.168.1.7:backup' as nas.mpxar.didx
Previous manifest does not contain an archive called 'nas.mpxar.didx', skipping download..
Previous manifest does not contain an archive called 'nas.ppxar.didx', skipping download..
processed 1.974 GiB in 1m, uploaded 1.932 GiB
processed 3.904 GiB in 2m, uploaded 3.864 GiB
processed 6.368 GiB in 3m, uploaded 6.321 GiB
processed 8.792 GiB in 4m, uploaded 8.77 GiB

When I look in Datastore -> backup -> Content, I can see the mxpar and ppxar files are there from the previous run.

 

[dietmar]

You used the same namespace?

 

[tdashmike]

Yes, I did. In the datastore, they are both listed under the same namespace. I noticed the verify state is "none" on all my backups, not sure if that affects the incremental backup.

proxmox-backup-client snapshot list --ns PROXMOX
host/PROXMOX/2025-01-12T19:02:25Z │ 645.379 GiB │ index.json nas.mpxar nas.ppxar
host/PROXMOX/2025-01-17T17:18:57Z │ 673.221 GiB │ index.json nas.mpxar nas.ppxar

 

[Chris]

Hi,
please download the file index.json.blob for both snapshots, the old and the new one and add them as attachment here. Also, what is your proxmox-backup-client version?

Also, can you access the previous snapshots without problems, e.g. a proxmox-backup-client catalog dump host/PROXMOX/2025-01-12T19:02:25Z --ns PROXMOX should dump you an index of all the files contained within that snapshot. Verify that each line starts with the ./nas.mpxar.didx prefix.

I noticed the verify state is "none" on all my backups, not sure if that affects the incremental backup.

If the previous snapshot failed verification, then it cannot be used as reference, but if there has been no verification, this should be fine.

 

[tdashmike]

Hi,
please download the file index.json.blob for both snapshots, the old and the new one and add them as attachment here. Also, what is your proxmox-backup-client version?

Also, can you access the previous snapshots without problems, e.g. a proxmox-backup-client catalog dump host/PROXMOX/2025-01-12T19:02:25Z --ns PROXMOX should dump you an index of all the files contained within that snapshot. Verify that each line starts with the ./nas.mpxar.didx prefix.


If the previous snapshot failed verification, then it cannot be used as reference, but if there has been no verification, this should be fine.

Chris,

proxmox-backup-client version
client version: 3.3.2
server version: 3.3.2

I switched to another backup for discussion purposes since it was much smaller and experiencing the same issue. I have attached both index files. I also checked both catalogs and all the files in both started with ./etc.mpxar.didx.

Example:

l "./etc.mpxar.didx/zfs/zpool.d/upath"
l "./etc.mpxar.didx/zfs/zpool.d/vendor"
l "./etc.mpxar.didx/zfs/zpool.d/w_proc"
l "./etc.mpxar.didx/zfs/zpool.d/w_ucor"

 

[Chris]

Thanks for providing the manifests (index files). These look okay to me.

Looking closer at your initial output once again, I did notice however that the client seems to download a different manifest, not shown at all in your output?

Downloading previous manifest (Fri Jan 17 12:18:23 2025)

Please do share the backup client logs for you smaller reproducer as well, especially the logged timestamps are of interest. Do you have concurrent backups going on? What is the output of ls -lahR.<datastore-path>/ns/PROXMOX/host/PROXMOX run on your PBS, given the correct datastore path?

 

[tdashmike]

I have 2 backups I'm running recently, one is the /etc directory and the other the /nas directory. I just ran another backup of the /etc directory and it was able to use the previous manifest. I did notice that the time in the snapshot list is off by 5 hours due to the time zone, would be nice if it matched but not a big deal.

Starting backup: [PROXMOX]:host/PROXMOX/2025-01-19T16:24:02Z
Client name: PROXMOX
Starting backup protocol: Sun Jan 19 11:24:02 2025
Downloading previous manifest (Sat Jan 18 13:06:25 2025)
Upload directory '/etc' to '192.168.1.7:backup' as etc.mpxar.didx
Using previous index as metadata reference for 'etc.mpxar.didx'
Change detection summary:
 - 699 total files (0 hardlinks)
 - 0 unchanged, reusable files with 0 B data
 - 699 changed or non-reusable files with 2.167 MiB data
 - 0 B padding in 0 partially reused chunks
etc.mpxar: had to backup 256.127 KiB of 256.127 KiB (compressed 53.67 KiB) in 0.24 s (average 1.023 MiB/s)
etc.ppxar: had to backup 1.017 MiB of 2.167 MiB (compressed 230.802 KiB) in 0.25 s (average 4.102 MiB/s)
etc.ppxar: backup was done incrementally, reused 1.15 MiB (53.1%)
Duration: 0.72s
End Time: Sun Jan 19 11:24:03 2025

host/PROXMOX/2025-01-17T17:18:23Z │   2.418 MiB │ etc.mpxar etc.ppxar index.json
host/PROXMOX/2025-01-17T17:18:57Z │ 673.221 GiB │ index.json nas.mpxar nas.ppxar
host/PROXMOX/2025-01-18T18:06:25Z │   2.418 MiB │ etc.mpxar etc.ppxar index.json
host/PROXMOX/2025-01-19T16:24:02Z │   2.418 MiB │ etc.mpxar etc.ppxar index.json

I then tried to backup /nas with --dry-run and it looks like it is trying to use the manifest from the /etc backup I just ran a few minutes ago.

Starting backup: [PROXMOX]:host/PROXMOX/2025-01-19T16:33:08Z
Client name: PROXMOX
Starting backup protocol: Sun Jan 19 11:33:08 2025
Downloading previous manifest (Sun Jan 19 11:24:02 2025)
Would upload directory '/storage/nas' to '192.168.1.7:backup' as nas.pxar.didx
dry-run: no upload happened

ls -lahR /mnt/datastore/backup/encrypted/ns/PROXMOX/host/PROXMOX

/mnt/datastore/backup/encrypted/ns/PROXMOX/host/PROXMOX:
total 56K
drwxr-xr-x 6 backup backup 7 Jan 19 11:33 .
drwxr-xr-x 3 backup backup 3 Jan 11 22:36 ..
drwxr-xr-x 2 backup backup 5 Jan 18 23:37 2025-01-17T17:18:23Z
drwxr-xr-x 2 backup backup 5 Jan 18 23:37 2025-01-17T17:18:57Z
drwxr-xr-x 2 backup backup 5 Jan 18 13:10 2025-01-18T18:06:25Z
drwxr-xr-x 2 backup backup 5 Jan 19 11:24 2025-01-19T16:24:02Z
-rw-r--r-- 1 backup backup 9 Jan 11 22:36 owner

'/mnt/datastore/backup/encrypted/ns/PROXMOX/host/PROXMOX/2025-01-17T17:18:23Z':
total 31K
drwxr-xr-x 2 backup backup    5 Jan 18 23:37 .
drwxr-xr-x 6 backup backup    7 Jan 19 11:33 ..
-rw-r--r-- 1 backup backup 4.1K Jan 17 12:18 etc.mpxar.didx
-rw-r--r-- 1 backup backup 4.2K Jan 17 12:18 etc.ppxar.didx
-rw-r--r-- 1 backup backup  483 Jan 18 23:37 index.json.blob

'/mnt/datastore/backup/encrypted/ns/PROXMOX/host/PROXMOX/2025-01-17T17:18:57Z':
total 7.8M
drwxr-xr-x 2 backup backup    5 Jan 18 23:37 .
drwxr-xr-x 6 backup backup    7 Jan 19 11:33 ..
-rw-r--r-- 1 backup backup  502 Jan 18 23:37 index.json.blob
-rw-r--r-- 1 backup backup 4.1K Jan 17 19:28 nas.mpxar.didx
-rw-r--r-- 1 backup backup 7.8M Jan 17 19:28 nas.ppxar.didx

'/mnt/datastore/backup/encrypted/ns-PROXMOX/host/PROXMOX/2025-01-18T18:06:25Z':
total 31K
drwxr-xr-x 2 backup backup    5 Jan 18 13:10 .
drwxr-xr-x 6 backup backup    7 Jan 19 11:33 ..
-rw-r--r-- 1 backup backup 4.1K Jan 18 13:06 etc.mpxar.didx
-rw-r--r-- 1 backup backup 4.2K Jan 18 13:06 etc.ppxar.didx
-rw-r--r-- 1 backup backup  481 Jan 18 13:10 index.json.blob

'/mnt/datastore/backup/encrypted/ns/PROXMOX/host/PROXMOX/2025-01-19T16:24:02Z':
total 31K
drwxr-xr-x 2 backup backup    5 Jan 19 11:24 .
drwxr-xr-x 6 backup backup    7 Jan 19 11:33 ..
-rw-r--r-- 1 backup backup 4.1K Jan 19 11:24 etc.mpxar.didx
-rw-r--r-- 1 backup backup 4.2K Jan 19 11:24 etc.ppxar.didx
-rw-r--r-- 1 backup backup  381 Jan 19 11:24 index.json.blob

 

[Chris]

Ah, well than explains your troubles. The backup client will always use the previous snapshot of the backup group as reference. It cannot infer which snapshot belongs to which backup run (meaning backups of different archives).

You can however create a snapshot containing both archives by running: proxmox-backup-client backup etc.pxar:/etc nas.pxar:/storage/nas --repository 192.168.1.7:backup --ns PROXMOX --change-detection-mode=metadata.

 

[tdashmike]

Sounds good. I would probably suggest a feature request to be able to specify the snapshot to reference or simply look for the last pxar file with the same name. For now, I deleted all the other snapshots up to the one I need the manifest from and run the incremental backup that way. Does the normal VM or container backups use the metadata detection mode? I'm wondering if I can attach the /nas folder to either a VM or container and have it automatically backup that way using the metadata mode.

Thanks for the quick support, hope you're having a great weekend.