Backup of multiple pxar files in different invocations from same server

[netadair]

Hello all,
in the context of checking out PBS I tried running backups from another machine hosting some docker containers to PBS.
Because of the large size of the setup (lots of archive and mirror folders, plus a chunky dm file for docker) I started two pxar files (excl boot) each with incremental backups using different .pxarexclude stages.
So for the docker part, I used another pxar file ie

proxmox-backup-client backup root.pxar:/ boot.pxar:/boot --change-detection-mode=metadata --ns ${PBS_NAMESPACE}
proxmox-backup-client backup docker.pxar:/var/lib/docker --change-detection-mode=metadata --ns ${PBS_NAMESPACE}

Obviously, this seem to not work... for the next invocation, root.pxar complains about missing 'root.mpxar.didx' and restarts the whole thing from scratch, wasting bw and time (ultimately failing after three days).

Is this true: just one manifest per client per folder? Or am I missing something?

Ciao
Michael

 

[netadair]

PS: Then solution would be two different name spaces?!

 

[Chris]

Hi, you can also just specify all the archives in a single command invocation, just like you do for root.pxar and boot.pxar. Another option would be to set the --backup-id so the backup are stored in different groups, default being the hostname.

 

[netadair]

Thanks Chris, I'll try that. However, still the question, when not using backup-id or name space, this looks like they MUST be in a single command invocation, otherwise the index.json will not contain the files from the other invocation. Is that correct?
(It looks a bit like https://forum.proxmox.com/threads/pbs-client-not-detecting-previous-manifest.170090/)

 

[Chris]

Yes, the change detection mode metadata will use the metadata archives (mpxar) of the previous backup snapshot to lookup and compare files and detect unchanged ones. If you perform backups to the same group in the same namespace, this will therefore not work as you expected.

 

[netadair]

Ok, understood.
I'm not sure if it would be worth it, but could the index blob not be carried forward and pxar possibly linked from last snapshot if a pxar file is not changed/considered in one invocation. Example (not good one): backup root.pxar and boot.pxar, but subsequently boot.pxar only if something is changed explicitely (ie left out during runs where only incremental backup of root.pxar is made and done manually after kernel upgrade or the like changing /boot) ?
Benefit: have one "latest snapshot" containing everything, even if the backups where made with completely different frequencies per pxar file.

 

[Chris]

Ok, understood.
I'm not sure if it would be worth it, but could the index blob not be carried forward and pxar possibly linked from last snapshot if a pxar file is not changed/considered in one invocation. Example (not good one): backup root.pxar and boot.pxar, but subsequently boot.pxar only if something is changed explicitely (ie left out during runs where only incremental backup of root.pxar is made and done manually after kernel upgrade or the like changing /boot) ?
Benefit: have one "latest snapshot" containing everything, even if the backups where made with completely different frequencies per pxar file.

No, that is not possible, each backup snapshot must remain independent. Further, the index json (manifest) does not contain the relevant information, that is stored in the mpxar archive. Just use either different namespaces or different backup ids.

 

[netadair]

Backup id works fine, and some manual rearrangement of the name space folder dissecting the two backups also prevented a full backup (taking three days).
Once again, thanks a lot.