backup client transfers data repeatedly

dejhost

Active Member
Dec 13, 2020
67
1
28
45
Hello.

I call the proxmox backup client from within a bash script - on a daily basis using cron. The computer I am backing up is a desktop PC with linux Mint. For an unknown reason, much of the data get transferred everytime the script runs - even if the data has not been altered. Please take a look at the example log:

Code:
[ 2025-04-09 13:12:08 ] Backing up Katja's home directory...
Starting backup: host/Mint-workstation/2025-04-09T11:12:08Z
Client name: Mint-workstation
Starting backup protocol: Wed Apr  9 13:12:08 2025
Downloading previous manifest (Wed Apr  9 11:30:01 2025)
Upload directory '/home/katja' to 'backupuser@pbs@192.168.1.174:8007:backup02' as katja.pxar.didx
Error downloading .didx from previous manifest: Unable to open dynamic index "/mnt/backup/host/Mint-workstation/2025-04-09T09:30:01Z/katja.pxar.didx" - No such file or directory (os error 2)
katja.pxar: had to backup 1.917 GiB of 1.943 GiB (compressed 1.246 GiB) in 269.12s
katja.pxar: average backup speed: 7.296 MiB/s
katja.pxar: backup was done incrementally, reused 26.114 MiB (1.3%)
Uploaded backup catalog (534.827 KiB)
Duration: 269.70s
End Time: Wed Apr  9 13:16:38 2025

Data within the mountpoint /home/katja/ has been preexisting and has not been altered. So why are only 1.3% MB reused? This happens similar for other mountspoints as well - in one of them, more than 100GB are transferred over and over again. The script already ran for several weeks on a daily basis.

PBS has been on Version 3.2.X until today. I just updated to version 3.4 - I will monitor if that should change anything.

Can you please tell me how to investigate this issue further?
Thank you in advance.
 
Hi,
Hello.

I call the proxmox backup client from within a bash script - on a daily basis using cron. The computer I am backing up is a desktop PC with linux Mint. For an unknown reason, much of the data get transferred everytime the script runs - even if the data has not been altered. Please take a look at the example log:

Code:
[ 2025-04-09 13:12:08 ] Backing up Katja's home directory...
Starting backup: host/Mint-workstation/2025-04-09T11:12:08Z
Client name: Mint-workstation
Starting backup protocol: Wed Apr  9 13:12:08 2025
Downloading previous manifest (Wed Apr  9 11:30:01 2025)
Upload directory '/home/katja' to 'backupuser@pbs@192.168.1.174:8007:backup02' as katja.pxar.didx
Error downloading .didx from previous manifest: Unable to open dynamic index "/mnt/backup/host/Mint-workstation/2025-04-09T09:30:01Z/katja.pxar.didx" - No such file or directory (os error 2)
katja.pxar: had to backup 1.917 GiB of 1.943 GiB (compressed 1.246 GiB) in 269.12s
katja.pxar: average backup speed: 7.296 MiB/s
katja.pxar: backup was done incrementally, reused 26.114 MiB (1.3%)
Uploaded backup catalog (534.827 KiB)
Duration: 269.70s
End Time: Wed Apr  9 13:16:38 2025

Data within the mountpoint /home/katja/ has been preexisting and has not been altered. So why are only 1.3% MB reused? This happens similar for other mountspoints as well - in one of them, more than 100GB are transferred over and over again. The script already ran for several weeks on a daily basis.

PBS has been on Version 3.2.X until today. I just updated to version 3.4 - I will monitor if that should change anything.

Can you please tell me how to investigate this issue further?
Thank you in advance.
do you have multiple backup jobs backup up to the same backup group, e.g. one for different folders? The reason the client re-processes all your data is because Error downloading .didx from previous manifest: Unable to open dynamic index "/mnt/backup/host/Mint-workstation/2025-04-09T09:30:01Z/katja.pxar.didx" - No such file or directory (os error 2), so it seems the previous snapshot does not contain any of the relevant data, and can therefore only reuse a small subset of the chunks.

If you want to backup multiple folders to the same backup group, either invoke the client via proxmox-backup-client bacukp dir1.pxar:/dir1 dir2.pxar:/dir2 or create sub-namespaces for them, see https://pbs.proxmox.com/docs/storage.html#backup-namespaces
 
  • Like
Reactions: dejhost