[SOLVED] How do I improve incremental ext4 FS backups?

[Big4SMK]

I'm a bit disappointed by the chunk reuse of my backups of a (piece of) my ext4 fileystem.
fstab:
/dev/mapper/vg_raid-lv_raid /mnt/raid ext4 defaults 0 0

mount:
/dev/mapper/vg_raid-lv_raid on /mnt/raid type ext4 (rw,relatime)

Running 2 backups right after one another, without any active use of the underlying directories i.e: no very active processes, the files are just sitting there.

proxmox-backup-client backup --backup-id private_sander private_sander.pxar:/mnt/raid/Private/sander --repository root@pam\!myvmhost@xxx.xxx.xxx.xxx:data --verbose

Here is the tail of the logs
private_sander.pxar.didx: had to upload 2.81 GiB of 8.55 GiB in 530.54s, average speed 5.42 MiB/s). private_sander.pxar.didx: backup was done incrementally, reused 5.74 GiB (67.2%) private_sander.pxar.didx: Reused 1895 from 2489 chunks. private_sander.pxar.didx: Average chunk size was 3.52 MiB. private_sander.pxar.didx: Average time per request: 213153 microseconds. append chunks list len (2) Uploaded backup catalog (460.27 KiB) catalog.pcat1.didx: Reused 0 from 2 chunks. catalog.pcat1.didx: Average chunk size was 230.13 KiB. catalog.pcat1.didx: Average time per request: 265444930 microseconds. Upload index.json to 'root@pam!myvmhost@xxx.xxx.xxx.xxx:8007:data' Duration: 531.18s End Time: Sun Mar 14 12:29:14 2021

private_sander.pxar.didx: had to upload 2.81 GiB of 8.55 GiB in 528.85s, average speed 5.43 MiB/s). private_sander.pxar.didx: backup was done incrementally, reused 5.74 GiB (67.2%) private_sander.pxar.didx: Reused 1895 from 2489 chunks. private_sander.pxar.didx: Average chunk size was 3.52 MiB. private_sander.pxar.didx: Average time per request: 212475 microseconds. append chunks list len (2) Uploaded backup catalog (460.27 KiB) catalog.pcat1.didx: Reused 0 from 2 chunks. catalog.pcat1.didx: Average chunk size was 230.13 KiB. catalog.pcat1.didx: Average time per request: 264596808 microseconds. Upload index.json to 'root@pam!myvmhost@xxx.xxx.xxx.xxx:8007:data' Duration: 529.48s End Time: Sun Mar 14 12:38:52 2021

I'm not seeing why more than 30% of the filedata has apparantly changed in the 9 minutes between the two backups.
1) any hints on what might have happened?
2) Is there a way for me to see which files have changed?
3) The full log shows me a lot of filenames, but those are not just the files that have changed, right?

root@MyVMHost:~# dpkg -l proxmox-backup-client Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=====================-============-============-================================= ii proxmox-backup-client 1.0.10-1 amd64 Proxmox Backup Client tools root@MyVMHost:~# uname -a Linux MyVMHost 5.4.101-1-pve #1 SMP PVE 5.4.101-1 (Fri, 26 Feb 2021 13:13:09 +0100) x86_64 GNU/Linux

 

[fabian]

might be hit by the issue mentioned over there in a thread with similar symptoms: https://forum.proxmox.com/threads/p...ackup-time-after-some-days.85703/#post-376587

 

[Big4SMK]

Ah great, I will be monitoring the repos to see when a build is available.

 

[Big4SMK]

@fabian, it looks like there is no new build on pvetest. Am I looking in the wrong place?

 

[fabian]

http://download.proxmox.com/debian/...md64/proxmox-backup-client_1.0.11-1.changelog contains the fix, among other things.

 

[Big4SMK]

It looks like that solved the issue! Only thing I'm wondering is: why does it take 50 minutes to figure that out?

Starting backup: host/shared/2021-03-19T12:16:22Z
Client name: MyVMHost
Starting backup protocol: Fri Mar 19 13:16:22 2021
Downloading previous manifest (Fri Mar 19 11:52:00 2021)
Upload directory '/mnt/raid/Shared' to 'root@pam!myvmhost@xxx.xxx.xxx.xxx:8007:data' as shared.pxar.didx
shared.pxar: had to upload 0 B of 614.57 GiB in 2987.88s, average speed 0 B/s).
shared.pxar: backup was done incrementally, reused 614.57 GiB (100.0%)
Uploaded backup catalog (1.74 MiB)
Duration: 2988.63s
End Time: Fri Mar 19 14:06:10 2021

 

[fabian]

because for host/ct backups, it needs to read (, chunk and hash) the data to figure out whether it changed. for (running) VMs, we can let Qemu keep track of which parts have changed, since it sits between the guest and storage and knows about all I/O. for arbitrary file systems, there is no such mechanism available that is fool proof.

 

[Big4SMK]

read: "your storage and/or hashing is slow"

Got it, thanks again. I'm becoming a huge PBS fan!

 

[fabian]

~200MB/s isn't that bad you can try poxmox-backup-client benchmark to get a rough idea whether it would be faster with faster storage.