.pxarexclude not working as expected

[msproact]

I have:

root@tuxmaster:/srv/pve# cat .pxarexclude
/dump/*
/images/*
/template/cache/*

root@tuxmaster:~# find / -name "vm-10001-disk-0.qcow2"
/srv/pve/images/10001/vm-10001-disk-0.qcow2

Yet

root@tuxmaster:~# LANG=en strace -e openat -fff proxmox-backup-client backup  --include-dev /boot/efi --include-dev /srv root.pxar:/

does this:

[pid 32405] openat(8, "pve", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 9
[pid 32405] openat(9, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 10
[pid 32405] openat(9, "dump", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 10
[pid 32405] openat(10, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32405] openat(9, "images", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 10
[pid 32405] openat(10, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32405] openat(10, "10001", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 11
[pid 32405] openat(11, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32405] openat(11, "vm-10001-cloudinit.qcow2", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 12
[pid 32405] openat(11, "vm-10001-disk-0.qcow2", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 12

Why?

Due to past experiences with rsync exclude patterns I also tried

/dump/**
/images/**
/template/cache/**

and:

/dump/**/*
/images/**/*
/template/cache/**/*

Both did not manage to exclude those VM images I backed up already as VM backups with PBS either.

 

[Alwin]

/images/**/*

Can you try without the last *?

 

[msproact]

Sure.

Does not help either.

root@tuxmaster:~# cat /srv/pve/.pxarexclude
/dump/**/
/images/**/
/template/cache/**/

still leads to:

[pid  4101] openat(9, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 10
[pid  4101] openat(9, "dump", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 10
[pid  4101] openat(10, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  4101] openat(9, "images", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 10
[pid  4101] openat(10, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  4101] openat(10, "10001", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 11
[pid  4101] openat(11, ".pxarexclude", O_RDONLY|O_NOCTTY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  4101] openat(11, "vm-10001-cloudinit.qcow2", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 12
[pid  4101] openat(11, "vm-10001-disk-0.qcow2", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 12

 

[t.lamprecht]

/dump/*
/images/*
/template/cache/*

Can you try dropping the initial slash / then it should work. That slash will be allowed in a future release, as it should be OK and else the documentation could be misleading for some as its "pattern root is the location of .pxarexclude file" suggests that a root / slash can be used - and here it's IMO better to fix the code, not change the docs.

 

[t.lamprecht]

Oh, and if you do not want to check the directories at all then just do:

dump/
images/
template/cache/

The trailing * glob is normally only required for a path if one wants to exclude most but still include specific files in that directory tree. If that's not required then using just the path is faster as it allows us to not even traverse in that directory.

 

[msproact]

Thank you very much, Thomas. This works.

So at the moment the documentation is a bit misleading.

Will let the backup run through now.

 

[msproact]

One thing I found is that with --verbose proxmox-backup-client shows which chunks it is writing, however it does not show which files it backups. An option to see just the files it tries to backup would be helpful to debug excludes without having to resort to strace

 

[msproact]

First backup:

root.pxar.didx: Uploaded 40596267257 bytes as 8854 chunks in 797 seconds (48 MB/s).
root.pxar.didx: Average chunk size was 4585076 bytes.
root.pxar.didx: Time per request: 90017 microseconds.
catalog.pcat1.didx: Uploaded 1633725 bytes as 3 chunks in 797 seconds (0 MB/s).
catalog.pcat1.didx: Average chunk size was 544575 bytes.
catalog.pcat1.didx: Time per request: 265673309 microseconds.

Second backup directly after that:

root.pxar.didx: Uploaded 40530972180 bytes as 8838 chunks in 136 seconds (284 MB/s).
root.pxar.didx: Average chunk size was 4585989 bytes.
root.pxar.didx: Time per request: 15395 microseconds.
catalog.pcat1.didx: Uploaded 1633773 bytes as 3 chunks in 136 seconds (0 MB/s).
catalog.pcat1.didx: Average chunk size was 544591 bytes.
catalog.pcat1.didx: Time per request: 45360199 microseconds.

Nice

 

[t.lamprecht]

One thing I found is that with --verbose proxmox-backup-client shows which chunks it is writing, however it does not show which files it backups. An option to see just the files it tries to backup would be helpful to debug excludes without having to resort to strace

Yeah, this is rather left over debugging information than verbose user information. Personally I'd rather like that the options is an integer with 0 being not-verbose, 1 showing some info, 2 all files it picks up and only a higher verbosity level showing the chunks and other such, rather internal, information.

 

[msproact]

Like me to file an enhancement about that?

 

[t.lamprecht]

Yeah, feel free to do so at https://bugzilla.proxmox.com/ thanks!

 

[msproact]

Bug 2849 - proxmox-backup-client: provide a way to see which file it is working on