Feature request: use ctime instead of mtime to detect changes in "metadata" mode

aldem

Member
Mar 14, 2021
3
2
8
Düsseldorf, Germany
Hi,

After discovering this option in newer versions I was really happy, as this significantly reduces backup times, especially for big CTs where most of the data is either static or files are not large - thank you!

However, I was surprised to find, after reading the documentation, that metadata mode uses mtime to detect changes.

Unfortunately there is a pitfall; mtime could be arbitrarily manipulated, be it with malicious intent or not, and changed files could be undetected.

On the other hand, ctime is guaranteed (at least on ext4) to be changed if either content or file metadata (including ACLs and any other attribute) has changed, and it cannot be set by any means (on any FS, to my knowledge).

There was once a similar issue with restic: Restic uses mtime to detect file changes, which can miss changes - it describes in detail why it is not really safe to use mtime only.

I believe it is relatively easy to use ctime to detect changes, and it should make backup a bit more robust for edge cases. Obviously, restoration of ctime is not necessary (and not possible) in this case, it needs to be stored only for comparison.

I (and probably many more users) would appreciate it if you could add this at least as an option to metadata backup mode. It would be also nice to have it when doing manual backups (not only when running the job).

Thank you!
 
  • Like
Reactions: Chris
Hi,

After discovering this option in newer versions I was really happy, as this significantly reduces backup times, especially for big CTs where most of the data is either static or files are not large - thank you!

However, I was surprised to find, after reading the documentation, that metadata mode uses mtime to detect changes.

Unfortunately there is a pitfall; mtime could be arbitrarily manipulated, be it with malicious intent or not, and changed files could be undetected.

On the other hand, ctime is guaranteed (at least on ext4) to be changed if either content or file metadata (including ACLs and any other attribute) has changed, and it cannot be set by any means (on any FS, to my knowledge).

There was once a similar issue with restic: Restic uses mtime to detect file changes, which can miss changes - it describes in detail why it is not really safe to use mtime only.

I believe it is relatively easy to use ctime to detect changes, and it should make backup a bit more robust for edge cases. Obviously, restoration of ctime is not necessary (and not possible) in this case, it needs to be stored only for comparison.

I (and probably many more users) would appreciate it if you could add this at least as an option to metadata backup mode. It would be also nice to have it when doing manual backups (not only when running the job).

Thank you!
Hi,
thanks for your feedback, seems like I missed this thread during holiday season. Just stumbled over this now.

Please note that when using the change-detection-mode=metadata file metadata is always being re-encoded (including xattrs, file size, ecc.) in the metadata archive, only the file payload data is reused if possible. ctime is not feasible as CT filesystems might be synced to a tempdir before backup, therefore being always different in such cases.

Edit: Please do open a feature request issue in our bugtracker for this
 
Last edited:
  • Like
Reactions: Johannes S

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!