better understand proxmox garbage collection - lots of avoidable utimensat() calls ?

[RolandK]

hello,

i'm trying to better understand verify/prune/garbage collection, which is an iops intensive process which keeps the pbs servers/storages i administer busy.

i'm often using fatrace as a utility for analysing filesystem access and i had a look at how proxmox-backup-server is accessing the datastores (they are on ext4 on my homebox).

for my curiosity, i see there is one chunk in both of my homeserver repos which is accessed orders of magnitudes more often then all the other chunks (for one repo roundabout 1000x more often) during gc run. anyhow, many chunks are being acessed repeatedly, but at a much lower rate.

i take a closer look (sort/uniq) into that because while watching fatrace output i saw repeating file pattern.

in strace, i you can see lots of utimensat() calls, which is the kernel function to change a files timestamp.

why does pbs repeatedly change timestamp(s) on the same file(s) over and over again ?

is it because underlying dedup/compression lib/engine does not know which chunks already been visited, because they are being referenced multiple times in the index files ?

wouldn't it be enough (in general) to change timestamp(s) once to mark chunks appropriately for being kept ?

wouldn't this "overhead" be worth to develop something more efficient to avoid all those syscalls to the io/vm subsystem ?

regards
roland


root@pve-t620:/backup# cat /root/fatrace-t620 |sort|uniq -c |sort -n
108 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/838e/838eec2398ca0c61aaec20b3022611eb645f3953b9282cb1e19ba97f78c29079
108 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/96bd/96bded1b06ce53dec40e75df401c732588db74884c8896dd709921adfa61e18c
108 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/c97b/c97b8ad269f88887ae459d0e3e852c8af5483d1cf6ac54b6e9a929a681573086
110 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/f281/f2811aba19cc144535ae14d1a1136d3dd9f72e9050748584b1ab0867ef98fdde
112 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/ebaf/ebaf841c4ece7aeab04b4fe7eed7a8ca7c648ed4e714335d6e45312ca7d09147
113 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/4483/4483184bb2b74361edb9c11aa1f77d7da4edeec7e98f2bdfe36a6b8dde41bed2
115 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/cc2a/cc2a3ddfafc3b1f777e1b7bca83386cc8bc2e28e695af94dbd319112fde74935
128 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/23f8/23f871bcefa3734e9e56f5cc1c858fdbfcbf0e3b103517a0c6aebab36967acdf
128 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/dd25/dd25b10089308e921a3109735e8e792552ad6f08bdd56e492d11baa270f3e4e5
34742 proxmox-backup-(1428): R /backup/pve-t620_backup/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8

root@pve-t620:/backup# cat /root/fatrace-bonn |sort |uniq -c |sort -n|tail
344 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/36e1/36e14d7001b211a9896a2a1c6d940bf72290ed6630d7b877890280dc0b7c26e5
345 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/5c47/5c47744f1959936b2bed3774ecbf958e9171893214616ebc3df8a9d3bddb96fe
345 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/65df/65df32fb7d26e0a7b2bfcf4ff8b427a2cc6811c80629bfb894b19ecf9a6243f4
345 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/8401/84014dc1a8bd57dc50bb4fdda050bbfc50b0b9102446a2b8a593b208eafd60e7
345 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/aa05/aa058bc455624d0a87c19003aa8206fcd0adddb9f37aa698a8aa1909f7bd336d
349 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/b0b4/b0b4a5798945e421dc99bd18725a681344311bf62191651a4f7a9524ae2fd7a7
355 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/6e43/6e43c5172b4684943ba394eb463f2fd4232eab2790344160500272baa2ae2cbf
363 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/74a1/74a1e1e4daddcebc9113b3440f1fea63d9f0624a9267eb209bb1ea9bea8edeea
997 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/e5b2/e5b244df0299989bdd78f565b733a3f6a7b1a737ec2ba0021490dc0e4bfa07bc
940079 proxmox-backup-(1428): R /backup/pve-maker-bonn_backup/.chunks/7b27/7b27b1eb4febae3273321255d8304e9b3e7938d9e254564bef859a4307a88638

 

[dietmar]

wouldn't it be enough (in general) to change timestamp(s) once to mark chunks appropriately for being kept ?

I guess this could be optimized if you keep the reference count in memory. So far we have tried to avoid
that to reduce memory usage.

wouldn't this "overhead" be worth to develop something more efficient to avoid all those syscalls to the io/vm subsystem ?

This depends on many factors and needs careful bench-marking ...