question regarding pbs lockfiles

[RolandK]

i'm struggling with some weird performance issues in my pbs (seems to be an issue with writes/stalls on the rootfs) and while investigating, i came across pbs lockfiles

from my understanding , lockfiles make sure that some process may have exclusive access to some ressource.

what i don't understand is, why do most of the pbs lockfiles in /etc/proxmox-backup have old timestamp from 2020 and won't update ?

with fatrace, i see that there are lots and opens or even writes (at least i see "W" in fatrace) to datastore.lck , but i don't see any corresponding timestamp change.

there is currently some verify active on my pbs, so i wonder why i can see sub-second locking activity for the datastore.

can someone explain how this works ?

then i see with strace that the .datastore.lck is getting opened with r/w and append flags, but apparently nothing is getting written to it.

furthermore, doesn't unix paradigm define lockfiles lockfiles to be stored in /var/lock instead of /etc ? ( https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s09.html )

regards
roland

root@pbs01:/etc/proxmox-backup# ls -la .*.lck
-rw-r--r-- 1 root root 0 Oct 27 2020 .acl.lck
-rw-r--r-- 1 backup backup 0 Oct 27 2020 .datastore.lck
-rw-rw---- 1 backup backup 0 Jan 11 13:58 .node.lck
-rw-rw---- 1 backup backup 0 Jul 6 2022 .prune.lck
-rw-r--r-- 1 root root 0 Oct 27 2020 .user.lck
-rw-r--r-- 1 root root 0 Dec 16 2020 .verification.lck

root@pbs01:/etc/proxmox-backup# stat .*.lck
File: .acl.lck
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 252,1 Inode: 525372 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-10-27 16:15:21.015759138 +0100
Modify: 2020-10-27 16:15:21.015759138 +0100
Change: 2020-10-27 16:15:21.015759138 +0100
Birth: 2020-10-27 16:15:21.015759138 +0100
File: .datastore.lck
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 252,1 Inode: 525370 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 34/ backup) Gid: ( 34/ backup)
Access: 2024-01-13 00:17:52.135147539 +0100
Modify: 2020-10-27 16:12:14.539232478 +0100
Change: 2022-12-02 13:25:44.746483075 +0100
Birth: 2020-10-27 16:12:14.539232478 +0100
File: .node.lck
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 252,1 Inode: 533013 Links: 1
Access: (0660/-rw-rw----) Uid: ( 34/ backup) Gid: ( 34/ backup)
Access: 2024-01-11 13:58:49.309733280 +0100
Modify: 2024-01-11 13:58:49.309733280 +0100
Change: 2024-01-11 13:58:49.309733280 +0100
Birth: 2024-01-11 13:58:49.309733280 +0100
File: .prune.lck
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 252,1 Inode: 524395 Links: 1
Access: (0660/-rw-rw----) Uid: ( 34/ backup) Gid: ( 34/ backup)
Access: 2022-07-06 11:40:54.099204151 +0200
Modify: 2022-07-06 11:40:54.099204151 +0200
Change: 2022-07-06 11:40:54.099204151 +0200
Birth: 2022-07-06 11:40:54.099204151 +0200
File: .user.lck
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 252,1 Inode: 525374 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-10-27 16:16:06.988678797 +0100
Modify: 2020-10-27 16:16:06.988678797 +0100
Change: 2020-10-27 16:16:06.988678797 +0100
Birth: 2020-10-27 16:16:06.988678797 +0100
File: .verification.lck
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 252,1 Inode: 525451 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-12-16 18:04:21.267469232 +0100
Modify: 2020-12-16 18:04:21.267469232 +0100
Change: 2020-12-16 18:04:21.267469232 +0100
Birth: 2020-12-16 18:04:21.267469232 +0100

root@pbs01:/etc/proxmox-backup# fatrace -t -c|grep lck
11:22:30.569188 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:30.569655 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:30.692484 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:30.692864 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:30.875491 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:30.875931 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:31.047277 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:31.047570 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:31.051673 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:31.052001 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:31.498113 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:31.498570 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:31.551914 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:31.552348 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:31.587321 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:31.587673 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:31.713974 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck
11:22:31.714337 proxmox-backup-(2812): CW /etc/proxmox-backup/.datastore.lck
11:22:33.574324 proxmox-backup-(2812): O /etc/proxmox-backup/.datastore.lck

root@pbs01:~/proxmox-backup.source# strace -f -p 2812 2>&1 |grep -i lck
[pid 226329] openat(AT_FDCWD, "/etc/proxmox-backup/.datastore.lck", O_RDWR|O_APPEND|O_CLOEXEC) = 32

 

[ggoller]

Hi!
these `.lck` files are used to lock the config files `.cfg` in `/etc/proxmox-backup/`.
The atime/mtime/etc.. is not updated because we simply open [0] them (which does not change the atime) and lock the file descriptor using flock [1].

Using `fatrace` I didn't see any "W", only "O" (normal open) and "CW" (close write). Nevertheless, we never write nor read to these files.

We lock/unlock every time we read and write from the config file, so during the verify-job we simply need to lookup the datastore config more often (f.e. because we need to list_snapshots, list_groups, check_privs).

We are currently working on moving the lockfiles for the datastore and single snapshots to `/var/run` (which is tmpfs), maybe we could include the .lck files for the config files as well... Although this whole thing is very complicated and critical, especially if you consider updating pbs and having multiple instances with different versions running at the same time.

[0]: https://www.man7.org/linux/man-pages/man2/open.2.html
[1]: https://www.man7.org/linux/man-pages/man2/flock.2.html

 

[RolandK]

ah, fantastic, thanks for the explanation!