[SOLVED] error updating atime on garbage collection

P

Por12

Member
Mar 6, 2023
59
3
13
Good morning,

I've been having issues on garbage collection on one of my datasets. The backups and the verification process work as well.

Code: 
2024-07-02T00:00:00+02:00: starting garbage collection on store pbs-high-value
2024-07-02T00:00:00+02:00: task triggered by schedule 'daily'
2024-07-02T00:00:00+02:00: Start GC phase1 (mark used chunks)
2024-07-02T00:01:52+02:00: marked 1% (5 of 498 index files)
2024-07-02T00:02:42+02:00: marked 2% (10 of 498 index files)
2024-07-02T00:02:53+02:00: marked 3% (15 of 498 index files)
2024-07-02T00:03:45+02:00: marked 4% (20 of 498 index files)
2024-07-02T00:04:49+02:00: marked 5% (25 of 498 index files)
2024-07-02T00:06:39+02:00: marked 6% (30 of 498 index files)
2024-07-02T00:06:45+02:00: TASK ERROR: update atime failed for chunk/file "/mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9" - EACCES: Permission denied

This is the mount info:

Code: 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=972940k,nr_inodes=243235,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=201224k,mode=755,inode64)
/dev/mapper/pbs-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6436)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/vda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
pbs-high-value on /mnt/pbs-high-value type virtiofs (rw,relatime)
pbs-low-value on /mnt/pbs-low-value type virtiofs (rw,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=201220k,nr_inodes=50305,mode=700,inode64)

Any ideas? Thanks!
 
what are the permissions of that file? please also check on the hypervisor side since it seems to be a passed-in mountpoint via virtiofs
 
Thanks Fabian. Here's the info:

From the Hypervisor:
Code: 
-rw-r--r-- 1 34 34 466375 Apr 27 13:15 /mnt/user/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9

From the PBS VM:
Code: 
-rw-r--r-- 1 backup backup 466375 Apr 27 13:15 /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
 
that looks okay in principle.. could you do `stat` on that path both ends and post the output?
 
Sure:

Hypervisor:
Code: 
File: /mnt/user/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
  Size: 466375          Blocks: 912        IO Block: 4096   regular file
Device: 0,56    Inode: 648799833160127497  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/ UNKNOWN)   Gid: (   34/ UNKNOWN)
Access: 2024-06-10 00:06:41.592044211 +0200
Modify: 2024-04-27 13:15:15.977699927 +0200
Change: 2024-06-10 00:06:41.592044211 +0200
 Birth: -

PBS VM:
Code: 
File: /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
  Size: 466375          Blocks: 912        IO Block: 4096   regular file
Device: 0,38    Inode: 648799833160127497  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2024-06-10 00:06:41.592044211 +0200
Modify: 2024-04-27 13:15:15.977699927 +0200
Change: 2024-06-10 00:06:41.592044211 +0200
 Birth: -
 
what happens if you run "touch /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9" inside the VM as user backup? what if you do it as user root?
 
runuser -u backup -g backup touch /...
 
Ok. With root user, no errors. With backup user I get a permission denied error.

Code: 
runuser -u backup -g backup touch /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
touch: cannot touch '/mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9': Permission denied
 
what about the parent directories? is it just that particular chunk., or are others also "untouchable"?
 
Hi Fabian,

I cannot touch anything inside /.chunks/65f9/ using the backup user.

Code: 
runuser -u backup -g backup touch /mnt/pbs-high-value/.chunks/65f9/test
touch: cannot touch '/mnt/pbs-high-value/.chunks/65f9/test': Permission denied

I can touch inside /.chunks/ with the backup user.

Thanks
 
Last edited:
how are the permissions of the 65f9 directory?
 
Here they are:

Code: 
ls -l /mnt/pbs-high-value/.chunks/65f9/
total 3104
-rw-r--r-- 1 backup backup 2711284 Dec 28  2023 65f902d11932522e00bc544deb19fffebfc270a13cc9f7ff0e568c81c8ed7132
-rw-r--r-- 1 backup backup  466375 Jul  2 18:43 65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9

Code: 
ls -l /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
-rw-r--r-- 1 backup backup 466375 Jul  2 18:43 /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9

Thanks!
 
I really meant the permissions of the directory ;) e.g., adding `-a` to the ls call, or calling stat on it should do the trick!
 
Oh I see. Thanks!

Code: 
stat /mnt/pbs-high-value/.chunks/65f9/
  File: /mnt/pbs-high-value/.chunks/65f9/
  Size: 162             Blocks: 0          IO Block: 4096   directory
Device: 0,38    Inode: 648799833160127517  Links: 1
Access: (0644/drw-r--r--)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2023-09-18 11:18:25.099039364 +0200
Modify: 2024-04-27 13:15:15.978699919 +0200
Change: 2024-06-10 14:01:18.482546616 +0200
 Birth: -
 
please do "chmod ug+x /mnt/pbs-high-value/.chunks/65f9/", that should solve the issue. please also double check the other dirs (well, GC will likely tell you if others are missing those bits anyway ;))
 
Very good! I had to change another directory but the GC process finally run to the end.

Checking all the logs I found that I have another (intermittent) error. This is the part of the log that returns the error (same dataset as before; full log attached). Any ideas here? It's the same directory so I don't know if was caused by the same permission error.

Code: 
2024-07-04T00:07:29+02:00: verify pbs-high-value:ct/501/2023-12-29T09:01:42Z
2024-07-04T00:07:29+02:00:   check pct.conf.blob
2024-07-04T00:07:29+02:00:   check root.pxar.didx
2024-07-04T00:10:31+02:00: can't verify chunk, load failed - store 'pbs-high-value', unable to load chunk '65f902d11932522e00bc544deb19fffebfc270a13cc9f7ff0e568c81c8ed7132' - Permission denied (os error 13)
2024-07-04T00:10:31+02:00: could not rename corrupted chunk "/mnt/pbs-high-value/.chunks/65f9/65f902d11932522e00bc544deb19fffebfc270a13cc9f7ff0e568c81c8ed7132" - Permission denied (os error 13)
2024-07-04T00:10:32+02:00:   verified 4948.81/16119.09 MiB in 182.04 seconds, speed 27.19/88.55 MiB/s (1 errors)

Thanks Fabian!
 

Attachments

Last edited:
probably. but you can try re-running the verification for that snapshot to make sure, now that the permissions are hopefully fixed.
 
Hello,

The solution in this post still works, but I need to do it every couple months. Any idea how to find the root cause of the permission error?

Best regards
 
You must log in or register to reply here.
Top Bottom