error updating atime on garbage collection

[Por12]

Good morning,

I've been having issues on garbage collection on one of my datasets. The backups and the verification process work as well.

2024-07-02T00:00:00+02:00: starting garbage collection on store pbs-high-value
2024-07-02T00:00:00+02:00: task triggered by schedule 'daily'
2024-07-02T00:00:00+02:00: Start GC phase1 (mark used chunks)
2024-07-02T00:01:52+02:00: marked 1% (5 of 498 index files)
2024-07-02T00:02:42+02:00: marked 2% (10 of 498 index files)
2024-07-02T00:02:53+02:00: marked 3% (15 of 498 index files)
2024-07-02T00:03:45+02:00: marked 4% (20 of 498 index files)
2024-07-02T00:04:49+02:00: marked 5% (25 of 498 index files)
2024-07-02T00:06:39+02:00: marked 6% (30 of 498 index files)
2024-07-02T00:06:45+02:00: TASK ERROR: update atime failed for chunk/file "/mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9" - EACCES: Permission denied

This is the mount info:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=972940k,nr_inodes=243235,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=201224k,mode=755,inode64)
/dev/mapper/pbs-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6436)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/vda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
pbs-high-value on /mnt/pbs-high-value type virtiofs (rw,relatime)
pbs-low-value on /mnt/pbs-low-value type virtiofs (rw,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=201220k,nr_inodes=50305,mode=700,inode64)

Any ideas? Thanks!

 

[fabian]

what are the permissions of that file? please also check on the hypervisor side since it seems to be a passed-in mountpoint via virtiofs

 

[Por12]

Thanks Fabian. Here's the info:

From the Hypervisor:

-rw-r--r-- 1 34 34 466375 Apr 27 13:15 /mnt/user/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9

From the PBS VM:

-rw-r--r-- 1 backup backup 466375 Apr 27 13:15 /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9

 

[fabian]

that looks okay in principle.. could you do `stat` on that path both ends and post the output?

 

[Por12]

Sure:

Hypervisor:

File: /mnt/user/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
  Size: 466375          Blocks: 912        IO Block: 4096   regular file
Device: 0,56    Inode: 648799833160127497  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/ UNKNOWN)   Gid: (   34/ UNKNOWN)
Access: 2024-06-10 00:06:41.592044211 +0200
Modify: 2024-04-27 13:15:15.977699927 +0200
Change: 2024-06-10 00:06:41.592044211 +0200
 Birth: -

PBS VM:

File: /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
  Size: 466375          Blocks: 912        IO Block: 4096   regular file
Device: 0,38    Inode: 648799833160127497  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2024-06-10 00:06:41.592044211 +0200
Modify: 2024-04-27 13:15:15.977699927 +0200
Change: 2024-06-10 00:06:41.592044211 +0200
 Birth: -

 

[fabian]

what happens if you run "touch /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9" inside the VM as user backup? what if you do it as user root?

 

[Por12]

How do I touch as backup user? Sorry for the dummy question.

 

[fabian]

runuser -u backup -g backup touch /...

 

[Por12]

Ok. With root user, no errors. With backup user I get a permission denied error.

runuser -u backup -g backup touch /mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9
touch: cannot touch '/mnt/pbs-high-value/.chunks/65f9/65f985af2d86767b079017009afa4e8a2ffad390ad86eb54c4c06985722293a9': Permission denied

 

[fabian]

what about the parent directories? is it just that particular chunk., or are others also "untouchable"?

 

[Por12]

Hi Fabian,

I cannot touch anything inside /.chunks/65f9/ using the backup user.

runuser -u backup -g backup touch /mnt/pbs-high-value/.chunks/65f9/test
touch: cannot touch '/mnt/pbs-high-value/.chunks/65f9/test': Permission denied

I can touch inside /.chunks/ with the backup user.

Thanks

 

[fabian]

how are the permissions of the 65f9 directory?