Receiving "pipelined request failed: update atime failed for chunk/file" when backing up to backup proxmox server?

[Nath2125]

Hi there,

one of my vms im hosting with proxmox its receiving this error when running a backup to my backup proxmox server over the network.

INFO: creating Proxmox Backup Server archive 'vm/107/2023-02-01T05:34:56Z'
INFO: started backup task '5fa01ab2-b4ef-4ad4-ae0b-2beaa9554452'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (5.7 GiB of 100.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 5.7 GiB dirty of 100.0 GiB total
INFO: 9% (556.0 MiB of 5.7 GiB) in 3s, read: 185.3 MiB/s, write: 184.0 MiB/s
INFO: 20% (1.1 GiB of 5.7 GiB) in 6s, read: 206.7 MiB/s, write: 204.0 MiB/s
INFO: 29% (1.7 GiB of 5.7 GiB) in 9s, read: 190.7 MiB/s, write: 189.3 MiB/s
INFO: 37% (2.1 GiB of 5.7 GiB) in 12s, read: 138.7 MiB/s, write: 137.3 MiB/s
INFO: 47% (2.7 GiB of 5.7 GiB) in 15s, read: 197.3 MiB/s, write: 192.0 MiB/s
INFO: 54% (3.1 GiB of 5.7 GiB) in 18s, read: 150.7 MiB/s, write: 149.3 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: update atime failed for chunk/file "/backups/.chunks/2625/2625304352cd863348c8036333b6312afbcd100ec5a4ce9bf913d46229fceed5" - EPERM: Operation not permitted
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 107 failed - backup write data failed: command error: write_data upload error: pipelined request failed: update atime failed for chunk/file "/backups/.chunks/2625/2625304352cd863348c8036333b6312afbcd100ec5a4ce9bf913d46229fceed5" - EPERM: Operation not permitted
INFO: Failed at 2023-02-01 16:35:15INFO: Backup job finished with errors
TASK ERROR: job errors

Not Sure why this is happening since my other vms are backing up regularly like scheduled and are not having any issues.

 

[fabian]

the server couldn't update the atime of the path referenced - is it always the same path? could you check with stat /backups/.chunks/2625/2625304352cd863348c8036333b6312afbcd100ec5a4ce9bf913d46229fceed5 on the PBS server?

 

[Nath2125]

is "stat /backups/.chunks/2625/2625304352cd863348c8036333b6312afbcd100ec5a4ce9bf913d46229fceed5" something thats ment to be pasted into a console? of the pbs or am i ment to search for it somewhere? if so where would i go about finding the console

 

[Nath2125]

Never mind i think i got it.

# stat /backups/.chunks/2625/2625304352cd863348c8036333b6312afbcd100ec5a4ce9bf913d46229fceed5
File: /backups/.chunks/2625/2625304352cd863348c8036333b6312afbcd100ec5a4ce9bf913d46229fceed5
Size: 1246051 Blocks: 2440 IO Block: 4096 regular file
Device: 3bh/59d Inode: 650207206942232087 Links: 1
Access: (0666/-rw-rw-rw-) Uid: ( 99/ UNKNOWN) Gid: ( 100/ users)
Access: 2023-01-31 00:01:49.404555757 -0500
Modify: 2023-01-29 08:06:34.122426042 -0500
Change: 2023-01-31 07:57:25.350004705 -0500
Birth: -

 

[Nath2125]

edit: Not sure if this helps either backup has 99:users not root:root is that an issue? although none of the other vms are having issues with backing up so maybe not permissions
I should mention i am hosting the backup proxmox as a container within my unraid server.

# ls -lh
total 0
drwxrwxrwx 1 99 users 65 Jan 30 00:10 backups
drwxr-xr-x 1 root root 1.2K Jan 1 14:52 bin
drwxr-xr-x 1 root root 0 Dec 9 14:15 boot
drwxr-xr-x 5 root root 340 Feb 1 00:24 dev
drwxr-xr-x 1 root root 1.9K Jan 11 05:25 etc
drwxr-xr-x 1 root root 0 Dec 9 14:15 home
drwxr-xr-x 1 root root 166 Jan 1 14:52 lib
drwxr-xr-x 1 root root 40 Dec 18 19:00 lib64
drwxr-xr-x 1 root root 0 Dec 18 19:00 media
drwxr-xr-x 1 root root 0 Dec 18 19:00 mnt
drwxr-xr-x 1 root root 0 Dec 18 19:00 opt
dr-xr-xr-x 1552 root root 0 Feb 1 00:24 proc
drwx------ 1 root root 30 Dec 18 19:00 root
drwxr-xr-x 4 root root 80 Feb 1 05:52 run
drwxr-xr-x 1 root root 76 Jan 1 16:58 runit
drwxr-xr-x 1 root root 2.1K Jan 1 14:52 sbin
drwxr-xr-x 1 root root 1.4K Jan 1 14:51 src
drwxr-xr-x 1 root root 0 Dec 18 19:00 srv
dr-xr-xr-x 13 root root 0 Jan 25 21:07 sys
drwxrwxrwt 1 root root 0 Jan 1 14:52 tmp
drwxr-xr-x 1 root root 84 Dec 18 19:00 usr
drwxr-xr-x 1 root root 90 Dec 18 19:00 var

 

[fabian]

permissions/ownership are definitely wrong.. the whole datastore including all files withing are supposed to be owned by the "backup" user/group with uid/gid 34..

 

[Nath2125]

Those permissions are for the proxmox server backup not for the proxmox itself I should add and if its backing up the rest of them from the proxmox server is it a permissions issue? What about the stats chunk log I printed? does that display anything of note?

 

[fabian]

yes, on the PBS system, the system user/group "backup" needs to own the datastore directory tree. it doesn't, and you get a permission related error, so first order of business is fixing the ownership/permissions

 

[Nath2125]

how do I change this container to have the "backup" user/group with uid/gid 34" permissions? as I'm new and not familiar with changing container permissions normally they are preset when a container is installed.

 

[fabian]

I don't know how unraid handles that. the command would be "chown -R backup:backup /backups", but it might require some changes on the unraid side as well before that works..

 

[Nath2125]

i can make variable permissions like this? would it be 34 34 for both? since u mentioned uid and gid? This photo was taken off another container

 

[fabian]

like I said - I don't know how unraid handles that you might be better off asking in unraid support channels if a plain chown doesn't work.

 

[Nath2125]

I'm now coming back into this thread I created since I'm now having sort of the same issues but in a different way. I went ahead and decided to scrap this container in my docker on unraid and use your PBS iso and install onto a vm within unraid. I created a fourm page for unraid and they think i should come here to look for help due to the permissions issue that is coming up.

INFO: starting new backup job: vzdump 108 --notes-template '{{guestname}}' --mode snapshot --remove 0 --storage unraidvm-proxbackupsvr --node pve1
INFO: Starting Backup of VM 108 (qemu)
INFO: Backup started at 2023-02-13 00:37:51
INFO: status = running
INFO: VM Name: Debian-Portainer
INFO: include disk 'scsi0' 'local-lvm:vm-108-disk-0' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/108/2023-02-12T13:37:51Z'
ERROR: VM 108 qmp command 'backup' failed - backup register image failed: command error: update atime failed for chunk/file "/mnt/proxbackupsvr/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" - EPERM: Operation not permitted
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 108 failed - VM 108 qmp command 'backup' failed - backup register image failed: command error: update atime failed for chunk/file "/mnt/proxbackupsvr/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" - EPERM: Operation not permitted
INFO: Failed at 2023-02-13 00:37:52
INFO: Backup job finished with errors
TASK ERROR: job errors

This is the error that now produces when I try to backup to my server

This is also the permissions that this "share" has, that stores the proxmox backups.

 

[Nath2125]

Someone suggested that on the unraid fourms

"Your permissions on the proxbackupsvr directory are 770 and they should probably be 777."

is that the case? and or would level permissions should it have inorder to properly use this share and access it.

 

[fabian]

no, the datastore just needs to be (fully) writable by the backup user/group (34/34), like I said.

 

[emoxam]

Can we go on here with my problem or should i create a new post ?
Got an error

Garbage collection failed: update atime failed for chunk/file "/opt/backup-store/.chunks/014a/014ab98c7468e734156e1d4b419db864e58fc5920fe7bcea56d170cc77cb3209.0.bad" - EPERM: Operation not permitted

But there is no such file

ls -l /opt/backup-store/.chunks/014a/014ab98c7468e734156e1d4b419db864e58fc5920fe7bcea56d170cc77cb3209.0.bad
ls: cannot access '/opt/backup-store/.chunks/014a/014ab98c7468e734156e1d4b419db864e58fc5920fe7bcea56d170cc77cb3209.0.bad': No such file or directory

that's all the content of this folder

ls -l /opt/backup-store/.chunks/014a/
total 40528
-rw-r--r-- 1 backup backup  663067 Jun 10 02:02 014a14eeda486ecb23adbad3ba4b4c69303360151da6c0c8af23ccbf29811678
-rw-r--r-- 1 backup backup 2384602 May 27 00:44 014a19b9845f7e5b7a2ba442302896475ed49778b43fddf2d35939c763ce12ec
-rw-r--r-- 1 backup backup   88462 Jun  3 01:17 014a1b2ea72769bcaba143b194b0aeb85bca12c74e2011919f390d22bc6169c8
-rw-r--r-- 1 backup backup 1025317 May  6 01:18 014a1e0d6b193bd26e952020ea8e3eb8dde12fb6903913082158138a2d8d6d51
-rw-r--r-- 1 backup backup  290632 Jun  3 01:22 014a2a2b8e6f65bafd6b2cbe1fb7c7e09cf6ed2c92e5532fa8b9079831636c25
-rw-r--r-- 1 backup backup 2067558 Jun  3 06:23 014a2fca393df702d83a76e06947c32d146e4f526197982149776b24e42d510d
-rw-r--r-- 1 backup backup   31652 Jun  3 05:13 014a35b0e81b9b82e6de2b51c6035648bd8ed617605203b64623e3e86b4d1d05
-rw-r--r-- 1 backup backup  558036 Jun  3 03:32 014a430d63a1612f08ccb52c8279c331577b64248fad4cf5d6335062b1e7f90b
-rw-r--r-- 1 backup backup  633734 May  6 01:08 014a5773f88a7c7426e358e4300e94d127077dd360a3e7d0d83f89074ede4e8b
-rw-r--r-- 1 backup backup 4194316 May  6 01:36 014a5a5e10260eabda496e892e7470c2871d898ac16967fd64b8d9d82d966831
-rw-r--r-- 1 backup backup 1123988 Jun  3 06:29 014a66de2341a2bfec4208e372bb1b7aa5dcbb05877be18c95e9b91eec6b37f7
-rw-r--r-- 1 backup backup 4194348 Jun  3 22:14 014a7f3d1ca3ad2056cb4186b686f6ac9db48b76993b7a0390cf332eb883756a
-rw-r--r-- 1 backup backup 1430578 Jun 10 01:49 014a844c62f43bf6bfe8d52f2b06afae140d44c0569f3f0661951594b512eb28
-rw-r--r-- 1 backup backup  770065 Jun  3 00:39 014aaf075a742bf8656f32ada095e2e49322b14efad2620c95858b85ae77e4c3
-rw-r--r-- 1 backup backup 4191266 Jun  3 02:16 014ab172e66d9220c3253101f4274bb4a96c61595630fca163933f3887b08597
-rw-r--r-- 1 backup backup 1076752 Jun  3 06:27 014ac4c6403153958e80fbe258cf13656d4cd65866a19c8338942711dc282c32
-rw-r--r-- 1 backup backup  309609 Jun  3 04:14 014acaf91026f9c8643a6ae18297ce39b211068d2503bef88693756e64d1d3e7
-rw-r--r-- 1 backup backup 4194348 Jun  3 05:13 014acb50889525056687a81ff2e4f3940bcad7cbded1b42525bef920c34b4601
-rw-r--r-- 1 backup backup  866951 Jun  3 06:18 014ad0ce06046431a58ce5991869d8d681db4038ff2d17031052f3301d408a6a
-rw-r--r-- 1 backup backup 3586765 May 27 00:37 014ad4931de616eeb4417eca3e7a0b16ba53b673ff6fd6e01a2d60682de5ee59
-rw-r--r-- 1 backup backup 2438076 Jun  3 00:59 014aec1f8378fe41a2db17be5ba14f7cc6fb7eec2ceaacf63ef6b7c104bad75a
-rw-r--r-- 1 backup backup 1133975 Apr 22 01:07 014af01468b45dd1212ca83848687088506109cef8b2de608657ee65f250d7a4
-rw-r--r-- 1 backup backup 4194348 Jun  3 01:34 014afe6a98b01fa7780c0724722925ebd36281e6e6f30f6ed0883d45d71f540d

 

[fabian]

that just means it went away between the GC and your check (possibly by another GC run in the meantime?)..

the error would not be fatal (or even printed) if the file had not existed at that point in time.

 

[emoxam]

that just means it went away between the GC and your check (possibly by another GC run in the meantime?)..

the error would not be fatal (or even printed) if the file had not existed at that point in time.

And this one ?

Garbage collection failed: unexpected error on datastore traversal: No such file or directory (os error 2) - "/opt/backup-store/host/dss-db-m11-s1-2/2023-06-10T00:15:05Z"

 

[fabian]

that one means that a snapshot has disappeared while we were trying to list snapshots. the code there can likely be improved to handle this better, since snapshots can disappear (GC doesn't hold a lock that prevents it, and shouldn't either). it is rather uncommon for that to trigger unless you have a slow storage and do lots of pruning, would be my guess

could you file a bug for it over at https://bugzilla.proxmox.com ? thanks!

 

[emoxam]

Pardon, forgot about your question. What should i wrote there ?
I got message "Garbage collection failed: unexpected error on datastore traversal: No such file or directory (os error 2) - "/opt/backup-store/host/dss-db-m11-s1-2/2023-06-10T00:15:05Z"" and i was asked to file a bug ?)
And Is it PVE ot PBS bug ?