Backup failed if files were deleted during backup

[achirkov]

$ proxmox-backup-client version
client version: 2.4.3

proxmox-backup-server/stable,now 3.0.2-1 amd64

I am trying to backup a folder. Some files were deleted during the backup and I am getting an I/O error. Is it possible to ignore the error when some files were deleted during the backup?

Upload directory '/data/' to 'nomad@pbs@my.dns.record:8007:/data' as data.pxar.didx
catalog upload error - channel closed
Error: error at "app/tenant/app/13/645913/578426/project": stat failed on "app/tenant/app/13/645913/578426/project/thumbs": EIO: I/O error

 

[Chris]

Hi, on what filesystem is the data located? The backup client handles vanished or even files changed while backing up just fine, but the EIO is unexpected and therefore returns with error.

P.S.: Not really related, but you should upgrade both, server and client to the latest versions.

 

[achirkov]

The data is located on glusterfs. All glusterfs nodes are connected, there are no errors in the logs \ gluster volume info. Free space on each node is 306 GB.

 

[Chris]

So I did try to reproduce your issue on a local test setup with glusterfs as data source, but was unable to reproduce the EIO error you reported. That could however be a timing issue: How reproducible is the error on your side?

I am however able to trigger a ESTALE stale file handle error when trying to access a files xattrs via the opened file handle, for cases where the file vanished while reading. That error should be non-fatal, as network filesystems tend to signal invalidated file handles that way. Working on a fix for that.

Edit: Created an issue for this in our bugtracker, see and subscribe for status updates https://bugzilla.proxmox.com/show_bug.cgi?id=5853

 

[Chris]

A patch for better handling of stale file handles has been send to the PBS devel mailing list [0].

[0] https://lore.proxmox.com/pbs-devel/20241105140153.282980-1-c.ebner@proxmox.com/T/

 

[achirkov]

How reproducible is the error on your side?

Of the 7 launches of proxmox-backup-client, 5 ended with an io error.
Thanks for your patch! I'll be waiting for the new release.

 

[Chris]

Of the 7 launches of proxmox-backup-client, 5 ended with an io error.
Thanks for your patch! I'll be waiting for the new release.

Note, the patches might not yet be included in an upcoming version of the proxmox-backup-client, depending on feedback and review. Also, this does not handle the EIO errors any differently, so will not help in your case.
Can you provide more details on how exactly you are reproducing the EIO error and what version of glusterfs is used? Further, did you already upgrade to the latest version of PBS (server and client)?

 

[achirkov]

glusterfs 10.1, os Ubuntu 22.04.3 LTS.
Yes, i update PBS server and client to latest version.
How i said, glusterfs status is ok, all nodes connected, ping between nodes 1-2ms.
mounted in fstab:

localhost:/ws /data glusterfs defaults,_netdev 0 0

Glusterfs mounted as docker volume in 16 docker containers