remote sync job - sync group vm/100 failed - No such file or directory (os error 2)

[alebeta]

Hi friends,

I have two PBS servers, one which I use for PVE backup and a second one where I want to do remote sync to store the data in a second location.

PBS for PVE version: 1.0-8
PBS Remote version: 1.0-8

The PBS server where I store the PVE backups use a NFS storage. It does works great and no problems had arise. In the other hand the PBS remote, is being configure with S3 storage mount with goofys. For the remote, the data store is added successful, but the error comers when the sync job runs and I see the logs:

2021-02-23T15:48:51+01:00: found 16 groups to sync
2021-02-23T15:48:53+01:00: sync snapshot "ct/100/2021-02-20T03:30:02Z"
2021-02-23T15:48:54+01:00: percentage done: 1.56% (0 of 16 groups, 1 of 4 group snapshots)
2021-02-23T15:48:54+01:00: sync group ct/100 failed - No such file or directory (os error 2)
2021-02-23T15:48:54+01:00: sync snapshot "ct/104/2021-02-20T03:30:01Z"
2021-02-23T15:48:55+01:00: percentage done: 7.81% (1 of 16 groups, 1 of 4 group snapshots)
2021-02-23T15:48:57+01:00: sync snapshot "vm/1006/2021-01-31T00:18:01Z"
2021-02-23T15:48:58+01:00: percentage done: 26.56% (4 of 16 groups, 1 of 4 group snapshots)
2021-02-23T15:48:58+01:00: sync group vm/1006 failed - No such file or directory (os error 2)
2021-02-23T15:48:58+01:00: sync snapshot "vm/113/2021-02-20T03:33:23Z"
2021-02-23T15:48:59+01:00: percentage done: 32.81% (5 of 16 groups, 1 of 4 group snapshots)
2021-02-23T15:48:59+01:00: sync group vm/113 failed - No such file or directory (os error 2)

When I check the logs on the goofys S3 mount, I can see the message

s3.DEBUG HEAD vm/905/2021-02-20T03:40:40Z/index.json.tmp/ = no such file or directory

I would like to know if some of you had the experience of using the remote jobs and with S3 and if you could provide some help would be fantastic.

thanks in advance

 

[Stefan_R]

I haven't looked into it too much, but just from the GitHub page of goofys (specifically this: https://github.com/kahing/goofys#current-status) I'd say it's probably a bad idea to use as backing storage for a PBS instance. Especially the atime/mtime support missing is going to cause issues with GC and such, and there's probably other incompatibilities as well...

 

[alebeta]

I haven't looked into it too much, but just from the GitHub page of goofys (specifically this: https://github.com/kahing/goofys#current-status) I'd say it's probably a bad idea to use as backing storage for a PBS instance. Especially the atime/mtime support missing is going to cause issues with GC and such, and there's probably other incompatibilities as well...

Hi,

thanks for the message and the useful information. Yes so far am doing some testing for remote PBS using a different kind of storages.

I tried s3fs which does not work at all for adding the data store and with goofys it does allow to add the datastore, but it fails when trying to perform the sync job.

At the moment I had been checking similar issues in goofys repository it seems a problem with the type of provider I use OVH Swift/S3

I will try to work a little bit more with goofys code and see if I can move any further.

Just to leave the info here I tried using Swift storage too with https://github.com/ovh/svfs but similar problem happens.

Do you know about another method to mount S3 and use it as datastore? If not rsync/rclone would still be the best choice at the moment to move backups to S3.

Thanks and all the best

 

[Stefan_R]

Do you know about another method to mount S3 and use it as datastore? If not rsync/rclone would still be the best choice at the moment to move backups to S3.

Manual sync is what we recommend at the moment, I am not aware of a "direct mount" variant that works without issues currently. We are considering adding official support for S3 and other storage providers as well, but no implementation is currently available. See also our bugtracker for more info.

 

[alebeta]

Manual sync is what we recommend at the moment, I am not aware of a "direct mount" variant that works without issues currently. We are considering adding official support for S3 and other storage providers as well, but no implementation is currently available. See also our bugtracker for more info.

Hi Stefan,

thanks, yes a native S3 implementation would be good too. So that one can use the PBS remote function with it.

At the moment I will keep moving with rclone.

thanks a lot