Sync Job http request timed out

[benka]

Hello,

I have two proxmox backup servers. One offiste to archive the backups from the first one.

The offsite server (offsite) gets the backups from the first one (backup1) with sync jobs. This works for multiple datastores except for one.
This datastore have about 2520 hosts. Sometimes the sync works but most of the time it failes at the beginning with following error message:

2022-10-26T13:37:45+02:00: Starting datastore sync job 'backup1:datastore:datastore::s-dc7537f1-4c85'
2022-10-26T13:37:45+02:00: sync datastore 'datastore' from 'backup1/datastore'
2022-10-26T13:37:45+02:00: ----
2022-10-26T13:37:45+02:00: Syncing datastore 'datastore', root namespace into datastore 'datastore', root namespace
2022-10-26T13:39:45+02:00: Encountered errors while syncing namespace - Failed to retrieve backup groups from remote - http request timed out
2022-10-26T13:39:45+02:00: TASK ERROR: sync failed with some errors.

So it seems that there is a problem with initial starting the sync job for this one datastore. But if it started sucessfully there are no problems and the sync job succeeds.

I upgraded proxmox backup to 2.2-7 but the behavior still exists.

Is there any idea how to fix this problem? Or any idea how to debug this kind of problem? I also tried to start the sync from the command line ($ proxmox-backup-manager pull) but I can't assign an owner so this does not work:

owner check failed (root@pam != datastore@pbs)

Best regards,
Benni

 

[sterzy]

Hi,

I'd need to go through the sync job logic again, but if I remember correctly it queries the remote for a list of backup groups that need to be synced. It seems that this request times out (likely due to the amount of backup groups you have in you root name space).

One thing that you could try as a work-around is to split up your datastore into multiple name spaces. That way the sync job will query each name space for a list of backup groups separetelly. Resulting in more requests, but each should take less time to carry out. This should be more stable. Any chance you could try setting that up?

 

[benka]

Hi,

I'd need to go through the sync job logic again, but if I remember correctly it queries the remote for a list of backup groups that need to be synced. It seems that this request times out (likely due to the amount of backup groups you have in you root name space).

One thing that you could try as a work-around is to split up your datastore into multiple name spaces. That way the sync job will query each name space for a list of backup groups separetelly. Resulting in more requests, but each should take less time to carry out. This should be more stable. Any chance you could try setting that up?

That's exactly the problem. I splitted up the datastore in multiple name spaces and the problem was gone. But only for a while. after the backup host items increased there are timeout problems again. This time syncing some spaces with large host item numbers.
Maybe there will be a solution pbs dealing with slow HDD drives in the future.