Sync job fails after some time

[mike2012]

I am just testing the sync job over the internet for offsite backup.
Everything starts fine.
Some snapshots get synced.
But after some hours there is a error.

2021-02-17T20:27:30+01:00: Starting datastore sync job 'XXXXXXXXXX'
2021-02-17T20:27:30+01:00: Sync datastore 'POEI' from 'XXXXXXX'
2021-02-17T20:27:30+01:00: found 4 groups to sync
2021-02-17T20:27:31+01:00: re-sync snapshot "vm/2001/2021-02-09T23:15:02Z"
2021-02-17T20:27:31+01:00: no data changes
2021-02-17T20:27:31+01:00: re-sync snapshot "vm/2001/2021-02-09T23:15:02Z" done
2021-02-17T20:27:31+01:00: percentage done: 17.39% (0 of 4 groups, 16 of 23 group snapshots)
2021-02-17T20:27:31+01:00: sync snapshot "vm/2001/2021-02-10T23:15:02Z"
2021-02-17T20:27:31+01:00: sync archive qemu-server.conf.blob
2021-02-17T20:27:31+01:00: sync archive drive-virtio1.img.fidx
2021-02-17T20:34:57+01:00: downloaded 528148066 bytes (1.13 MiB/s)
2021-02-17T20:34:57+01:00: sync archive drive-virtio0.img.fidx
2021-02-17T21:27:26+01:00: downloaded 3738745860 bytes (1.13 MiB/s)
2021-02-17T21:27:26+01:00: got backup log file "client.log.blob"
2021-02-17T21:27:26+01:00: sync snapshot "vm/2001/2021-02-10T23:15:02Z" done
2021-02-17T21:27:26+01:00: percentage done: 18.48% (0 of 4 groups, 17 of 23 group snapshots)
2021-02-17T21:27:26+01:00: sync snapshot "vm/2001/2021-02-11T23:15:02Z"
2021-02-17T21:27:26+01:00: sync archive qemu-server.conf.blob
2021-02-17T21:27:26+01:00: sync archive drive-virtio1.img.fidx
2021-02-17T21:34:09+01:00: downloaded 475241569 bytes (1.13 MiB/s)
2021-02-17T21:34:09+01:00: sync archive drive-virtio0.img.fidx
2021-02-17T22:15:20+01:00: downloaded 2928444593 bytes (1.13 MiB/s)
2021-02-17T22:15:20+01:00: got backup log file "client.log.blob"
2021-02-17T22:15:20+01:00: sync snapshot "vm/2001/2021-02-11T23:15:02Z" done
2021-02-17T22:15:20+01:00: percentage done: 19.57% (0 of 4 groups, 18 of 23 group snapshots)
2021-02-17T22:15:21+01:00: sync snapshot "vm/2001/2021-02-12T23:15:02Z"
2021-02-17T22:15:21+01:00: sync archive qemu-server.conf.blob
2021-02-17T22:15:21+01:00: sync archive drive-virtio1.img.fidx
2021-02-17T22:20:05+01:00: downloaded 334462520 bytes (1.13 MiB/s)
2021-02-17T22:20:05+01:00: sync archive drive-virtio0.img.fidx
2021-02-17T23:14:02+01:00: downloaded 3842997318 bytes (1.13 MiB/s)
2021-02-17T23:14:02+01:00: got backup log file "client.log.blob"
2021-02-17T23:14:02+01:00: sync snapshot "vm/2001/2021-02-12T23:15:02Z" done
2021-02-17T23:14:02+01:00: percentage done: 20.65% (0 of 4 groups, 19 of 23 group snapshots)

2021-02-17T23:14:05+01:00: sync group vm/2001 failed - permission check failed.
2021-02-17T23:14:08+01:00: sync group vm/2002 failed - authentication failed - invalid ticket - expired
2021-02-17T23:14:11+01:00: sync group vm/2003 failed - authentication failed - invalid ticket - expired
2021-02-17T23:14:14+01:00: sync group vm/2004 failed - authentication failed - invalid ticket - expired
2021-02-17T23:14:14+01:00: TASK ERROR: sync failed with some errors.

What could be the reason and how to solve ?

 

[fabian]

which package versions are installed (on the pulling side)?

 

[mike2012]

Both sides are on no subscription repository fully upgraded

 

[fabian]

anything in the logs on the other side?

 

[fabian]

to work around it you could setup an API token instead of a user and use that to configure the remote access - the token + secret is static so there is nothing that can expire

 

[mike2012]

Here the log from the other side

Feb 17 23:14:02 pbs proxmox-backup-api[812]: pam_unix(proxmox-backup-auth:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root
Feb 17 23:14:04 pbs proxmox-backup-api[812]: authentication failure; rhost=::ffff:xx.xx.xx.xx user=root@pam msg=AUTH_ERR (7)
Feb 17 23:14:05 pbs proxmox-backup-api[812]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:xx.xx.xx.xx]:54358] permission check failed.
Feb 17 23:14:08 pbs proxmox-backup-proxy[840]: GET /api2/json/admin/datastore/ex1/snapshots?backup-id=2002&backup-type=vm: 401 Unauthorized: [client [::ffff:80.123.184.186]:54360] authentication failed - invalid ticket - expired
Feb 17 23:14:11 pbs proxmox-backup-proxy[840]: GET /api2/json/admin/datastore/ex1/snapshots?backup-id=2003&backup-type=vm: 401 Unauthorized: [client [::ffff:80.123.184.186]:54360] authentication failed - invalid ticket - expired
Feb 17 23:14:14 pbs proxmox-backup-proxy[840]: GET /api2/json/admin/datastore/ex1/snapshots?backup-id=2004&backup-type=vm: 401 Unauthorized: [client [::ffff:80.123.184.186]:54360] authentication failed - invalid ticket - expired

 

[tom]

Both sides are on no subscription repository fully upgraded

Fully upgrade is a bit too less information.

Please post the detailed version of your packages (Proxmox VE and Proxmox Backup Server)

 

[mike2012]

Both Backupserver on same version

proxmox-backup 1.0-4 running kernel: 5.4.78-2-pve
proxmox-backup-server 1.0.8-1 running version: 1.0.8
pve-kernel-5.4 6.3-3
pve-kernel-helper 6.3-3
pve-kernel-5.4.78-2-pve 5.4.78-2
pve-kernel-5.4.65-1-pve 5.4.65-1
ifupdown2 3.0.0-1+pve3
libjs-extjs 6.0.1-10
proxmox-backup-docs 1.0.8-1
proxmox-backup-client 1.0.8-1
proxmox-mini-journalreader 1.1-1
proxmox-widget-toolkit 2.4-4
pve-xtermjs 4.7.0-3
smartmontools 7.1-pve2
zfsutils-linux 0.8.5-pve1

 

[mike2012]

The workaround with token worked, thx.

For everyone who is interrested how to use the credentials
as example

UserID : root@pam!tockenname
Password: Secret tocken u get when u create API Token

And dont forget to set the permissions for the token as i did first

https://pbs.proxmox.com/docs/managing-remotes.html