PBS Sync job fails with SSL error after 8min up to 30mins

[johen]

HI

I have the problem that longer sync jobs fail after a random time and I'm currently out of ideas....
Different to this case https://forum.proxmox.com/threads/failed-backup.91772/
the ssl error appear after a random time between 8min - 30min. During this time backup snapshots gets successfully transfered and with starting the sync job again, I'm able to step by step sync the datastores...

thanks for any idea

best regards
JOchen

Both Backup Server are Version 2.2-3

2022-08-06T10:37:30+02:00: Starting datastore sync job 'pve0pbs:4TBext4:10TBext4::s-96faba2b-839d'
2022-08-06T10:37:30+02:00: sync datastore '10TBext4' from 'pve0pbs/4TBext4'
2022-08-06T10:37:30+02:00: ----
2022-08-06T10:37:30+02:00: Syncing datastore '4TBext4', root namespace into datastore '10TBext4', root namespace
2022-08-06T10:37:30+02:00: found 3 groups to sync
2022-08-06T10:37:30+02:00: re-sync snapshot ct/111/2022-08-06T00:32:09Z
2022-08-06T10:37:30+02:00: no data changes
2022-08-06T10:37:30+02:00: re-sync snapshot ct/111/2022-08-06T00:32:09Z done
2022-08-06T10:37:30+02:00: percentage done: 33.33% (1/3 groups)
2022-08-06T10:37:30+02:00: skipped: 16 snapshot(s) (2022-06-13T00:32:30Z .. 2022-08-05T00:32:04Z) older than the newest local snapshot
2022-08-06T10:37:30+02:00: re-sync snapshot vm/109/2022-08-06T00:30:00Z
2022-08-06T10:37:30+02:00: no data changes
2022-08-06T10:37:30+02:00: re-sync snapshot vm/109/2022-08-06T00:30:00Z done
2022-08-06T10:37:30+02:00: percentage done: 66.67% (2/3 groups)
2022-08-06T10:37:30+02:00: skipped: 16 snapshot(s) (2022-06-13T00:30:01Z .. 2022-08-05T00:30:02Z) older than the newest local snapshot
2022-08-06T10:37:30+02:00: re-sync snapshot vm/110/2022-07-24T00:30:07Z
2022-08-06T10:37:30+02:00: no data changes
2022-08-06T10:37:30+02:00: re-sync snapshot vm/110/2022-07-24T00:30:07Z done
2022-08-06T10:37:30+02:00: percentage done: 74.51% (2/3 groups, 4/17 snapshots in group #3)
2022-08-06T10:37:30+02:00: sync snapshot vm/110/2022-07-25T00:30:06Z
2022-08-06T10:37:30+02:00: sync archive qemu-server.conf.blob
2022-08-06T10:37:30+02:00: sync archive drive-scsi1.img.fidx
2022-08-06T10:39:42+02:00: downloaded 0 bytes (0.00 MiB/s)
2022-08-06T10:39:42+02:00: sync archive drive-scsi0.img.fidx
2022-08-06T10:42:05+02:00: downloaded 3992006310 bytes (26.78 MiB/s)
2022-08-06T10:42:05+02:00: sync archive drive-efidisk0.img.fidx
2022-08-06T10:42:05+02:00: downloaded 0 bytes (0.00 MiB/s)
2022-08-06T10:42:05+02:00: got backup log file "client.log.blob"
2022-08-06T10:42:05+02:00: sync snapshot vm/110/2022-07-25T00:30:06Z done
2022-08-06T10:42:05+02:00: percentage done: 76.47% (2/3 groups, 5/17 snapshots in group #3)
2022-08-06T10:42:05+02:00: sync snapshot vm/110/2022-07-26T00:30:07Z
2022-08-06T10:42:05+02:00: sync archive qemu-server.conf.blob
2022-08-06T10:42:05+02:00: sync archive drive-scsi1.img.fidx
2022-08-06T10:42:09+02:00: downloaded 100695474 bytes (26.31 MiB/s)
2022-08-06T10:42:09+02:00: sync archive drive-scsi0.img.fidx
2022-08-06T10:44:33+02:00: downloaded 4773108008 bytes (31.78 MiB/s)
2022-08-06T10:44:33+02:00: sync archive drive-efidisk0.img.fidx
2022-08-06T10:44:33+02:00: downloaded 0 bytes (0.00 MiB/s)
2022-08-06T10:44:33+02:00: got backup log file "client.log.blob"
2022-08-06T10:44:33+02:00: sync snapshot vm/110/2022-07-26T00:30:07Z done
2022-08-06T10:44:33+02:00: percentage done: 78.43% (2/3 groups, 6/17 snapshots in group #3)
2022-08-06T10:44:33+02:00: sync snapshot vm/110/2022-07-27T00:30:08Z
2022-08-06T10:44:33+02:00: sync archive qemu-server.conf.blob
2022-08-06T10:44:33+02:00: sync archive drive-scsi1.img.fidx
2022-08-06T10:44:36+02:00: downloaded 63646586 bytes (26.23 MiB/s)
2022-08-06T10:44:36+02:00: sync archive drive-scsi0.img.fidx
2022-08-06T10:45:34+02:00: percentage done: 80.39% (2/3 groups, 7/17 snapshots in group #3)
2022-08-06T10:45:34+02:00: sync group vm/110 failed - error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:
2022-08-06T10:45:34+02:00: Finished syncing namespace , current progress: 2 groups, 7 snapshots
2022-08-06T10:45:34+02:00: TASK ERROR: sync failed with some errors.

 

[dcsapak]

is there some firewall in between that could be the issue?

 

[johen]

is there some firewall in between that could be the issue?

Hi Dominik,
thanks for your reply

no I don't have any firewall between the PBS.
to me the error behaviour doesn't fit to a firewall blocking something, as the error sometimes doesn't appear for 30min and is random

best regards
Jochen

 

[dcsapak]

did you already check the network stack (cables/switches) and the hardware? (memory test, etc.)
the error can have many causes, e.g. the issue here: https://github.com/openssl/openssl/issues/11727
mentions that in that case it was a 'bad' router with port forwarding (thats why i asked if there is a firewall)

 

[jenssen99]

see also https://forum.proxmox.com/threads/pbs-sync-failed-each-time.113921/#post-492989

 

[johen]

Thanks!
just also found it and wanted to update this thread and saw your reply with the same...
I have used following fix
https://gist.github.com/brunneis/0c27411a8028610117fefbe5fb669d10