Proxmox Sync Job - Connection reset by peer

[alterman1994]

Hello Guys,

We have 2 PBS servers in different locations (inside 1 city), identical hardware configuration:

E5-2640V3 (12 Core - 2.4Ghz)
16GB PC4-2133P DIMM
2 x 250 GB SSD (OS, ZFS Mirror)
12 x 4 TB HDD (storage, ZFS RAIDZ2)

Version: Backup Server 2.0-9

One of the servers connected to our PVE cluster for backups, all good there.

Today, I've configured the second one (pbs2) and connected to the first one (pbs1) via Remotes. I've created sync job for existing backups and run it manually. For some reason, it randomly fails backup groups:

2021-08-27T21:28:11+02:00: TASK ERROR: sync failed with some errors.

Two types:

2021-08-27T21:24:01+02:00: sync group vm/129 failed - broken pipe

and

2021-08-27T21:09:58+02:00: sync group vm/127 failed - connection reset

On pbs1 node, I'm getting the following via journalctl:

Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: GET /chunk
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: download chunk "/mnt/datastore/pbs1_storage/.chunks/ef75/ef75b017e09d33dd89f17c3584965c5514826efe3c77b5cfc3c25168>
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: GET /chunk
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: download chunk "/mnt/datastore/pbs1_storage/.chunks/4a8e/4a8e22c3101f761ff30827288d52cc1094a04bc70d793a78e864c715>
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: TASK ERROR: connection error: Connection reset by peer (os error 104)
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: starting new backup reader datastore 'pbs1_storage': "/mnt/datastore/pbs1_storage"
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: protocol upgrade done
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: GET /download
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: download "/mnt/datastore/pbs1_storage/vm/130/2021-08-26T10:06:00Z/index.json.blob"

We don't have any firewall rules on the server, network seems to be stable. I tried to start sync job once again and it failed on the same VM groups (screenshot attached)



Have you maybe faced with something similar?

Any assistance is appreciated, thank you in advance!

 

[alterman1994]

Meanwhile, on pbs2 I've found out the following in /var/log/syslog:

Aug 27 21:24:01 pbs2 proxmox-backup-proxy[18001]: HTTP/2.0 connection failed
Aug 27 21:24:01 pbs2 proxmox-backup-proxy[18001]: removing backup snapshot "/mnt/datastore/pbs2_storage/vm/129/2021-08-25T13:46:14Z"

 

[dcsapak]

Aug 27 21:24:01 pbs2 proxmox-backup-proxy[18001]: HTTP/2.0 connection failed

do you have any firewall/proxy/middlebox in between ? if yes, make sure that it supports/passes through http 2.0

 

[alterman1994]

@dcsapak thank you for getting back.

not something really

the logic is :
1) primary pbs1 located in the Datacenter (connected to Juniper EX4600-40F without any firewall rules configured)
2) secondary pbs2 located in the office and connected to pbs1 as Remotes (transparent switch connected to Mikrotik router)

I don't see that something really blocks HTTP 2.0 connection since part of the backups are synced to slave:




My guess that something kills connection. Does PBS has some timeout settings for HTTP 2.0 connections?

Thank you!

 

[dcsapak]

My guess that something kills connection

could be, i'd check the firewalls

Does PBS has some timeout settings for HTTP 2.0 connections?

not after an established connection afaik