Proxmox Sync Job - Connection reset by peer

alterman1994

New Member
Aug 6, 2021
15
4
3
30
Hello Guys,

We have 2 PBS servers in different locations (inside 1 city), identical hardware configuration:

E5-2640V3 (12 Core - 2.4Ghz)
16GB PC4-2133P DIMM
2 x 250 GB SSD (OS, ZFS Mirror)
12 x 4 TB HDD (storage, ZFS RAIDZ2)

Version: Backup Server 2.0-9

One of the servers connected to our PVE cluster for backups, all good there.

Today, I've configured the second one (pbs2) and connected to the first one (pbs1) via Remotes. I've created sync job for existing backups and run it manually. For some reason, it randomly fails backup groups:

Code:
2021-08-27T21:28:11+02:00: TASK ERROR: sync failed with some errors.

Two types:

Code:
2021-08-27T21:24:01+02:00: sync group vm/129 failed - broken pipe

and

Code:
2021-08-27T21:09:58+02:00: sync group vm/127 failed - connection reset

On pbs1 node, I'm getting the following via journalctl:

Code:
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: GET /chunk
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: download chunk "/mnt/datastore/pbs1_storage/.chunks/ef75/ef75b017e09d33dd89f17c3584965c5514826efe3c77b5cfc3c25168>
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: GET /chunk
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: download chunk "/mnt/datastore/pbs1_storage/.chunks/4a8e/4a8e22c3101f761ff30827288d52cc1094a04bc70d793a78e864c715>
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: TASK ERROR: connection error: Connection reset by peer (os error 104)
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: starting new backup reader datastore 'pbs1_storage': "/mnt/datastore/pbs1_storage"
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: protocol upgrade done
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: GET /download
Aug 27 21:24:01 pbs1 proxmox-backup-proxy[1827287]: download "/mnt/datastore/pbs1_storage/vm/130/2021-08-26T10:06:00Z/index.json.blob"



We don't have any firewall rules on the server, network seems to be stable. I tried to start sync job once again and it failed on the same VM groups (screenshot attached)

Screenshot_353.png

Have you maybe faced with something similar?

Any assistance is appreciated, thank you in advance!
 
Meanwhile, on pbs2 I've found out the following in /var/log/syslog:


Code:
Aug 27 21:24:01 pbs2 proxmox-backup-proxy[18001]: HTTP/2.0 connection failed
Aug 27 21:24:01 pbs2 proxmox-backup-proxy[18001]: removing backup snapshot "/mnt/datastore/pbs2_storage/vm/129/2021-08-25T13:46:14Z"
 
Aug 27 21:24:01 pbs2 proxmox-backup-proxy[18001]: HTTP/2.0 connection failed
do you have any firewall/proxy/middlebox in between ? if yes, make sure that it supports/passes through http 2.0
 
@dcsapak thank you for getting back.

not something really

the logic is :
1) primary pbs1 located in the Datacenter (connected to Juniper EX4600-40F without any firewall rules configured)
2) secondary pbs2 located in the office and connected to pbs1 as Remotes (transparent switch connected to Mikrotik router)

I don't see that something really blocks HTTP 2.0 connection since part of the backups are synced to slave:


Screenshot_466.pngScreenshot_467.png

My guess that something kills connection. Does PBS has some timeout settings for HTTP 2.0 connections?

Thank you!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!