Remote sync fails with 502 Bad gateway

E

etoel

New Member
May 19, 2023
8
0
1
I have a remote PBS that connects to my homelab through a HAproxy - this has worked flawlessly before.
After the upgrade to PBS 3.0-2 the sync fails with the following messages:

Code: 
2023-08-22T22:09:01+02:00: Starting datastore sync job 'pbs-prod:backup:Storage::s-04f39ca0-a9ff'
2023-08-22T22:09:01+02:00: sync datastore 'Storage' from 'pbs-prod/backup'
2023-08-22T22:09:01+02:00: ----
2023-08-22T22:09:01+02:00: Syncing datastore 'backup', root namespace into datastore 'Storage', root namespace
2023-08-22T22:09:01+02:00: found 5 groups to sync
2023-08-22T22:09:01+02:00: sync group ct/110
2023-08-22T22:09:01+02:00: sync group ct/110 failed - <html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>

So the remote can talk to the PBS, but fails when trying to transfer the container (it seems).

I have upgraded the HAproxy also, but I believe that HAproxy is not to blame here since the remote can reach the PBS.

What exactly is returning the 502 and why?

Edit: Forgot to include the log from the backup server (being pulled from) - note: timestamp is different, but error text the same
Code: 
2023-08-22T23:45:05+02:00: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
2023-08-22T23:45:05+02:00: protocol upgrade done
2023-08-22T23:45:05+02:00: TASK ERROR: connection error: connection closed before reading preface
 
Last edited:
how is your proxy configured?
 
It is a pfsense box with the HAproxy package. The frontend listens to 80/443/8007 and forwards it to a backend with internalIP:8007.
I have no issues at all reaching the web gui through this proxy, it is just the sync that now fails.
I only use IP4, but the hosts also have IP6 link local addresses.
 
The haproxy logs report the following:
Code: 
PBS_ipvANY/pbs 0/0/3/5/8 200 953 - - ---- 16/7/1/1/0 0/0 "POST //api2/json/access/ticket HTTP/1.1"  - initial
PBS_ipvANY/pbs 0/0/0/-1/2 502 360 - - PH-- 16/7/1/1/0 0/0 "GET //api2/json/reader?backup-id=105&backup-time=1691789400&backup-type=vm&debug=true&store=internalnvme HTTP/1.1"

I digged through old logs and before the upgrade it looked like this (date and target is different, but text is the same):

Code: 
PBS_ipvANY/pbs 0/0/2/6/8 200 958 - - ---- 17/6/0/0/0 0/0 "POST //api2/json/access/ticket HTTP/1.1"
PBS_ipvANY/pbs 0/0/0/1/90898 101 2041921071 - - ---- 25/10/0/0/0 0/0 "GET //api2/json/reader?backup-id=105&backup-time=1691357412&backup-type=vm&debug=true&store=internalnvme HTTP/1.1"
 
etoel said:
Code: 
PBS_ipvANY/pbs 0/0/0/-1/2 502 360 - - PH-- 16/7/1/1/0 0/0 "GET //api2/json/reader?backup-id=105&backup-time=1691789400&backup-type=vm&debug=true&store=internalnvme HTTP/1.1"
Click to expand...
the termination state says:

Code: 
PH---

which when i interpret this correctly means:
https://www.haproxy.org/download/2.7/doc/configuration.txt
P : the session was prematurely aborted by the proxy, because of a
connection limit enforcement, because a DENY filter was matched,
because of a security check which detected and blocked a dangerous
error in server response which might have caused information leak
(e.g. cacheable cookie).
Click to expand...

and
H : the proxy was waiting for complete, valid response HEADERS from the
server (HTTP only).
Click to expand...

so it seems the haproxy did abort the connection
 
Thank you for taking the time to dig into that :)
I do follow you, but my understanding is that if a deny rule was in place I would not be able to reach the server at all (and I do).

The H code is more interesting, has there been any changes to what the server responds with? The error appears right after the server reports "protocol upgrade done"
 
etoel said:
The H code is more interesting, has there been any changes to what the server responds with? The error appears right after the server reports "protocol upgrade done"
Click to expand...
not that i am aware of, from which version did you upgrade?
 
I first upgraded from 2.4 to 3.0-1 and that worked, it broke when I upgraded to 3.0-2.
I just did a very fast verification from a fresh install inside my homelab (not using the haproxy) and I was able to sync just fine between the two so it has to be something the haproxy suddenly expects :(

I do not expect you to spend time on this of course, but could you confirm that there are no changes in PBS in 3.0-2 that could affect this?
 
i quickly checked the source between those versions, and there was nothing that i could identify which should cause such a change (so nothing api/http related changed on our side..)
 
I'm having the same issue. Listing the backups works without any issues, writing a new one shows 502. The only difference I found in HAproxy log is that during backups, there are two slashes at the beginning of the path:

Code: 
Nov 3 17:43:49 fw haproxy[73209]: xx.xx.xx.xx:51834 [03/Nov/2023:17:43:49.356] pbs.domain.tld~ pbs.domain.tld_ipvANY/xx.xx.xx.xx_8007 0/0/6/17/24 200 9354 - - ---- 5/1/0/0/0 0/0 "GET /api2/json/admin/datastore/datastore-name/snapshots?backup-id=ID HTTP/1.1"
Nov 3 17:43:50 fw haproxy[73209]: xx.xx.xx.xx:51846 [03/Nov/2023:17:43:50.907] pbs.domain.tld~ pbs.domain.tld_ipvANY/xx.xx.xx.xx_8007 0/0/11/-1/26 502 326 - - PH-- 5/1/0/0/0 0/0 "GET //api2/json/backup?backup-id=ID&backup-time=1234567890&backup-type=vm&benchmark=false&debug=false&store=datastore-name HTTP/1.1"

EDIT: I have now changed the PBS connection from HAproxy to NAT, and provided the correct fingerprint—both listing the previous backups and also writing a new backup worked without any issues. This has to be an issue with HAproxy in some way... Still looking forward to any possible solutions!
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom