[SOLVED] Setup Reverse Proxy (nginx) for backup client

[haukenx]

Hi there,

we are just integrating PBS 1.1 as our primary backup solution (love it!). We would like to make it avaliable to backup-clients via an nginx reverse proxy. The proxy works perfectly fine with the Web-UI, but the backup clients fail with broken pipes:

Starting backup: host/myhost/2021-07-22T10:46:31Z
Client name: myhost
Starting backup protocol: Thu Jul 22 12:46:31 2021
Couldn't download previous manifest - broken pipe
Upload directory '/my/dir' to 'user@pbs!token@x.x.x.x:8007:Data' as dir.pxar.didx
HTTP/2.0 connection failed
catalog upload error - broken pipe
Error: broken pipe

Probably, I am just missing the correct way for configuring the nginx reverse-proxy:

server {
        listen  x.x.x.x:8007 ssl http2;
        client_max_body_size 0;

        ssl_certificate     /etc/nginx/ssl/some.crt;
        ssl_certificate_key /etc/nginx/ssl/some.key;

        server_name server.name;
        location / {
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "upgrade";
             proxy_pass https://y.y.y.y:8007/;
        }
}

with x.x.x.x being the proxy address and y.y.y.y being the address of the PBS-host. I found something in the forum for configuring apache2, but I cannot get the grip on it.

Can someone give me a starting point, please?

Kind regards
Hauke

 

[haukenx]

Noone? Or did I miss an obvious documentation ?

 

[SINOS]

Your config is not correct.

proxy_set_header Connection "upgrade" is the problem. it must be "Upgrade" instead of "upgrade", otherwise it leads to broken pipes.
This is mentioned in the offical nginx docs somewhere on this page: https://www.nginx.com/blog/websocket-nginx/


Please let me know if it works, PBS behind a reverse proxy sounds interesting.

 

[ph0x]

I see the benefit for the WebGUI. For example, I check for a user certificate. But what is the benefit of connecting to the pbs service via a reverse proxy?

 

[haukenx]

Hi Sinos,

Thanks for your reply and your help.

Your config is not correct.

proxy_set_header Connection "upgrade" is the problem. it must be "Upgrade" instead of "upgrade", otherwise it leads to broken pipes.
This is mentioned in the offical nginx docs somewhere on this page: https://www.nginx.com/blog/websocket-nginx/


Please let me know if it works, PBS behind a reverse proxy sounds interesting.

That alone was not yet the solution, I still get broken pipes. But further reading seems to show that nginx still does not support http2-reverse-proxying. I also gave it a try with the grpc-module, but with more or less the same results. Since http1-forwarding (to the UI) works like a a charm, I am starting to believe that I might have to switch to apache. I was just hoping that someone here got it already running with nginx.

Kind regards,
Hauke

 

[haukenx]

Hi ph0x,

I see the benefit for the WebGUI. For example, I check for a user certificate. But what is the benefit of connecting to the pbs service via a reverse proxy?

For me ... simply isolation. I do not want my vms to have a route into the network, where the backup server resides. So I decided to run a reverse proxy on each of my cluster nodes and provide an isolated ovs-bridge for my vms to access it.

|vm| <-> |storage transfer-network| <-> |nginx| <-> |backup network| <-> |pbs|

Maybe a little overkill (scnr), but fits my needs.

Kind regards,
Hauke

 

[Cookiefamily]

HAProxy does also support HTTP/2 and might be a bit more performant than apache2.

 

[haukenx]

Hi Cookiefamily,

HAProxy does also support HTTP/2 and might be a bit more performant than apache2.

I would expect a very simple config like:

frontend pbs
   mode http
   bind :8007 ssl crt /etc/haproxy/my.crt alpn h2,http/1.1
   default_backend servers

backend servers
   mode http
   server s1 y.y.y.y:8007 ssl alpn h2,http/1.1

,correct?

When I do so, the pbsclient's request is correctly received by the pbs server, who then agrees to switch protocols (http 101):

"GET //api2/json/backup?backup-id=<id>&backup-time=1627472290&backup-type=host&benchmark=false&debug=false&store=Data" 101 0 proxmox-backup-client/1.0

but the proxy seems to not understand and therefore returns a 502 (bad gateway).

pbs~ servers/s1 0/0/7/-1/12 502 340 - - PH-- 1/1/0/0/0 0/0 "GET //api2/json/backup?backup-id=<id>&backup-time=1627472916&backup-type=host&benchmark=false&debug=false&store=Data HTTP/1.1"

Any idea, what I am missing, here?

Kind regards,
Hauke

 

[haukenx]

Got it!

frontend pbs
   mode tcp
   bind :8007 ssl crt /etc/haproxy/my.crt alpn h2
   http-request redirect scheme https unless { ssl_fc }
   default_backend servers

backend servers
   mode tcp
   server s1 y.y.y.y:8007 ssl alpn h2

Just had to switch to tcp mode. Now it works!

Thanks for your help!

Hauke