[SOLVED] Does not follow HTTP standard

M

McTwist

New Member
Jun 14, 2023
9
2
3
Sweden
I found an issue regarding putting PBS behind a reverse proxy. I have tried Apache, Caddy and Nginx. For Nginx it "just works", while for the others it stops working when PBS tries to initiate an upgrade to their own protocol, resulting in PBS responding that the Upgrade header is missing.

I did some forensic on Apache, got some logs on Caddy and also briefly checked the code in Caddy, and after reading the Connection/Upgrade standards, PBS seem to be missing sending the Connection header to initiate the upgrade, which the standard requires. Caddy seems to require Connection, so they follow this standard, and apparently Apache does this as well, in relation to the same response they both give. However, Nginx seems to not follow this standard, hence why it "just works".

I wonder if this is really the case with PBS, that when trying to upgrade the connection to proxmox-backup-protocol-v1 or proxmox-backup-reader-protocol-v1 it wont send the Connection header? If so, is it considered a bug or a feature? I would say a bug, because it does not follow the standard.

An additional issue, not regarding to PBS but good for anyone stumbling upon this issue, is that Apache proxy module does not support upgrade names longer than 16 characters. It is possible to get around this with the rewrite module.
 
I'm joining in with the same problem. The bug report is marked fixed, but it's still broken for me.

(headers) {
header -x-powered-by
header -server
header Content-Security-Policy "frame-ancestors 'self'; upgrade-insecure-requests;"
header X-LB-Server {system.hostname}
header X-LB-Requested-Host {host}
header X-LB-Server {system.hostname}
header X-Forwarded-Port {http.request.port}
header X-Forwarded-For {http.request.header.CF-Connecting-IP}
header X-Real-IP {http.request.header.CF-Connecting-IP}
encode zstd gzip
}

pbs.x.dev, pbs.x.net {
import cloudflare
import headers
reverse_proxy https://192.168.2.58:8007 {
transport http {
tls_insecure_skip_verify
}
}
}

I am on the latest build of PBS. I would greatly appreciate a solution that will work with my architecture using a cloudflare tunnel.
 
McTwist said:
I found an issue regarding putting PBS behind a reverse proxy. I have tried Apache, Caddy and Nginx. For Nginx it "just works", while for the others it stops working when PBS tries to initiate an upgrade to their own protocol, resulting in PBS responding that the Upgrade header is missing.

I did some forensic on Apache, got some logs on Caddy and also briefly checked the code in Caddy, and after reading the Connection/Upgrade standards, PBS seem to be missing sending the Connection header to initiate the upgrade, which the standard requires. Caddy seems to require Connection, so they follow this standard, and apparently Apache does this as well, in relation to the same response they both give. However, Nginx seems to not follow this standard, hence why it "just works".

I wonder if this is really the case with PBS, that when trying to upgrade the connection to proxmox-backup-protocol-v1 or proxmox-backup-reader-protocol-v1 it wont send the Connection header? If so, is it considered a bug or a feature? I would say a bug, because it does not follow the standard.

An additional issue, not regarding to PBS but good for anyone stumbling upon this issue, is that Apache proxy module does not support upgrade names longer than 16 characters. It is possible to get around this with the rewrite module.
Click to expand...
@McTwist did you find a solution w/Caddy?
 
ctrlbrk said:
@McTwist did you find a solution w/Caddy?
Click to expand...
I checked with both Apache and Caddy and somehow it still doesn't work. The errors for either of them seem to be different too. One of the developers managed to get Apache running on his end, but I failed to replicate their config, so I suspect that there is something else you and I are missing. I have put this on hold for now, but will eventually come back to it later in the future when I can spend my full attention on it.
 
McTwist said:
I checked with both Apache and Caddy and somehow it still doesn't work. The errors for either of them seem to be different too. One of the developers managed to get Apache running on his end, but I failed to replicate their config, so I suspect that there is something else you and I are missing. I have put this on hold for now, but will eventually come back to it later in the future when I can spend my full attention on it.
Click to expand...

I added these two headers and now have a different error message: sync group vm/101 failed - invalid protocol name

reverse_proxy https://192.168.2.58:8007 {
transport http {
tls_insecure_skip_verify
}
header_up Connection {http.request.header.Connection}
header_up Upgrade {http.request.header.Upgrade}
}
 
@McTwist I hope you can try my last post settings and see if you get it working. If so lets compare configs. I have two PBS on different continents and both behind Caddy. one works as a remote for the other. In one direction it works fine, and in the other direction I get these errors (now invalid protocol name). Yet both of my Caddy's are the same best I can tell!!
 
I have decided to make use of nginx instead, because it "just works" and I currently do not want to spend more time figuring out how to get this to work, especially when the developers themselves cannot replicate it. If I dive more into this in the future, I will probably check other alternatives too, like haproxy and traefik.
 
McTwist said:
I have decided to make use of nginx instead, because it "just works" and I currently do not want to spend more time figuring out how to get this to work, especially when the developers themselves cannot replicate it. If I dive more into this in the future, I will probably check other alternatives too, like haproxy and traefik.
Click to expand...
There's been another bug report, so I'm in the process of trying to replicate it again.

It seems that for some reverse proxies, the response to the upgrade request "doesn't make it" - it's as if it's never been received in the first place, even though we definitely send the correct response. Thus, the reverse proxy sends a response with an HTTP 200 instead of forwarding the server's response with the 101, cancelling the protocol upgrade process.`

I'm not sure why this doesn't happen for nginx. Caddy at least is affected by this. So, I'm about to use yet another proxy to see if I can sniff the packets being sent and received and do some analysis with Wireshark. The Caddy source code doesn't seem to suggest that anything's broken either, and I cannot find any related issues or bug reports. It honestly eludes me why specifically the 101 response gets trashed.

If nginx is the only thing that's confirmed to work at the moment, I suggest just sticking with it.
 
Max Carrara said:
There's been another bug report, so I'm in the process of trying to replicate it again.

It seems that for some reverse proxies, the response to the upgrade request "doesn't make it" - it's as if it's never been received in the first place, even though we definitely send the correct response. Thus, the reverse proxy sends a response with an HTTP 200 instead of forwarding the server's response with the 101, cancelling the protocol upgrade process.`

I'm not sure why this doesn't happen for nginx. Caddy at least is affected by this. So, I'm about to use yet another proxy to see if I can sniff the packets being sent and received and do some analysis with Wireshark. The Caddy source code doesn't seem to suggest that anything's broken either, and I cannot find any related issues or bug reports. It honestly eludes me why specifically the 101 response gets trashed.

If nginx is the only thing that's confirmed to work at the moment, I suggest just sticking with it.
Click to expand...
Thank you for taking your time to look into this issue further. It is indeed peculiar that 101 is not being sent, so one could assume there are two reasons why this is the case:

1. While PBS does send the correct response, somehow there is something if fails to do that all the other services require.
2. Nginx is implementing upgrade correctly, and the other reverse proxies somehow fail it. However, I too see no issues in Caddy source for it to fail, unless it modifies the status code. Maybe running Caddy in debug mode may retrieve additional information.

From your perspective, we can assume that the first one is incorrect, therefore there may be something on the other services, but it sounds impossible that there is only one service that have implemented this correctly.

I will stick to nginx for now, but want to switch to something else in the future if possible. I was not in a position to be able to do this at all, but have spent some time to prepare the move to nginx very soon. I later can switch to any reverse proxy if desired.
 
I don't want to return to nginx. I worked another workaround solely for this issue but it's unique to my deployment.

I really want to see this fixed in PBS.
 
McTwist said:
but it sounds impossible that there is only one service that have implemented this correctly.
Click to expand...
Exactly, this is what baffles me - there must be some other underlying issue (regarding PBS).

McTwist said:
Maybe running Caddy in debug mode may retrieve additional information.
Click to expand...
The only thing I discovered is that Caddy complains that it got an empty string for the Upgrade header from PBS. This initially led me to think that we don't send the header at all, but we do. Without a reverse proxy the response has all the expected headers, including Upgrade. That's why I will analyze the TCP flows between Caddy and PBS.

I'll keep you posted!
 
I can't believe it, but I got it working.

This was probably one of the most vexing debugging odysseys I have ever ventured on, but nevertheless, I learnt a lot.


Something I want to elaborate on:

Max Carrara said:
The only thing I discovered is that Caddy complains that it got an empty string for the Upgrade header from PBS.
Click to expand...

Caddy does actually receive the Upgrade header (it shows up in the debug logs), but it won't pass the header along internally if the Connection header isn't included in the server's response. This is, to my own surprise, not explicitly specified in any of the HTTP/1.1 and HTTP/2 RFCs. Just in the examples.

The message of the patch I just sent to the mailing list describes everything in detail. If you're curious, you may read it over here: https://lists.proxmox.com/pipermail/pbs-devel/2024-February/007947.html

With this patch, Caddy is most definitely guaranteed to work with PBS (and nginx continues to work). I have yet to test other reverse proxies, but unless they reject the Connection header in the response (for god knows what reason) they should also work.

Thank you all for your patience!
 
  • Like
Reactions: VictorSTS, McTwist and fluxX04
As they work with ordinary websockets, like PVE/PBS console, it most definitely will work. I've already moved to Nginx, but this gives me more reason to move to Caddy for the proxy. Thank you for your excellent work, and lets hope that it does not continue to be a problem for all other proxies.
 
  • Like
Reactions: Max Carrara
Version 2 of the patch recently got merged - the fix will most likely be available in package proxmox-backup-server version 3.1.5-1 or later.

The commit message contains a bunch of extra info in case you're curious.

Happy proxying!
 
  • Like
Reactions: Lukas Wagner and McTwist
This is great news! That something so simple, although small fix, could result in such a hunt toward correct solution that it took almost 9 months before it was actually resolved properly. I am very impressed by your work and the help everyone have provided.
 
  • Like
Reactions: Max Carrara
You must log in or register to reply here.
Top Bottom