[SOLVED] Does not follow HTTP standard

[McTwist]

I found an issue regarding putting PBS behind a reverse proxy. I have tried Apache, Caddy and Nginx. For Nginx it "just works", while for the others it stops working when PBS tries to initiate an upgrade to their own protocol, resulting in PBS responding that the Upgrade header is missing.

I did some forensic on Apache, got some logs on Caddy and also briefly checked the code in Caddy, and after reading the Connection/Upgrade standards, PBS seem to be missing sending the Connection header to initiate the upgrade, which the standard requires. Caddy seems to require Connection, so they follow this standard, and apparently Apache does this as well, in relation to the same response they both give. However, Nginx seems to not follow this standard, hence why it "just works".

I wonder if this is really the case with PBS, that when trying to upgrade the connection to proxmox-backup-protocol-v1 or proxmox-backup-reader-protocol-v1 it wont send the Connection header? If so, is it considered a bug or a feature? I would say a bug, because it does not follow the standard.

An additional issue, not regarding to PBS but good for anyone stumbling upon this issue, is that Apache proxy module does not support upgrade names longer than 16 characters. It is possible to get around this with the rewrite module.

 

[fiona]

Hi,
please open a bug report for this, so it won't get lost in the forum: https://bugzilla.proxmox.com/

 

[mstefan]

Are there any news on this?
Got the same Problem with PBS behind traefik reverse proxy.

 

[fiona]

Hi,

Are there any news on this?
Got the same Problem with PBS behind traefik reverse proxy.

yes, there is a corresponding bug report. Can you provide the requested details there?

 

[mstefan]

I opened my own thread for this.
Dont want to hijack this one.
https://forum.proxmox.com/threads/error-missing-upgrade-header-pbs-behind-traefik.131327/

 

[ctrlbrk]

I'm joining in with the same problem. The bug report is marked fixed, but it's still broken for me.

(headers) {
header -x-powered-by
header -server
header Content-Security-Policy "frame-ancestors 'self'; upgrade-insecure-requests;"
header X-LB-Server {system.hostname}
header X-LB-Requested-Host {host}
header X-LB-Server {system.hostname}
header X-Forwarded-Port {http.request.port}
header X-Forwarded-For {http.request.header.CF-Connecting-IP}
header X-Real-IP {http.request.header.CF-Connecting-IP}
encode zstd gzip
}

pbs.x.dev, pbs.x.net {
import cloudflare
import headers
reverse_proxy https://192.168.2.58:8007 {
transport http {
tls_insecure_skip_verify
}
}
}

I am on the latest build of PBS. I would greatly appreciate a solution that will work with my architecture using a cloudflare tunnel.

 

[ctrlbrk]

I found an issue regarding putting PBS behind a reverse proxy. I have tried Apache, Caddy and Nginx. For Nginx it "just works", while for the others it stops working when PBS tries to initiate an upgrade to their own protocol, resulting in PBS responding that the Upgrade header is missing.

I did some forensic on Apache, got some logs on Caddy and also briefly checked the code in Caddy, and after reading the Connection/Upgrade standards, PBS seem to be missing sending the Connection header to initiate the upgrade, which the standard requires. Caddy seems to require Connection, so they follow this standard, and apparently Apache does this as well, in relation to the same response they both give. However, Nginx seems to not follow this standard, hence why it "just works".

I wonder if this is really the case with PBS, that when trying to upgrade the connection to proxmox-backup-protocol-v1 or proxmox-backup-reader-protocol-v1 it wont send the Connection header? If so, is it considered a bug or a feature? I would say a bug, because it does not follow the standard.

An additional issue, not regarding to PBS but good for anyone stumbling upon this issue, is that Apache proxy module does not support upgrade names longer than 16 characters. It is possible to get around this with the rewrite module.

@McTwist did you find a solution w/Caddy?

 

[McTwist]

@McTwist did you find a solution w/Caddy?

I checked with both Apache and Caddy and somehow it still doesn't work. The errors for either of them seem to be different too. One of the developers managed to get Apache running on his end, but I failed to replicate their config, so I suspect that there is something else you and I are missing. I have put this on hold for now, but will eventually come back to it later in the future when I can spend my full attention on it.

 

[ctrlbrk]

I checked with both Apache and Caddy and somehow it still doesn't work. The errors for either of them seem to be different too. One of the developers managed to get Apache running on his end, but I failed to replicate their config, so I suspect that there is something else you and I are missing. I have put this on hold for now, but will eventually come back to it later in the future when I can spend my full attention on it.

I added these two headers and now have a different error message: sync group vm/101 failed - invalid protocol name

reverse_proxy https://192.168.2.58:8007 {
transport http {
tls_insecure_skip_verify
}
header_up Connection {http.request.header.Connection}
header_up Upgrade {http.request.header.Upgrade}
}

 

[ctrlbrk]

@McTwist I hope you can try my last post settings and see if you get it working. If so lets compare configs. I have two PBS on different continents and both behind Caddy. one works as a remote for the other. In one direction it works fine, and in the other direction I get these errors (now invalid protocol name). Yet both of my Caddy's are the same best I can tell!!

 

[McTwist]

I have decided to make use of nginx instead, because it "just works" and I currently do not want to spend more time figuring out how to get this to work, especially when the developers themselves cannot replicate it. If I dive more into this in the future, I will probably check other alternatives too, like haproxy and traefik.

 

[Max Carrara]

I have decided to make use of nginx instead, because it "just works" and I currently do not want to spend more time figuring out how to get this to work, especially when the developers themselves cannot replicate it. If I dive more into this in the future, I will probably check other alternatives too, like haproxy and traefik.

There's been another bug report, so I'm in the process of trying to replicate it again.

It seems that for some reverse proxies, the response to the upgrade request "doesn't make it" - it's as if it's never been received in the first place, even though we definitely send the correct response. Thus, the reverse proxy sends a response with an HTTP 200 instead of forwarding the server's response with the 101, cancelling the protocol upgrade process.`

I'm not sure why this doesn't happen for nginx. Caddy at least is affected by this. So, I'm about to use yet another proxy to see if I can sniff the packets being sent and received and do some analysis with Wireshark. The Caddy source code doesn't seem to suggest that anything's broken either, and I cannot find any related issues or bug reports. It honestly eludes me why specifically the 101 response gets trashed.

If nginx is the only thing that's confirmed to work at the moment, I suggest just sticking with it.

 

[McTwist]

There's been another bug report, so I'm in the process of trying to replicate it again.

It seems that for some reverse proxies, the response to the upgrade request "doesn't make it" - it's as if it's never been received in the first place, even though we definitely send the correct response. Thus, the reverse proxy sends a response with an HTTP 200 instead of forwarding the server's response with the 101, cancelling the protocol upgrade process.`

I'm not sure why this doesn't happen for nginx. Caddy at least is affected by this. So, I'm about to use yet another proxy to see if I can sniff the packets being sent and received and do some analysis with Wireshark. The Caddy source code doesn't seem to suggest that anything's broken either, and I cannot find any related issues or bug reports. It honestly eludes me why specifically the 101 response gets trashed.

If nginx is the only thing that's confirmed to work at the moment, I suggest just sticking with it.

Thank you for taking your time to look into this issue further. It is indeed peculiar that 101 is not being sent, so one could assume there are two reasons why this is the case:

1. While PBS does send the correct response, somehow there is something if fails to do that all the other services require.
2. Nginx is implementing upgrade correctly, and the other reverse proxies somehow fail it. However, I too see no issues in Caddy source for it to fail, unless it modifies the status code. Maybe running Caddy in debug mode may retrieve additional information.

From your perspective, we can assume that the first one is incorrect, therefore there may be something on the other services, but it sounds impossible that there is only one service that have implemented this correctly.

I will stick to nginx for now, but want to switch to something else in the future if possible. I was not in a position to be able to do this at all, but have spent some time to prepare the move to nginx very soon. I later can switch to any reverse proxy if desired.

 

[ctrlbrk]

I don't want to return to nginx. I worked another workaround solely for this issue but it's unique to my deployment.

I really want to see this fixed in PBS.

 

[Max Carrara]

but it sounds impossible that there is only one service that have implemented this correctly.

Exactly, this is what baffles me - there must be some other underlying issue (regarding PBS).

Maybe running Caddy in debug mode may retrieve additional information.

The only thing I discovered is that Caddy complains that it got an empty string for the Upgrade header from PBS. This initially led me to think that we don't send the header at all, but we do. Without a reverse proxy the response has all the expected headers, including Upgrade. That's why I will analyze the TCP flows between Caddy and PBS.

I'll keep you posted!

 

[Max Carrara]

I can't believe it, but I got it working.

This was probably one of the most vexing debugging odysseys I have ever ventured on, but nevertheless, I learnt a lot.


Something I want to elaborate on:

The only thing I discovered is that Caddy complains that it got an empty string for the Upgrade header from PBS.

Caddy does actually receive the Upgrade header (it shows up in the debug logs), but it won't pass the header along internally if the Connection header isn't included in the server's response. This is, to my own surprise, not explicitly specified in any of the HTTP/1.1 and HTTP/2 RFCs. Just in the examples.

The message of the patch I just sent to the mailing list describes everything in detail. If you're curious, you may read it over here: https://lists.proxmox.com/pipermail/pbs-devel/2024-February/007947.html

With this patch, Caddy is most definitely guaranteed to work with PBS (and nginx continues to work). I have yet to test other reverse proxies, but unless they reject the Connection header in the response (for god knows what reason) they should also work.

Thank you all for your patience!

 

[McTwist]

As they work with ordinary websockets, like PVE/PBS console, it most definitely will work. I've already moved to Nginx, but this gives me more reason to move to Caddy for the proxy. Thank you for your excellent work, and lets hope that it does not continue to be a problem for all other proxies.

 

[Max Carrara]

Version 2 of the patch recently got merged - the fix will most likely be available in package proxmox-backup-server version 3.1.5-1 or later.

The commit message contains a bunch of extra info in case you're curious.

Happy proxying!

 

[McTwist]

This is great news! That something so simple, although small fix, could result in such a hunt toward correct solution that it took almost 9 months before it was actually resolved properly. I am very impressed by your work and the help everyone have provided.