Disable default http redirects on 8007

[basisnederland]

Hello Forum members.

I am running a PBS Server, and I see that it's auto-redirecting HTTP requests on 8007 to HTTPS on the same port 8007.
I'd like to disable that and have port 8007 HTTPS only, not accepting anything else than HTTPS requests.
@ least disable http listening on 8007


I hope you folks can help me.

PS:
Is this using Nginx or Apache? Can I edit the config files for HTTP/HTTPS?

Thanks.

 

[Hannes Laimer]

Hey,

this was introduced with [1], if you think having this configurable makes sense feel free to open a feature request on [2], if you do also add your use-case for such an option to give some context.

[1] https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=532e7d9522ac1ecc7ba8a3ad89bb36d43da10d77
[2] https://bugzilla.proxmox.com

 

[basisnederland]

The context:
When opening port 8007 and exposing it to the internet, then from the internet side, when you SSL connect to port 8007, you now encounter random errors. I expect that it returns an HTTP reply to an HTTPS request!


You get (random) error :
tls_get_more_records: packet length too long when connecting to port 8007 with openssl s-client connect serverip:8007

I am expecting this to be solved after the pbs is running https only and not messing with http replys!

 

[basisnederland]

ps thanks for the reply !!

 

[Max Carrara]

Hey there,
I've implemented this redirect a while ago. Allow me to explain a little how it works - maybe that will clear some things up:

1. The port 8007 listens to both HTTP and HTTPS requests.
   
   
2. When receiving any HTTP request, the server will respond with either 301 Moved Permanently or 308 Permanent Redirect(depends on some HTTP semantics).
   • The response contains a corresponding Location header. If you e.g. tried to connect to http://my-server.tld:8007, the header will be Location: [B]https[/B]://my-server.tld:8007.
   • This means that the server tells the client to try to connect again using HTTPS. That's it.
     
3. When receiving any HTTPS request, the server will respond as it would do normally.

To answer some of your questions:

Is this using Nginx or Apache?

Neither. We have our own server.

Can I edit the config files for HTTP/HTTPS?

No. You currently cannot change this behaviour, but you're welcome to open a feature request!

Alternatively, you can set up Nginx as reverse proxy. There's a pretty good guide for PVE which you can adapt to your needs.

I expect that it returns an HTTP reply to an HTTPS request!

No, that should not be the case. HTTPS is simply HTTP over TLS/SSL. HTTPS requires that the connection is secured via TLS/SSL - the connection itself is encrypted. Over this connection, the HTTP protocol is then used.

You get (random) error :
tls_get_more_records: packet length too long when connecting to port 8007 with openssl s-client connect serverip:8007

What does your network configuration look like? Have you just forwarded the port? Does openssl s_client -connect serverip:8007 work when you execute it from your local network?

The command should always work because it will make a request over HTTPS directly. There will be no redirect. To see the redirect happen more clearly, you can use curl:

• curl -i -k serverip:8007
  • This should show you a response with HTTP/1.1 301 Moved Permanently
    
• curl -i -k https://serverip:8007
  • This should show you a response with HTTP/1.1 200 OK
    
• curl -i -k -L serverip:8007
  • This should redirect you correctly and then also show you a response with HTTP/1.1 200 OK

I am expecting this to be solved after the pbs is running https only and not messing with http replys!

PBS does not mess with HTTP replies. As mentioned above, clients making a plain HTTP request will simply be redirected and instructed to use HTTPS.


I hope this could clear some things up. If not, please provide a more detailed description of your network configuration - maybe I can help.

 

[basisnederland]

Thanks again for the fast reply.

I understand you; the focus is not on the HTTP reply on the HTTPS port, which could not be the issue.

Locally, within the PBS network, out of 200 tries, it works 200 times consistently, like a rock, at https://privateip:8007. Port 8007 is open on the internet and firewalled to the PVE of the customer. The port is always open, and open port checks run fine. Data transfer works; however, only the SSL handshake randomly throws an error. When attempting it 100 times, the error occurs 10 to 20 times. The error appears after the command:

openssl s-client connect serverwanip:8007

Error: tls_get_more_records: packet length too long when connecting to port 8007 with openssl s-client connect serverip:8007.

When the handshake works, the backups run perfectly directly from PVE to the remote PBE. However, when the handshake fails, the backup task stops and fails.

P.S.: When I connect another server on port 8007 in the same server site network (a Synology DSM), the handshake always returns okay. This way, I ruled out that it could be related to NATting or firewall issues. The fact that after the handshake the data transfer is always working tells me the 8007 forwarding is oke too.

Have a nice day and thanks for co-operating

 

[basisnederland]

When you want i can open extra ip's so you can test the ssl connect
please mail me than your ip so i whitelist it.

 

[Max Carrara]

Thank you for the detailed explanation!

What you just described might be related to bug #5105 - a user reported a similar thing happening, but I wasn't able to successfully reproduce the issue. Your problem seems to be very similar - connections working most of the time, but not all of the time. But in your case it's only when you try to connect via WAN... very strange. The user that reported this bug got a different error message, however.

I have one last question: Is there anything that is otherwise "unusual" about your network configuration? E.g. did you configure a different MTU?

I will try to investigate further and let you know if I need anything else. Thank you again for the detailed response!

 

[basisnederland]

Good morning

All normal:

Server side,
MTU 1492 (pppoe uplink).
Client side(s),
MTU 1500 side(s) where I tested it in 2 locations (with browsers, pve's, and OpenSSL CLI).

PS What I ruled out to be correlated before starting this post:

Played with other nics/switches @PBS
Other WAN side port numbers (NAT redirect)
TCP offload settings on NIC PBS
1Gb NIC on 1Gb switch (it was always 10Gb without jumbo frames)
Disabled MSS clamping on firewall/modem/router.
Other https servers do not have the issue @all on the same setup. Only the pbs server.

 

[Max Carrara]

Good morning!

Thanks for the additional information. I will continue to look into this, but I cannot say what the underlying cause of this issue is yet, unfortunately, as I couldn't yet reproduce the bug.

I will keep you updated if I find anything, but I think this will take a while to reproduce. Thanks again for all the information you provided, it helps a lot!

 

[basisnederland]

Thanks a lot!

For now, I am out of options and use other backup solution.
However, I am sure I will try PBS again after updating it in the future.

Thanks for your time, maybe the solution will become clear in the future!
Have a nice day."

 

[basisnederland]

After all the latest updates available with pve and pbs an update:

Sill the same:
from pve to pbs on port 8007 over internet:
SSL connect attempt failed error:0A00010B:SSL routines::wrong version number) (500

sometimes it works, but the ssl handshake is unstable.

 

[ruben.urresti]

THIS REALLY NEEDS TO BE FIXED!!!

I am experiencing the same issue on a freshly installed PBS server. When I make a request over WAN to that server like this:

curl -v https://pbs1.example.nl:8007

, I get this error:

error:0A00010B routines::wrong version number

.

Exactly like bug #5105 mentioned, when trying a couple of times in a row, the problem goes away. Of course, my PVE server can't retry its scheduled backups to the PBS server multiple times in a row, so every night the backups are failing.

I tried with self-signed and a 'real' (bought) certificate. Both are failing...

 

[basisnederland]

Check, thats the same!

 

[ruben.urresti]

I also started another thread before I saw this one: https://forum.proxmox.com/threads/i...ines-wrong-version-number.148881/#post-673728

Here we also have a Spanish guy claiming he has the same issue.

@Max Carrara , are you working on this issue?

Here is the full output:

*   Trying xx.xx.xx.xx:8007...
* Connected to pbs1.example.nl (xx.xx.xx.xx) port 8007 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* (5454) (IN), , Unknown (72):
* error:0A00010B:SSL routines::wrong version number
* Closing connection 0
curl: (35) error:0A00010B:SSL routines::wrong version number

 

[ruben.urresti]

@basisnederland , I just tried disabling the UFW firewall I had installed, and it seems to resolve the problem (for now). I will check again tomorrow because the results have been really inconsistent until now.

So, I would suggest that you also check your firewalls (on the PBS and on the router).

If this really fixes the issue permanently, then my question would be: why is UFW causing this strange behavior? My rules are nothing special, only this:

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW       192.168.0.0/16     
Anywhere                   ALLOW       <MY_REMOTE_NETWORK>/30

 

[Max Carrara]

Hello! Sorry for the late reply, I was on holiday.

@basisnederland , I just tried disabling the UFW firewall I had installed, and it seems to resolve the problem (for now). I will check again tomorrow because the results have been really inconsistent until now.

So, I would suggest that you also check your firewalls (on the PBS and on the router).

If this really fixes the issue permanently, then my question would be: why is UFW causing this strange behavior? My rules are nothing special, only this:

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW       192.168.0.0/16    
Anywhere                   ALLOW       <MY_REMOTE_NETWORK>/30

That's a very interesting find, I will see if I can reproduce this. Thanks for reporting this! I'll let you know once I've found out more.

 

[basisnederland]

Is the UFW firewall's default setting on?
i just installed an pbs server, and use hardware firewall so i did not take time to investigate any local firewall because the 2nd 3rd or 4th time it works... so than i think a firewall cant be an issue.

?

 

[Max Carrara]

Is the UFW firewall's default setting on?

UFW is not installed by default and also isn't activated if you install it. You have to do that manually.

i just installed an pbs server, and use hardware firewall so i did not take time to investigate any local firewall because the 2nd 3rd or 4th time it works... so than i think a firewall cant be an issue.

Well, we don't know that yet

@ruben.urresti From what I've read in the other thread, it seems to work now in most cases? And it's simply because you turned off UFW? Have you changed any other configuration in the meantime?

I wasn't able to reproduce the issue with UFW yet, unfortunately ... I will keep my eye on it, though. I will continue to investigate.

 

[Max Carrara]

@basisnederland @ruben.urresti Could you perhaps run a tcpdump for me, please? Specifically, try to replicate the scenario where your client fails to connect (e.g. @ruben.urresti enable UFW again, if possible). Then, run the following:

On the server: tcpdump -i <interface> 'port 8007' -w server.pcap

On the client: tcpdump -i <interface> 'port 8007' -w client.pcap

These commands will capture all packets that are sent to / from port 8007 via the interface you provide.

<interface> here is obviously just a placeholder - if your PBS listens on e.g. eth0, run tcpdump -i eth0 'port 8007' -w server.pcap on the server.

The same goes for the client - if your client uses e.g. enp5s0 to connect to your PBS, run tcpdump -i enp5s0 'port 8007' -w client.pcap on the client.

While both tcpdumps are running, try to connect to the server with the client of your choice. What's important is that a connection failure is captured, so it's okay if you try a couple times to replicate the bug. Once captured, I'll be able to see what's really going on on both the server's and the client's side.

You can upload the pcap files here, or at #5105 if you prefer.

Thanks a lot for your help!