Connection closed before reading preface

[unam]

Hello,

I daily use and backup 5 pve servers on my pbs instance.

I added another host (HOST-A) yesterday and had an error (this server never made any backup on this pbs) with the following log message :

2024-09-18T10:16:49+02:00: starting new backup on datastore 'pbs-prod04' from ::ffff:10.10.10.1: "vm/100/2024-09-18T08:16:36Z"
2024-09-18T10:16:49+02:00: backup failed: connection error: connection closed before reading preface
2024-09-18T10:16:49+02:00: removing failed backup
2024-09-18T10:16:49+02:00: TASK ERROR: connection error: connection closed before reading preface

But I can list the datastore and see existing backups.

From another host (HOST-B with same datastore, credentials etc), everything is ok and I have the following log when trying to backup a container :

2024-09-18T09:53:54+02:00: starting new backup on datastore 'pbs-prod' from ::ffff:10.10.10.1: "ct/999/2024-09-18T07:53:43Z"
2024-09-18T09:53:54+02:00: protocol upgrade done
2024-09-18T09:53:54+02:00: GET /previous_backup_time
2024-09-18T09:53:54+02:00: POST /blob

On the faulty server protocol upgrade done is missing, so it looks like a ssl/tls session not initiated on the server side.

In front of the pbs there is a haproxy server (10.10.10.1) with classic backend configuration :

backend pbs-prod
        mode http
        option forwardfor
        option httpchk GET /
        cookie SERVER insert indirect nocache
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        server pbs-prod 10.10.10.10:8007 check ssl verify none

When I bypass the haproxy server and connect directly to the pbs, everything is ok from HOST-A and HOST-B

Facts : haproxy offload "public" ssl/tls and create another ssl/tls session to the pbs backends.
Question is : Why pbs does not "enforce" a ssl/tls upgrade from HOST-A but makes it for HOST-B.

 

[Chris]

Hello,

I daily use and backup 5 pve servers on my pbs instance.

I added another host (HOST-A) yesterday and had an error (this server never made any backup on this pbs) with the following log message :

2024-09-18T10:16:49+02:00: starting new backup on datastore 'pbs-prod04' from ::ffff:10.10.10.1: "vm/100/2024-09-18T08:16:36Z"
2024-09-18T10:16:49+02:00: backup failed: connection error: connection closed before reading preface
2024-09-18T10:16:49+02:00: removing failed backup
2024-09-18T10:16:49+02:00: TASK ERROR: connection error: connection closed before reading preface

But I can list the datastore and see existing backups.

From another host (HOST-B with same datastore, credentials etc), everything is ok and I have the following log when trying to backup a container :

2024-09-18T09:53:54+02:00: starting new backup on datastore 'pbs-prod' from ::ffff:10.10.10.1: "ct/999/2024-09-18T07:53:43Z"
2024-09-18T09:53:54+02:00: protocol upgrade done
2024-09-18T09:53:54+02:00: GET /previous_backup_time
2024-09-18T09:53:54+02:00: POST /blob

On the faulty server protocol upgrade done is missing, so it looks like a ssl/tls session not initiated on the server side.

In front of the pbs there is a haproxy server (10.10.10.1) with classic backend configuration :

backend pbs-prod
        mode http
        option forwardfor
        option httpchk GET /
        cookie SERVER insert indirect nocache
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        server pbs-prod 10.10.10.10:8007 check ssl verify none

When I bypass the haproxy server and connect directly to the pbs, everything is ok from HOST-A and HOST-B

Facts : haproxy offload "public" ssl/tls and create another ssl/tls session to the pbs backends.
Question is : Why pbs does not "enforce" a ssl/tls upgrade from HOST-A but makes it for HOST-B.

Hi,
this sounds more like an issue with the HAProxy configuration, it seems like the HAProxy terminates the connection to the PBS instance when upgrading to http2 for the backup protocol. Do you get anything in the system logs on the HAProxy side?

Regarding your setup: Do you need the TLS session to be terminated on the HAProxy? You could set up ssl pass-trough based on SNI matching? That would probably be better because the data remains encrypted from client to PBS instance, reducing load and probably increase throughput.

 

[unam]

Hi Chris, thanks for your reply.

Indeed, it looks like an issue on haproxy but I don't have any trace in logs. The issue seems to be really linked to haproxy and this specific client because everything is ok with another client coming from another source ip.
I'm currently trying to enable full debug log but I still don't see anything new between info and debug haproxy logs.

I do not forward encrypted trafic directly to backend because I have multiple backends and it's easier for me to handle my letsencrypt certificates on the haproxy host.

 

[Chris]

The issue seems to be really linked to haproxy and this specific client because everything is ok with another client coming from another source ip

What client versions are involved here, proxmox-backup-client version? And what is the PBS version proxmox-backup-manager version --verbose?

Can you verify that the working client is routed trough the HAProxy, e.g. via tcpdump on the HAProxy interface or the PBS?

 

[unam]

Hi Chris, thanks for you reply and sorry about my late answer.

Here is the pb manager version :

proxmox-backup                    3.2.0        running kernel: 6.8.8-3-pve
proxmox-backup-server             3.2.7-1      running version: 3.2.7    
proxmox-kernel-helper             8.1.0                                  
proxmox-kernel-6.8                6.8.12-2                              
proxmox-kernel-6.8.8-3-pve-signed 6.8.8-3                                
proxmox-kernel-6.8.4-2-pve-signed 6.8.4-2                                
ifupdown2                         3.2.0-1+pmx9                          
libjs-extjs                       7.0.0-4                                
proxmox-backup-docs               3.2.7-1                                
proxmox-backup-client             3.2.7-1                                
proxmox-mail-forward              0.2.3                                  
proxmox-mini-journalreader        1.4.0                                  
proxmox-offline-mirror-helper     0.6.7                                  
proxmox-widget-toolkit            4.2.3                                  
pve-xtermjs                       5.3.0-3                                
smartmontools                     7.3-pve1                              
zfsutils-linux                    2.2.6-pve1

And the pb client version is 3.0.4

Concerning the verification from client I believe that everything is routed correctly:

UPID:sh02-171:00000273:0000024F:000004EF:66F50D0E:backup:user1\x3avm-118:user1@pbs: 66F50D0E connection error: connection closed before reading preface
UPID:sh02-171:00000273:0000024F:000004F0:66F50D10:backup:user1\x3avm-120:user1@pbs: 66F50D10 connection error: connection closed before reading preface
UPID:sh02-171:00000273:0000024F:000004F1:66F50D11:backup:user1\x3avm-500:user1@pbs: 66F50D11 connection error: connection closed before reading preface
UPID:sh02-171:00000273:0000024F:000004F2:66F50D12:backup:user1\x3avm-501:user1@pbs: 66F50D12 connection error: connection closed before reading preface
UPID:sh02-171:00000273:0000024F:000004F3:66F50D14:backup:user1\x3avm-1000:user1@pbs: 66F50D14 connection error: connection closed before reading preface

Regards,

 

[Chris]

Concerning the verification from client I believe that everything is routed correctly:

Well, can you verify this, e.g. by filtering traffic on the interfaces via tcpdump. Have you tried to announce the http/2 support from the HAProxy to the backend server (PBS), according to the HAProxy docs you can set the alpn option, see https://www.haproxy.com/documentation/haproxy-configuration-tutorials/load-balancing/http/#http/2-over-https-to-the-server

Edit: To add more information, this is likely caused because your HAProxy does not send the connection preface as required for http/2, see https://httpwg.org/specs/rfc7540.html#ConnectionHeader. Given that, I would however not expect other clients to work. So either they connect to the PBS directly or are using a different route?

 

[unam]

Hi Chris and again, thanks for your help.

Using tcpdump the traffic comes to haproxy, go to the pbs backend.

Here is a log from /var/log/proxmox-backup/tasks/archive

/sh02-171:00000273:0000024F:0000065B:66F6C564:backup:user1\x3avm-120:user1@pbs: 66F6C564 connection error: connection closed before reading preface

And here is the line from the proxy logs :

2024-09-27T16:17:08.655005+02:00 front171 haproxy[11189]: <134>Sep 27 16:17:08 haproxy[11189]: 7.4.57.12:39776 [27/Sep/2024:16:17:08.653] pbs~ sh02-171/sh02-171 0/0/0/-1/1 502 342 - - PH-- 6/6/0/0/0 0/0 "GET //api2/json/backup?backup-id=109&backup-time=1727446628&backup-type=vm&benchmark=false&debug=false&store=user1 HTTP/1.1"

I enabled http2 for all my backends according to the documentation but nothing changed. My "web" http connections from my web browser are logged as HTTP2 but pve status checks are still made through HTTP1.1 according to haproxy logs.

To finish, I have :

- sh01 : 4/5 datastores with PVE backups ok
- sh02 : 3/4 datastores with PVE backups ok

On both side I have the same error for 1 datastore and no matter if I delete this datastore/user and recreate it on another instance (sh01 on sh02 for example), it still does not work.

I already deployed a new haproxy server, same error for the same user on his server. With this user I dont have any error from my server

 

[Chris]

I already deployed a new haproxy server, same error for the same user on his server.

Well, in that case there might be an issue in-between the users PVE host and your HAProxy? Is there an additional proxy inbetween? Does the HTTP2 connection between the users PVE host and the HAProxy work as expected?