Connection closed before reading preface

[unam]

Hello,

I daily use and backup 5 pve servers on my pbs instance.

I added another host (HOST-A) yesterday and had an error (this server never made any backup on this pbs) with the following log message :

2024-09-18T10:16:49+02:00: starting new backup on datastore 'pbs-prod04' from ::ffff:10.10.10.1: "vm/100/2024-09-18T08:16:36Z"
2024-09-18T10:16:49+02:00: backup failed: connection error: connection closed before reading preface
2024-09-18T10:16:49+02:00: removing failed backup
2024-09-18T10:16:49+02:00: TASK ERROR: connection error: connection closed before reading preface

But I can list the datastore and see existing backups.

From another host (HOST-B with same datastore, credentials etc), everything is ok and I have the following log when trying to backup a container :

2024-09-18T09:53:54+02:00: starting new backup on datastore 'pbs-prod' from ::ffff:10.10.10.1: "ct/999/2024-09-18T07:53:43Z"
2024-09-18T09:53:54+02:00: protocol upgrade done
2024-09-18T09:53:54+02:00: GET /previous_backup_time
2024-09-18T09:53:54+02:00: POST /blob

On the faulty server protocol upgrade done is missing, so it looks like a ssl/tls session not initiated on the server side.

In front of the pbs there is a haproxy server (10.10.10.1) with classic backend configuration :

backend pbs-prod
        mode http
        option forwardfor
        option httpchk GET /
        cookie SERVER insert indirect nocache
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        server pbs-prod 10.10.10.10:8007 check ssl verify none

When I bypass the haproxy server and connect directly to the pbs, everything is ok from HOST-A and HOST-B

Facts : haproxy offload "public" ssl/tls and create another ssl/tls session to the pbs backends.
Question is : Why pbs does not "enforce" a ssl/tls upgrade from HOST-A but makes it for HOST-B.

 

[Chris]

Hello,

I daily use and backup 5 pve servers on my pbs instance.

I added another host (HOST-A) yesterday and had an error (this server never made any backup on this pbs) with the following log message :

2024-09-18T10:16:49+02:00: starting new backup on datastore 'pbs-prod04' from ::ffff:10.10.10.1: "vm/100/2024-09-18T08:16:36Z"
2024-09-18T10:16:49+02:00: backup failed: connection error: connection closed before reading preface
2024-09-18T10:16:49+02:00: removing failed backup
2024-09-18T10:16:49+02:00: TASK ERROR: connection error: connection closed before reading preface

But I can list the datastore and see existing backups.

From another host (HOST-B with same datastore, credentials etc), everything is ok and I have the following log when trying to backup a container :

2024-09-18T09:53:54+02:00: starting new backup on datastore 'pbs-prod' from ::ffff:10.10.10.1: "ct/999/2024-09-18T07:53:43Z"
2024-09-18T09:53:54+02:00: protocol upgrade done
2024-09-18T09:53:54+02:00: GET /previous_backup_time
2024-09-18T09:53:54+02:00: POST /blob

On the faulty server protocol upgrade done is missing, so it looks like a ssl/tls session not initiated on the server side.

In front of the pbs there is a haproxy server (10.10.10.1) with classic backend configuration :

backend pbs-prod
        mode http
        option forwardfor
        option httpchk GET /
        cookie SERVER insert indirect nocache
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        server pbs-prod 10.10.10.10:8007 check ssl verify none

When I bypass the haproxy server and connect directly to the pbs, everything is ok from HOST-A and HOST-B

Facts : haproxy offload "public" ssl/tls and create another ssl/tls session to the pbs backends.
Question is : Why pbs does not "enforce" a ssl/tls upgrade from HOST-A but makes it for HOST-B.

Hi,
this sounds more like an issue with the HAProxy configuration, it seems like the HAProxy terminates the connection to the PBS instance when upgrading to http2 for the backup protocol. Do you get anything in the system logs on the HAProxy side?

Regarding your setup: Do you need the TLS session to be terminated on the HAProxy? You could set up ssl pass-trough based on SNI matching? That would probably be better because the data remains encrypted from client to PBS instance, reducing load and probably increase throughput.

 

[unam]

Hi Chris, thanks for your reply.

Indeed, it looks like an issue on haproxy but I don't have any trace in logs. The issue seems to be really linked to haproxy and this specific client because everything is ok with another client coming from another source ip.
I'm currently trying to enable full debug log but I still don't see anything new between info and debug haproxy logs.

I do not forward encrypted trafic directly to backend because I have multiple backends and it's easier for me to handle my letsencrypt certificates on the haproxy host.