Error ssl after upgrade 3 to 4

G

gizmo15

Renowned Member
Sep 20, 2015
18
0
66
Hi,

I just upgrade from 3 to 4 and now i have this ssl error when i try to access the webui :
Code: 
root@pbs:~# curl -v -k https://127.0.0.1:8007
*   Trying 127.0.0.1:8007...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: O=Proxmox Backup Server; OU=ED83684B-7866-4CBD-B45A-9507FC09D884; CN=pbs.adm.securmail.fr
*  start date: Jan 10 11:41:20 2025 GMT
*  expire date: May 13 11:41:20 3024 GMT
*  issuer: O=Proxmox Backup Server; OU=ED83684B-7866-4CBD-B45A-9507FC09D884; CN=pbs.adm.securmail.fr
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to 127.0.0.1 (127.0.0.1) port 8007
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 127.0.0.1:8007
> User-Agent: curl/8.14.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Recv failure: Connection reset by peer
* OpenSSL SSL_read: Connection reset by peer, errno 104
* closing connection #0
curl: (56) Recv failure: Connection reset by peer

If i try with more verbose :
Code: 
23:43:48.823601 [0-0] == Info: [SSL] ossl_bio_cf_in_read(len=5) -> 0, err=0
23:43:48.823915 [0-0] => Send SSL data, 5 bytes (0x5)
0000: .....
23:43:48.824154 [0-0] => Send SSL data, 1 bytes (0x1)
0000: .
23:43:48.824496 [0-0] == Info: Send failure: Broken pipe
23:43:48.824742 [0-0] == Info: [SSL] ossl_bio_cf_out_write(len=24) -> -1, err=55
23:43:48.825053 [0-0] == Info: OpenSSL SSL_read: OpenSSL/3.5.1: error:0A000126:SSL routines::unexpected eof while reading, errno 32
23:43:48.825580 [0-0] == Info: [SSL] cf_recv(len=102400) -> -1, 56
23:43:48.825841 [0-0] == Info: [WRITE] [OUT] done
23:43:48.826024 [0-0] == Info: closing connection #0
curl: (56) Send failure: Broken pipe

My installation informations :
Code: 
root@pbs:~# proxmox-backup-manager version --verbose
proxmox-backup                      4.0.0        running kernel: 6.14.8-2-pve
proxmox-backup-server               4.0.11-2     running version: 4.0.11     
proxmox-kernel-helper               9.0.3                                    
proxmox-kernel-6.14.8-2-pve-signed  6.14.8-2                                 
proxmox-kernel-6.14                 6.14.8-2                                 
proxmox-kernel-6.8.12-13-pve-signed 6.8.12-13                                
proxmox-kernel-6.8                  6.8.12-13                                
proxmox-kernel-6.8.12-11-pve-signed 6.8.12-11                                
proxmox-kernel-6.8.12-4-pve-signed  6.8.12-4                                 
ifupdown2                           3.3.0-1+pmx9                             
libjs-extjs                         7.0.0-5                                  
proxmox-backup-docs                 4.0.11-2                                 
proxmox-backup-client               4.0.11-1                                 
proxmox-mail-forward                1.0.2                                    
proxmox-mini-journalreader          1.6                                      
proxmox-offline-mirror-helper       0.7.0                                    
proxmox-widget-toolkit              5.0.5                                    
pve-xtermjs                         5.5.0-2                                  
smartmontools                       7.4-pve1                                 
zfsutils-linux                      2.3.3-pve1

If i restart those two services, it's ok for a couple of seconds and the error come back :
Code: 
systemctl restart proxmox-backup-proxy.service proxmox-backup.service

Nothing in dmesg or with journalctl for those services.

Can you help me on that ?

Thanks!
 
do you have any kind of monitoring/reverse proxy/.. or other custom network setup in place?
 
No monitoring or reverse proxy.
My computer is in the same LAN.

The network configuration :
Code: 
root@pbs:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 22:36:4d:07:98:c6 brd ff:ff:ff:ff:ff:ff
    altname enx22364d0798c6
    inet 192.168.0.41/24 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet6 2a0e:f41:0:3:2036:4dff:fe07:98c6/64 scope global dynamic mngtmpaddr proto kernel_ra
       valid_lft 86090sec preferred_lft 14090sec
    inet6 2a0e:f41:0:3::1e/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::2036:4dff:fe07:98c6/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

I did the same test : restart proxmox-backup-proxy.service proxmox-backup.service and i can access the web ui for some seconds and then, i get the reset connection
 
From my access.log and auth.log I can confirm: I had access entries here right after start, but the GUI never made it to load before it dies.

In the non-working state not a single line in access.log - seems to break even before being logged.
 
thanks! it seems to be a bug related to the connection handling, we are currently trying to find a reproducer!
 
Just to add I am having the same issue. Are there are any logs or anything that would be useful?

My PVE server with PBS volume mounted gets an error 500 and the backup fails, not sure if that is happening for the other posters
 
Last edited:
i have an error 500 as well :
Code: 
TASK ERROR: could not activate storage 'PBS-home': PBS-home: error fetching datastores - 500 interrupted by signal
 
Is your PBS bare-metal or virtualized? Would it be possible to create a tcpdump of the traffic while you are reproducing the problem?

Code: 
tcpdump -w output.pcap port 8007
 
gizmo15 said:
No monitoring or reverse proxy.
My computer is in the same LAN.

The network configuration :
Code: 
root@pbs:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 22:36:4d:07:98:c6 brd ff:ff:ff:ff:ff:ff
    altname enx22364d0798c6
    inet 192.168.0.41/24 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet6 2a0e:f41:0:3:2036:4dff:fe07:98c6/64 scope global dynamic mngtmpaddr proto kernel_ra
       valid_lft 86090sec preferred_lft 14090sec
    inet6 2a0e:f41:0:3::1e/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::2036:4dff:fe07:98c6/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

I did the same test : restart proxmox-backup-proxy.service proxmox-backup.service and i can access the web ui for some seconds and then, i get the reset connection
Click to expand...
In addition to the tcpdump, could you try to generate an strace when the connection reset error is happening when connecting via curl? This might help narrow down the issue.

Code: 
strace -ftt  -p $(pidof proxmox-backup-proxy)

But be VERY careful in what you post and/or provide, as this can contain sensitive information, e.g. username and password/token in writev syscalls! If you can share this information via some private channel would be great. You should see something along the line of (prefiltered by the corresponding process id) when the TCP connection is established
Code: 
[pid  9266] 11:20:14.551427 setsockopt(18, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid  9266] 11:20:14.551484 setsockopt(18, SOL_TCP, TCP_KEEPIDLE, [90], 4) = 0
[pid  9266] 11:20:14.551540 connect(18, {sa_family=AF_INET, sin_port=htons(82), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid  9266] 11:20:14.551654 epoll_ctl(5, EPOLL_CTL_ADD, 18, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data=0x75c7cc067500}) = 0
[pid  9266] 11:20:14.551713 epoll_wait(3, [{events=EPOLLOUT, data=0x75c7cc142f80}, {events=EPOLLOUT, data=0x75c7cc067500}], 1024, 739) = 2
[pid  9266] 11:20:14.551782 getsockopt(18, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
[pid  9266] 11:20:14.551840 setsockopt(18, SOL_TCP, TCP_NODELAY, [0], 4) = 0
[pid  9266] 11:20:14.551894 getpeername(18, {sa_family=AF_INET, sin_port=htons(82), sin_addr=inet_addr("127.0.0.1")}, [128 => 16]) = 0
[pid  9266] 11:20:14.551954 getsockname(18, {sa_family=AF_INET, sin_port=htons(54226), sin_addr=inet_addr("127.0.0.1")}, [128 => 16]) = 0
 
gizmo15 said:
No monitoring or reverse proxy.
Click to expand...
Also no health check e.g. tools which would only establish a tcp connection to the port and drop it again if the connection could be established?
 
@shanreich bare-metal
i made the pcap, do you have a prefered method to give it to you ?

@Chris same, strace done, do you have a prefered method to give it to you ?

no health check or anything
 
gizmo15 said:
@Chris same, strace done, do you have a prefered method to give it to you ?
Click to expand...
You could send me a direct message with a secure link to private file hosting or the like, or even attach it to the private message directly if the size allows.
 
It looks like disabling ipv6 on the PBS server has fixed the issue for me, I am just running a backup and will report back, but 5 mins in so far and it appears to be working
 
this issue should be fixed with PBS 4.0.12-1, now available on pbs-test!
 
  • Like
Reactions: ampfinger
You must log in or register to reply here.
Top Bottom