Updated certificate, now WebUI won't work

Y

yaleman

New Member
Mar 8, 2020
4
1
3
42
Installed a new certificate via the web UI, it said it'd restart to get working and it just went offline.

I've tried the documented steps to generate a self-signed CA/certificate and no change. I've rebooted and can't find any errors in logs. I'm stumped and all my VMs are offline, can someone suggest any ideas? I can't use letsencrypt locally because the server is not accessible from the internet.

curl gives zero-length replies from local or remote:

Bash: 
# curl -vvv http://localhost:8006
* Rebuilt URL to: http://localhost:8006/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8006 (#0)
> GET / HTTP/1.1
> Host: localhost:8006
> User-Agent: curl/7.52.1
> Accept: */*
>
* Curl_http_done: called premature == 0
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
# curl -vvvk https://localhost:8006
* Rebuilt URL to: https://localhost:8006/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to localhost:8006
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to localhost:8006
#

Code: 
# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
 
i would first upgrade to a recent version (at least 5.4)
and then do a 'pvecm updatecerts' (maybe needs --force)
 
  • Like
Reactions: stubbe
Are there specific bugs that updating would fix?

I've tried pvecm updatecerts (with --force) and without and it doesn't seem to fix it. Are there additional steps to follow after that one command?

Bash: 
root@pve1# pvecm updatecerts --force
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
root@pve1:/etc/pve/priv# systemctl restart pveproxy
root@pve1:/etc/pve/priv# systemctl restart pvedaemon
    root@pve1:/etc/pve/priv# systemctl status pvedaemon
● pvedaemon.service - PVE API Daemon
   Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-10 06:35:28 AEST; 15s ago
  Process: 29326 ExecStop=/usr/bin/pvedaemon stop (code=exited, status=0/SUCCESS)
  Process: 29330 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
Main PID: 29352 (pvedaemon)
    Tasks: 4 (limit: 4915)
   Memory: 114.7M
      CPU: 611ms
   CGroup: /system.slice/pvedaemon.service
           ├─29352 pvedaemon
           ├─29355 pvedaemon worker
           ├─29356 pvedaemon worker
           └─29357 pvedaemon worker

Mar 10 06:35:27 pve1 systemd[1]: Starting PVE API Daemon...
Mar 10 06:35:28 pve1 pvedaemon[29352]: starting server
Mar 10 06:35:28 pve1 pvedaemon[29352]: starting 3 worker(s)
Mar 10 06:35:28 pve1 pvedaemon[29352]: worker 29355 started
Mar 10 06:35:28 pve1 pvedaemon[29352]: worker 29356 started
Mar 10 06:35:28 pve1 pvedaemon[29352]: worker 29357 started
Mar 10 06:35:28 pve1 systemd[1]: Started PVE API Daemon.
root@pve1:/etc/pve/priv# ss -ltnp
State          Recv-Q          Send-Q                    Local Address:Port                   Peer Address:Port
LISTEN         0               5                             10.0.0.10:3551                        0.0.0.0:*             users:(("apcupsd",pid=1251,fd=5))
LISTEN         0               128                             0.0.0.0:8006                        0.0.0.0:*             users:(("pveproxy worker",pid=29310,fd=6),("pveproxy worker",pid=29309,fd=6),("pveproxy worker",pid=29308,fd=6),("pveproxy",pid=29307,fd=6))
LISTEN         0               128                             0.0.0.0:111                         0.0.0.0:*             users:(("rpcbind",pid=999,fd=8))
LISTEN         0               128                           127.0.0.1:85                          0.0.0.0:*             users:(("pvedaemon worke",pid=29357,fd=6),("pvedaemon worke",pid=29356,fd=6),("pvedaemon worke",pid=29355,fd=6),("pvedaemon",pid=29352,fd=6))
LISTEN         0               128                             0.0.0.0:22                          0.0.0.0:*             users:(("sshd",pid=1646,fd=3))
LISTEN         0               128                             0.0.0.0:3128                        0.0.0.0:*             users:(("spiceproxy work",pid=20235,fd=6),("spiceproxy",pid=2143,fd=6))
LISTEN         0               100                           127.0.0.1:25                          0.0.0.0:*             users:(("master",pid=1864,fd=13))
LISTEN         0               128                                [::]:111                            [::]:*             users:(("rpcbind",pid=999,fd=11))
LISTEN         0               128                                [::]:22                             [::]:*             users:(("sshd",pid=1646,fd=4))
LISTEN         0               100                               [::1]:25                             [::]:*             users:(("master",pid=1864,fd=14))
root@pve1:/etc/pve/priv# curl localhost:85
root@pve1:/etc/pve/priv# curl -vvv localhost:85
* Rebuilt URL to: localhost:85/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 85 (#0)
> GET / HTTP/1.1
> Host: localhost:85
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 501 no such file '/'
< Cache-Control: max-age=0
< Connection: close
< Date: Mon, 09 Mar 2020 20:36:07 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Expires: Mon, 09 Mar 2020 20:36:07 GMT
<
* Curl_http_done: called premature == 0
* Closing connection 0
root@pve1:/etc/pve/priv# curl -vvv localhost:8006
* Rebuilt URL to: localhost:8006/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8006 (#0)
> GET / HTTP/1.1
> Host: localhost:8006
> User-Agent: curl/7.52.1
> Accept: */*
>
* Curl_http_done: called premature == 0
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
# curl -vvvk https://localhost:8006
* Rebuilt URL to: https://localhost:8006/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to localhost:8006
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to localhost:8006
 
Last edited:
what does "pvenode cert info" say? what does "openssl x509 -noout -text /etc/pve/local/pveproxy-ssl.pem" say? what does "journalctl -b -u pveproxy" say?
 
Here's the output:

Bash: 
root@pve1:~# journalctl -b -u pveproxy
-- Logs begin at Sun 2020-03-08 21:43:27 AEST, end at Tue 2020-03-10 22:29:11 AEST. --
<snipped>
Mar 10 06:35:20 pve1 systemd[1]: Starting PVE API Proxy Server...
Mar 10 06:35:21 pve1 pveproxy[29275]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Mar 10 06:35:21 pve1 pveproxy[29307]: starting server
Mar 10 06:35:21 pve1 pveproxy[29307]: starting 3 worker(s)
Mar 10 06:35:21 pve1 pveproxy[29307]: worker 29308 started
Mar 10 06:35:21 pve1 pveproxy[29307]: worker 29309 started
Mar 10 06:35:21 pve1 pveproxy[29307]: worker 29310 started
Mar 10 06:35:21 pve1 systemd[1]: Started PVE API Proxy Server.
root@pve1:~# pvenode cert info
┌─────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ filename    │ pve-root-ca.pem                                                                                  │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ fingerprint │ E7:51:68:36:38:CE:DE:BD:F1:30:E7:66:19:CC:85:09:69:61:EC:39:BB:EF:D5:78:C0:30:42:E5:46:36:18:C8  │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ subject     │ /CN=Proxmox Virtual Environment/OU=3a78e4e5-553e-411a-9ac4-400d5fefc585/O=PVE Cluster Manager CA │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ issuer      │ /CN=Proxmox Virtual Environment/OU=3a78e4e5-553e-411a-9ac4-400d5fefc585/O=PVE Cluster Manager CA │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notbefore   │ 2020-03-09 06:35:12                                                                              │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notafter    │ 2030-03-07 06:35:12                                                                              │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ san         │ []                                                                                               │
└─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ filename    │ pve-ssl.pem                                                                                      │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ fingerprint │ E6:38:8F:0C:F0:6C:D0:55:03:CC:38:CF:57:83:F5:33:10:21:41:37:55:53:53:05:58:E1:A5:7C:85:03:8D:CB  │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ subject     │ /OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=pve1.housenet.yaleman.org                  │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ issuer      │ /CN=Proxmox Virtual Environment/OU=3a78e4e5-553e-411a-9ac4-400d5fefc585/O=PVE Cluster Manager CA │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notbefore   │ 2020-03-09 06:35:12                                                                              │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notafter    │ 2030-03-07 06:35:12                                                                              │
├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ san         │ - 127.0.0.1                                                                                      │
│             │ - 0000:0000:0000:0000:0000:0000:0000:0001                                                        │
│             │ - localhost                                                                                      │
│             │ - 10.0.0.146                                                                                     │
│             │ - pve1                                                                                           │
│             │ - pve1.housenet.yaleman.org                                                                      │
└─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────┐
│ filename    │ pveproxy-ssl.pem                                                                                │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ fingerprint │ 25:84:7D:66:8E:B4:F0:4F:DD:40:B1:2B:6B:07:40:C5:67:DA:7D:02:43:08:EB:6C:2C:96:FE:41:D9:DE:21:8D │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ subject     │ /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3                                             │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ issuer      │ /O=Digital Signature Trust Co./CN=DST Root CA X3                                                │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notbefore   │ 2016-03-18 02:40:46                                                                             │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notafter    │ 2021-03-18 02:40:46                                                                             │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ san         │ []                                                                                              │
└─────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘
root@pve1:~# openssl x509 -noout -text /etc/pve/local/pveproxy-ssl.pem
x509: Unknown parameter /etc/pve/local/pveproxy-ssl.pem
x509: Use -help for summary.
root@pve1:~# openssl x509 -noout -text -in /etc/pve/local/pveproxy-ssl.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Mar 17 16:40:46 2016 GMT
            Not After : Mar 17 16:40:46 2021 GMT
        Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    <snip>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            Authority Information Access:
                OCSP - URI:http://isrg.trustid.ocsp.identrust.com
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier:
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

            X509v3 Subject Key Identifier:
                A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
    Signature Algorithm: sha256WithRSAEncryption
         <snip>
 
Update, figured it out. Something to do with the cert file /etc/pve/local/pveproxy-ssl.pem (and /etc/pve/local/pveproxy-ssl.key) were borked. Generated new certs using letsencrypt on another box, copied the cert and key to those filenames and "systemctl restart pveproxy" and it came good.

Thanks for leading me to the answer in the end :)
 
  • Like
Reactions: fabian
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom