[SOLVED] Let's Encrypt renew failed

[Fchen]

Hi,

I have some problems with my Proxmox 5.0 installation, I set up a Let's Encrypt certificate around 4 months ago, the first auto renew 2 months ago works fine but this time it doesn't work.

If I try to execute the cronjob command, I will receive the following:

root@rh01 ~/.acme.sh # ./acme.sh --cron --home "/root/.acme.sh"
[Thu Oct 26 18:15:57 CEST 2017] ===Starting cron===
[Thu Oct 26 18:15:57 CEST 2017] Renew: 'my.domain.com'
[Thu Oct 26 18:15:57 CEST 2017] Skip, Next renewal time is: Fri Dec 22 07:08:49 UTC 2017
[Thu Oct 26 18:15:57 CEST 2017] Add '--force' to force to renew.
[Thu Oct 26 18:15:57 CEST 2017] Skipped my.domain.com
[Thu Oct 26 18:15:57 CEST 2017] ===End cron===

If I will force it:

root@rh01 ~/.acme.sh # ./acme.sh --cron --home "/root/.acme.sh" --force
[Thu Oct 26 18:18:24 CEST 2017] ===Starting cron===
[Thu Oct 26 18:18:24 CEST 2017] Renew: 'my.domain.com'
[Thu Oct 26 18:18:24 CEST 2017] Standalone mode.
[Thu Oct 26 18:18:24 CEST 2017] Signing from existing CSR.
[Thu Oct 26 18:18:24 CEST 2017] Getting domain auth token for each domain
[Thu Oct 26 18:18:24 CEST 2017] Getting webroot for domain='my.domain.com'
[Thu Oct 26 18:18:24 CEST 2017] Getting new-authz for domain='my.domain.com'
[Thu Oct 26 18:18:25 CEST 2017] The new-authz request is ok.
[Thu Oct 26 18:18:25 CEST 2017] my.domain.com is already verified, skip http-01.
[Thu Oct 26 18:18:25 CEST 2017] Verify finished, start to sign.
[Thu Oct 26 18:18:26 CEST 2017] Cert success.
-----BEGIN CERTIFICATE-----
MIIFBTCCA+2gAwIBAgISA7ZPuXFcJ6PPzjUA3kceXJtqMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
...
VN7LG/1DCIX0LFshIiuNv3Ltw6nASMb2POTkedWLtO3YpH7oR3mVgMNje5Ptbfk3
MYz7jH73EDWPFlTnfcTTPhvyMHP5v1OR4ttmal/M16hMXJ6uojQm5aHveiB9EPCe
o8iuA5vPNBBMtfIUKjvqMNTFI4oW90eLoBXqvtXnspvPqRXicV5POrU=
-----END CERTIFICATE-----
[Thu Oct 26 18:18:26 CEST 2017] Your cert is in /root/.acme.sh/my.domain.com/my.domain.com.cer
[Thu Oct 26 18:18:26 CEST 2017] The intermediate CA cert is in /root/.acme.sh/my.domain.com/ca.cer
[Thu Oct 26 18:18:26 CEST 2017] And the full chain certs is there: /root/.acme.sh/my.domain.com/fullchain.cer
[Thu Oct 26 18:18:26 CEST 2017] Installing key to:/etc/pve/local/pveproxy-ssl.key
cat: /root/.acme.sh/my.domain.com/my.domain.com.key: No such file or directory
[Thu Oct 26 18:18:26 CEST 2017] Installing full chain to:/etc/pve/local/pveproxy-ssl.pem
[Thu Oct 26 18:18:26 CEST 2017] Run reload cmd: systemctl restart pveproxy
[Thu Oct 26 18:18:28 CEST 2017] Reload success

Now I will check the directory /etc/pve/local/:

root@rh01 ~/.acme.sh # la /etc/pve/local/
total 2.0K
drwxr-xr-x 2 root www-data 0 Jul 17 22:00 .
drwxr-xr-x 2 root www-data 0 Jul 17 22:00 ..
-rw-r----- 1 root www-data 83 Oct 26 18:24 lrm_status
drwxr-xr-x 2 root www-data 0 Jul 17 22:00 lxc
drwxr-xr-x 2 root www-data 0 Jul 17 22:00 openvz
drwx------ 2 root www-data 0 Jul 17 22:00 priv
-rw-r----- 1 root www-data 0 Oct 26 18:18 pveproxy-ssl.key
-rw-r----- 1 root www-data 3.4K Oct 26 18:18 pveproxy-ssl.pem

But proxmox webinterface isn't running, systemctl prints this:

root@rh01 ~/.acme.sh # systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2017-10-26 18:18:28 CEST; 8min ago
Process: 29765 ExecStop=/usr/bin/pveproxy stop (code=exited, status=0/SUCCESS)
Process: 29780 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 29789 (pveproxy)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/pveproxy.service
├─ 6543 pveproxy worker
├─ 6544 pveproxy worker
├─ 6545 pveproxy worker
└─29789 pveproxy

Oct 26 18:26:45 rh01 pveproxy[6446]: worker exit
Oct 26 18:26:45 rh01 pveproxy[29789]: worker 6445 finished
Oct 26 18:26:45 rh01 pveproxy[29789]: starting 1 worker(s)
Oct 26 18:26:45 rh01 pveproxy[6543]: /etc/pve/local/pveproxy-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1626.
Oct 26 18:26:45 rh01 pveproxy[29789]: worker 6544 started
Oct 26 18:26:45 rh01 pveproxy[6544]: /etc/pve/local/pveproxy-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1626.
Oct 26 18:26:45 rh01 pveproxy[29789]: worker 6446 finished
Oct 26 18:26:45 rh01 pveproxy[29789]: starting 1 worker(s)
Oct 26 18:26:45 rh01 pveproxy[29789]: worker 6545 started
Oct 26 18:26:45 rh01 pveproxy[6545]: /etc/pve/local/pveproxy-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1626.

I also tried to issue the certificate manual and everything works fine, then I copied the key and the fullcert, restarted pveproxy and some error happend again.
After too much of troubleshooting, my domain got blocked by Let's Encrypt.

Anything else I can try?

 

[fabian]

[Thu Oct 26 18:18:26 CEST 2017] Installing key to:/etc/pve/local/pveproxy-ssl.key
cat: /root/.acme.sh/my.domain.com/my.domain.com.key: No such file or directory

-rw-r----- 1 root www-data 0 Oct 26 18:18 pveproxy-ssl.key

you copied an empty key.. you should probably find out why the key file does not exist.

 

[Fchen]

I doesn't changed anything, but you'r right, there is no key file, how do I get it?
I take a look in the backup folder, 2 months ago was a key file existing, why is it missing right now?

 

[fabian]

if you have backup, and it is the right key, then you can use the one from the backup. otherwise, you'll need to issue a completely new certificate with a new key.

 

[Fchen]

I have tried to copy the key file from the backup folder, but if I restart the pveproxy this will happen:

root@rh01 /etc/pve/local # systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2017-10-30 19:35:46 CET; 1s ago
Process: 5797 ExecStop=/usr/bin/pveproxy stop (code=exited, status=0/SUCCESS)
Process: 5888 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 5892 (pveproxy)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/pveproxy.service
├─5892 pveproxy
├─5894 pveproxy worker
├─5895 pveproxy worker
└─5897 pveproxy worker

Oct 30 19:35:45 rh01 systemd[1]: Starting PVE API Proxy Server...
Oct 30 19:35:46 rh01 pveproxy[5888]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Oct 30 19:35:46 rh01 pveproxy[5892]: starting server
Oct 30 19:35:46 rh01 pveproxy[5892]: starting 3 worker(s)
Oct 30 19:35:46 rh01 pveproxy[5892]: worker 5894 started
Oct 30 19:35:46 rh01 pveproxy[5892]: worker 5895 started
Oct 30 19:35:46 rh01 pveproxy[5892]: worker 5897 started
Oct 30 19:35:46 rh01 systemd[1]: Started PVE API Proxy Server.

Should be normal or? But I can't connect to https://my.domain.com:8006, I still receive "ERR_CONNECTION_CLOSED"
How do I issue a completely new certificate, without changing the (sub-)domain name?

 

[Fchen]

I fixed my problem, before I tried to copy the cert key from the domains backup folder over /etc/pve/local/pveproxy-ssl.key, this action breaks the pveproxy after restarting.
Now I tried to copy the backup key to /root/.acme.sh/my.domain.com/my.domain.com.key and reissued the cert using --force, now acme.sh copied the key from the requested location and everything works fine.
But the strangest thing is, both keys are identical, a simple vimdiff doesn't show any difference.