ACME update fails: Server reply does not look like a PEM encoded certificate

D

dsi

Renowned Member
Dec 15, 2015
28
5
68
Germany
Hello,

I'm seeking support for a failure that appeared recently. The Web GUI certificate is provided via ACME from a FreeIPA server via DNS-plugin for quite some time without problems. Last successful renewal was September 30, 2025. Since November 30, 2025 the renewal fails:
Code: 
Dec 03 03:29:38 pve-1 pveupdate[3695637]: Loading ACME account details
Dec 03 03:29:38 pve-1 pveupdate[3695637]: Placing ACME order
Dec 03 03:29:39 pve-1 pveupdate[3695637]: Order URL: https://idm-1.int.domain.de/acme/order/8AV9HaJ5K8
Dec 03 03:29:39 pve-1 pveupdate[3695637]: Getting authorization details from 'https://idm-1.int.domain.de/acme/authz/AHAR3o66dp'
Dec 03 03:29:39 pve-1 pveupdate[3695637]: The validation for pve-1.domain is pending!
Dec 03 03:29:39 pve-1 pveupdate[3695637]: [Wed Dec  3 03:29:39 CET 2025] adding _acme-challenge.pve-1.int.domain.de. 60 in txt "Vv4z7zo6KXWVnZ-X4_nU3mIhb4FjAMRYK6zQfxuoqkU"
Dec 03 03:29:39 pve-1 pveupdate[3695637]: Add TXT record: _acme-challenge.pve-1.int.domain.de
Dec 03 03:29:39 pve-1 pveupdate[3695637]: Sleeping 30 seconds to wait for TXT record propagation
Dec 03 03:30:09 pve-1 pveupdate[3695637]: Triggering validation
Dec 03 03:30:09 pve-1 pveupdate[3695637]: Sleeping for 5 seconds
Dec 03 03:30:14 pve-1 pveupdate[3695637]: Status is 'valid', domain 'pve-1.int.domain.de' OK!
Dec 03 03:30:14 pve-1 pveupdate[3695637]: [Wed Dec  3 03:30:14 CET 2025] removing _acme-challenge.pve-1.int.domain.de. txt
Dec 03 03:30:14 pve-1 pveupdate[3695637]: Remove TXT record: _acme-challenge.pve-1.int.domain.de
Dec 03 03:30:14 pve-1 pveupdate[3695637]: All domains validated!
Dec 03 03:30:14 pve-1 pveupdate[3695637]: Creating CSR
Dec 03 03:30:19 pve-1 pveupdate[3695637]: Checking order status
Dec 03 03:30:19 pve-1 pveupdate[3695637]: Order is ready, finalizing order
Dec 03 03:30:25 pve-1 pveupdate[3695637]: valid!
Dec 03 03:30:25 pve-1 pveupdate[3695637]: Downloading certificate
Dec 03 03:30:25 pve-1 pveupdate[3707587]: POST of 'https://idm-1.int.domain.de/acme/cert/H_4ADw' failed - Server reply does not look like a PEM encoded certificate
Dec 03 03:30:25 pve-1 pveupdate[3695637]: POST of 'https://idm-1.int.domain.de/acme/cert/H_4ADw' failed - Server reply does not look like a PEM encoded certificate
Dec 03 03:30:25 pve-1 pveupdate[3695637]: <root@pam> end task UPID:pve-1:003892C3:01AFB564:692FA092:acmerenew::root@pam: POST of 'https://idm-1.int.domain.de/acme/cert/H_4ADw' failed - Server reply does not look like a PEM encoded certificate
I can download the certificate and manually import into pve-1 like this:
Code: 
root@pve-1:~# curl https://idm-1.int.domain.de/acme/cert/H_4ADw -o pve-1_cert.pem
root@pve-1:~# pvenode cert set pve-1_cert.pem -force
root@pve-1:~# systemctl restart pveproxy
Then the Web GUI is no longer reachable with following failure:
Code: 
root@pve-1:~# openssl s_client -connect pve-1.int.domain.de:8006
CONNECTED(00000003)
404764FF9A7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:322:
Checking the manually downloaded certificate with openssl x509 -in pve-1_cert.pem -text -noout or pvenode cert info gives no errors. Nevertheless the Web GUI (and API) are no longer reachable.

Any suggestions are highly appreciated.

Best regards

Dirk

ps: proxmox-ve: 8.4.0 latest updates installed
 
fabian said:
https://bugzilla.proxmox.com/show_bug.cgi?id=5978

what does "journalctl -b -u pveproxy | tail -n 200" say? likely there is some extra garbage in the file that you can remove..
Click to expand...
Thanks for quick response!

At 07:58 I installed the certificate manually:
Code: 
Dec 03 07:58:15 pve-1 systemd[1]: Stopping pveproxy.service - PVE API Proxy Server...
Dec 03 07:58:16 pve-1 pveproxy[2206098]: received signal TERM
Dec 03 07:58:16 pve-1 pveproxy[2206098]: server closing
Dec 03 07:58:16 pve-1 pveproxy[3921520]: worker exit
Dec 03 07:58:16 pve-1 pveproxy[3890200]: worker exit
Dec 03 07:58:16 pve-1 pveproxy[2206098]: worker 3909492 finished
Dec 03 07:58:16 pve-1 pveproxy[2206098]: worker 3890200 finished
Dec 03 07:58:16 pve-1 pveproxy[2206098]: worker 3921520 finished
Dec 03 07:58:16 pve-1 pveproxy[2206098]: server stopped
Dec 03 07:58:17 pve-1 systemd[1]: pveproxy.service: Deactivated successfully.
Dec 03 07:58:17 pve-1 systemd[1]: Stopped pveproxy.service - PVE API Proxy Server.
Dec 03 07:58:17 pve-1 systemd[1]: pveproxy.service: Consumed 32min 25.447s CPU time.
Dec 03 07:58:17 pve-1 systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
Dec 03 07:58:20 pve-1 pveproxy[3923271]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Dec 03 07:58:20 pve-1 pveproxy[3923278]: starting server
Dec 03 07:58:20 pve-1 pveproxy[3923278]: starting 3 worker(s)
Dec 03 07:58:20 pve-1 pveproxy[3923278]: worker 3923279 started
Dec 03 07:58:20 pve-1 pveproxy[3923278]: worker 3923281 started
Dec 03 07:58:20 pve-1 pveproxy[3923278]: worker 3923283 started
Dec 03 07:58:20 pve-1 systemd[1]: Started pveproxy.service - PVE API Proxy Server.

At 08:47 I restored the former one:
Code: 
Dec 03 08:47:46 pve-1 systemd[1]: Stopping pveproxy.service - PVE API Proxy Server...
Dec 03 08:47:48 pve-1 pveproxy[3923278]: received signal TERM
Dec 03 08:47:48 pve-1 pveproxy[3923278]: server closing
Dec 03 08:47:48 pve-1 pveproxy[3923283]: worker exit
Dec 03 08:47:48 pve-1 pveproxy[3923281]: worker exit
Dec 03 08:47:48 pve-1 pveproxy[3923279]: worker exit
Dec 03 08:47:48 pve-1 pveproxy[3923278]: worker 3923279 finished
Dec 03 08:47:48 pve-1 pveproxy[3923278]: worker 3923283 finished
Dec 03 08:47:48 pve-1 pveproxy[3923278]: worker 3923281 finished
Dec 03 08:47:48 pve-1 pveproxy[3923278]: server stopped
Dec 03 08:47:49 pve-1 systemd[1]: pveproxy.service: Deactivated successfully.
Dec 03 08:47:49 pve-1 systemd[1]: Stopped pveproxy.service - PVE API Proxy Server.
Dec 03 08:47:49 pve-1 systemd[1]: pveproxy.service: Consumed 8.710s CPU time.
Dec 03 08:47:49 pve-1 systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
Dec 03 08:47:51 pve-1 pveproxy[3961528]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Dec 03 08:47:51 pve-1 pveproxy[3961529]: starting server
Dec 03 08:47:51 pve-1 pveproxy[3961529]: starting 3 worker(s)
Dec 03 08:47:51 pve-1 pveproxy[3961529]: worker 3961530 started
Dec 03 08:47:51 pve-1 pveproxy[3961529]: worker 3961531 started
Dec 03 08:47:51 pve-1 pveproxy[3961529]: worker 3961532 started
Dec 03 08:47:51 pve-1 systemd[1]: Started pveproxy.service - PVE API Proxy Server.

Otherwise no errors, just plenty of those:
Code: 
Dec 03 10:25:52 pve-1 pveproxy[3961529]: worker 3961532 finished
Dec 03 10:25:52 pve-1 pveproxy[3961529]: starting 1 worker(s)
Dec 03 10:25:52 pve-1 pveproxy[3961529]: worker 4038120 started
Dec 03 10:26:12 pve-1 pveproxy[4038120]: Clearing outdated entries from certificate cache

I can try the bugfix later
 
fabian said:
https://bugzilla.proxmox.com/show_bug.cgi?id=5978

what does "journalctl -b -u pveproxy | tail -n 200" say? likely there is some extra garbage in the file that you can remove..
Click to expand...
Thanks Fabian, I used the bugfix from Comment 2 and the provided certificate from IDM was now accepted.

However, why is pvenode cert set accepting a possible invalid certificate and later pveproxy can be started without error? I would assume respective syntax checks. What's your opinion?

Kind regards, Dirk
 
You must log in or register to reply here.
Top Bottom