Could not renew certificate

danb35 · Jan 11, 2023

tl;dr: I've been getting emails from by backup server for about the last 10 days that it was unable to renew my Let's Encrypt certificate.

My PBS has been set up to get a cert from Let's Encrypt using DNS validation via acme-dns since September 2021. It's successfully renewed every 60 days since then, until 2 Nov 22. The next renewal would thus have been on 2 Jan 23, but that morning (and every morning since), I got an email saying:

Code:

Proxmox Backup Server was not able to renew a TLS certificate.

Error: '/usr/share/proxmox-acme/proxmox-acme setup' exited with error (1)

Please visit the web interface for further details:

<https://pbs.[redacted]:8007/#pbsCertificateConfiguration>

When I try the certificate order via the web GUI, I get a bit more information:

Code:

2023-01-11T06:35:46-05:00: Placing ACME order
2023-01-11T06:35:47-05:00: Order URL: https://acme-v02.api.letsencrypt.org/acme/order/189289420/157787151517
2023-01-11T06:35:47-05:00: Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/193780393007'
2023-01-11T06:35:47-05:00: The validation for pbs.[redacted] is pending
2023-01-11T06:35:47-05:00: Setting up validation plugin
2023-01-11T06:35:47-05:00: [Wed Jan 11 06:35:47 EST 2023] Using acme-dns

2023-01-11T06:35:47-05:00: /usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 29: _clearaccountconf_mutable: command not found

2023-01-11T06:35:47-05:00: /usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 31: _clearaccountconf_mutable: command not found

2023-01-11T06:35:47-05:00: /usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 33: _clearaccountconf_mutable: command not found

2023-01-11T06:35:48-05:00: [Wed Jan 11 06:35:48 EST 2023] invalid response of acme-dns

2023-01-11T06:35:48-05:00: [Wed Jan 11 06:35:48 EST 2023] Error add txt for domain:_acme-challenge.pbs.[redacted]

2023-01-11T06:35:48-05:00: Sleeping 5 seconds to wait for TXT record propagation
2023-01-11T06:35:53-05:00: TASK ERROR: '/usr/share/proxmox-acme/proxmox-acme setup' exited with error (1)

It looks like a bug in the dns_acmedns.sh script, except that that appears identical to the one from acme.sh.

I suspect this issue is going to show up on my PVE nodes as well, but their certs aren't due for renewal yet, so the only place I've seen it so far is on my backup server.

Stoiko Ivanov · Jan 11, 2023

see the thread on the pve-user list (also click on 'Next message' in thread) - seems the configuration changed a bit with the latest acme.sh updates libproxmox-acme-plugins pulled in
https://lists.proxmox.com/pipermail/pve-user/2023-January/016958.html

From a quick glance the _clearaccountconf_mutable is used to fetch those (updated) config-values

If this is the cause for your setup as well it should be fixed with the next update of libproxmox-acme-plugins (as @t.lamprecht said on the list )

I hope this helps!

danb35 · Jan 11, 2023

Stoiko Ivanov said:
I hope this helps!

Indeed it did, thanks--editing /usr/share/proxmox-acme/dnsapi/dns_acmedns.sh to add the ACMEDNS_BASE_URL lets it work for the time being, though it's obviously a bit of a hack. Any idea when that next update is expected?

Stoiko Ivanov · Jan 11, 2023

danb35 said:
it's obviously a bit of a hack. Any idea when that next update is expected?

No hard deadline here - with the dns-plugins it mostly depends on how many users run into issues and we usually try to wait for at least one cycle of renewals (so that issues in other plugins also have a chance to be discovered)

you just need to make sure to recheck your plugin config on the next update (or in my case that would be when the first warning notification reaches me ... )

guerby · Jan 19, 2023

Stoiko Ivanov said:
No hard deadline here - with the dns-plugins it mostly depends on how many users run into issues and we usually try to wait for at least one cycle of renewals (so that issues in other plugins also have a chance to be discovered)

you just need to make sure to recheck your plugin config on the next update (or in my case that would be when the first warning notification reaches me ... )

Just hit this issue on our proxmox, applied the patch from the mail thread.

Code:

root@pve:/usr/share/proxmox-acme# diff -u dns-challenge-schema.json~ dns-challenge-schema.json
--- dns-challenge-schema.json~    2022-12-07 13:20:49.000000000 +0100
+++ dns-challenge-schema.json    2023-01-19 15:05:25.289579061 +0100
@@ -10,8 +10,8 @@
             "description" : "The subdomain you got from acme-dns registration",
             "type" : "string"
          },
-         "ACMEDNS_UPDATE_URL" : {
-            "description" : "The API update endpoint",
+         "ACMEDNS_BASE_URL" : {
+            "description" : "The API base url",
             "type" : "string"
          },
          "ACMEDNS_USERNAME" : {

Then systemctl restart pveproxy and re-fill the new ACMEDNS_BASE_URL in the datacenter/acme web ui, then requesting the certificate worked.

faramirza · Jan 20, 2023

danb35 said:
Indeed it did, thanks--editing /usr/share/proxmox-acme/dnsapi/dns_acmedns.sh to add the ACMEDNS_BASE_URL lets it work for the time being, though it's obviously a bit of a hack. Any idea when that next update is expected?

Can you please share what you changed to fix the problem?

MasterPhi · Jan 21, 2023

I've applied the fix, systemctl restart pveproxy and re-filled the new ACMEDNS_BASE_URL in the datacenter/acme web ui.
Though the renew works, log output still has the error:

Code:

Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/85548185/160055827507

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/196917919747'
The validation for pve.somebody.com is pending!
[Sat Jan 21 12:03:54 CET 2023] Using acme-dns
/usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 29: _clearaccountconf_mutable: command not found
/usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 31: _clearaccountconf_mutable: command not found
/usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 33: _clearaccountconf_mutable: command not found
Add TXT record: _acme-challenge.pve.somebody.com
Sleeping 30 seconds to wait for TXT record propagation
Triggering validation
Sleeping for 5 seconds
Status is 'valid', domain 'pve.somebody.com' OK!
[Sat Jan 21 12:04:30 CET 2023] Using acme-dns
Remove TXT record: _acme-challenge.pve.somebody.com

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/196917919757'
The validation for trust.somebody.com is pending!
[Sat Jan 21 12:04:31 CET 2023] Using acme-dns
/usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 29: _clearaccountconf_mutable: command not found
/usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 31: _clearaccountconf_mutable: command not found
/usr/share/proxmox-acme/dnsapi/dns_acmedns.sh: line 33: _clearaccountconf_mutable: command not found
Add TXT record: _acme-challenge.trust.somebody.com
Sleeping 30 seconds to wait for TXT record propagation
Triggering validation
Sleeping for 5 seconds
Status is 'valid', domain 'trust.somebody.com' OK!
[Sat Jan 21 12:05:07 CET 2023] Using acme-dns
Remove TXT record: _acme-challenge.trust.somebody.com

All domains validated!

Creating CSR
Checking order status
Order is ready, finalizing order
valid!

Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
TASK OK

danb35 · Jan 26, 2023

guerby said:
re-fill the new ACMEDNS_BASE_URL in the datacenter/acme web ui,

That seems to have disappeared in my cluster:

Anyone know where Datacenter -> ACME has moved to?

Neobin · Jan 26, 2023

danb35 said:
That seems to have disappeared in my cluster:
View attachment 46057
Anyone know where Datacenter -> ACME has moved to?

Are you logged in as another user than root?
Currently it is root-only:
https://forum.proxmox.com/threads/d...-root-and-user-with-administrator-role.120982

danb35 · Jan 26, 2023

Neobin said:
Are you logged in as another user than root?

That was indeed it, thanks.

leompl · Feb 11, 2023

guerby said:

Just hit this issue on our proxmox, applied the patch from the mail thread.

Code:

root@pve:/usr/share/proxmox-acme# diff -u dns-challenge-schema.json~ dns-challenge-schema.json
--- dns-challenge-schema.json~    2022-12-07 13:20:49.000000000 +0100
+++ dns-challenge-schema.json    2023-01-19 15:05:25.289579061 +0100
@@ -10,8 +10,8 @@
             "description" : "The subdomain you got from acme-dns registration",
             "type" : "string"
          },
-         "ACMEDNS_UPDATE_URL" : {
-            "description" : "The API update endpoint",
+         "ACMEDNS_BASE_URL" : {
+            "description" : "The API base url",
             "type" : "string"
          },
          "ACMEDNS_USERNAME" : {

Then systemctl restart pveproxy and re-fill the new ACMEDNS_BASE_URL in the datacenter/acme web ui, then requesting the certificate worked.

Your fix it's working too for PMG and plugin acme-dns. Big thanks!

# Change below lines and restart: systemctl restart pmgproxy

# I have these errors too

... BUT?

Success

Search

Search

Could not renew certificate

danb35

Renowned Member

Stoiko Ivanov

Proxmox Staff Member

danb35

Renowned Member

Stoiko Ivanov

Proxmox Staff Member

guerby

Member

faramirza

Member

MasterPhi

Well-Known Member

danb35

Renowned Member

Neobin

Famous Member

danb35

Renowned Member

leompl

New Member

Attachments