[SOLVED] ACME possibly not renewing certificate

[Daniel_Dog]

Hello everyone,

I was just doing some maintenance and updating my monitoring software and noticed that the Proxmox VE pveproxy-ssl certificate is only valid for another 18 days.
I have setup the certificate via the webGUI ACME function.
Now normally (or at least to what I am used to) 29/30 days before a let's encrypt certificate is about to expire, it should be auto renewed.
And given that 18 days left is less then 29/30 days left, I would have expected that the ACME renewed the certificate already.

So my question is:
Is this normal or should have the ACME already renewed the certificate?
i can also find nowhere when the ACME of Proxmox VE should renew the certificate so I would expect the normal 29/30 days in advance.
But if someone know if Proxmox VE uses different renew timings I would love to know what they are so that I know when I should start checking why it is not renewed yet.

 

[gfngfn256]

But if someone know if Proxmox VE uses different renew timings

I think it uses regular timings as shown here in the wiki:

If a node has been successfully configured with an ACME-provided certificate (either via pvenode or via the GUI), the certificate will be automatically renewed by the pve-daily-update.service. Currently, renewal will be attempted if the certificate has expired already, or will expire in the next 30 days.

If your node is running constantly - I believe it should have (already) updated the certs through the pve-daily-update.service.

Can you show output for the following:

systemctl list-timers
systemctl cat pve-daily-update.timer
systemctl cat pve-daily-update.service

 

[Daniel_Dog]

Thanks for the anwser.

I did find the article that you mentioned but totally overlook that it already stated it should auto renew 30/29 days before the certificate expires.
I also looked at the services you mentioned and they are also functioning normally.

I did find however the following in the pve-update.service log:

Jan 26 02:40:02 **myhost** systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
Jan 26 02:40:03 **myhost** pveupdate[2782727]: <root@pam> starting task UPIDve1:002A760C:0240C46C:67959273:aptupdate::root@pam:
Jan 26 02:40:05 **myhost** pveupdate[2782732]: update new package list: /var/lib/pve-manager/pkgupdates
Jan 26 02:40:07 **myhost** pveupdate[2782727]: <root@pam> end task UPIDve1:002A760C:0240C46C:67959273:aptupdate::root@pam: OK
Jan 26 02:40:07 **myhost** pveupdate[2782727]: Custom certificate does not expire soon, skipping ACME renewal.
Jan 26 02:40:07 **myhost** systemd[1]: pve-daily-update.service: Deactivated successfully.
Jan 26 02:40:07 **myhost** systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
Jan 26 02:40:07 **myhost** systemd[1]: pve-daily-update.service: Consumed 4.235s CPU time.

I have no clue why is says that the custom certificate does not expire soon and thus does not renew.
I do know that let's encrypt is only used for the pveproxy-ssl since let's encrypt is not used for clustering. (And my node is a standalone node that is not configured to be a cluster either.)

And here are all the other debug things:

root@**myhost**:~# systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2025-01-26 17:56:16 CET 2h 7min left Sat 2025-01-25 17:56:16 CET 21h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2025-01-27 00:00:00 CET 8h left Sun 2025-01-26 00:00:02 CET 15h ago dpkg-db-backup.timer dpkg-db-backup.service
Mon 2025-01-27 00:00:00 CET 8h left Sun 2025-01-26 00:00:02 CET 15h ago logrotate.timer logrotate.service
Mon 2025-01-27 01:02:11 CET 9h left Mon 2025-01-20 01:15:20 CET 6 days ago fstrim.timer fstrim.service
Mon 2025-01-27 03:30:28 CET 11h left Sun 2025-01-26 02:40:02 CET 13h ago pve-daily-update.timer pve-daily-update.service
Mon 2025-01-27 04:46:02 CET 12h left Sun 2025-01-26 13:07:02 CET 2h 41min ago apt-daily.timer apt-daily.service
Mon 2025-01-27 06:53:50 CET 15h left Sun 2025-01-26 06:54:45 CET 8h ago apt-daily-upgrade.timer apt-daily-upgrade.service
Mon 2025-01-27 07:59:47 CET 16h left Sun 2025-01-26 09:36:02 CET 6h ago man-db.timer man-db.service
Sun 2025-02-02 03:10:12 CET 6 days left Sun 2025-01-26 03:11:02 CET 12h ago e2scrub_all.timer e2scrub_all.service

9 timers listed.
Pass --all to see loaded but inactive timers, too.

root@**myhost**:~# systemctl cat pve-daily-update.timer
# /lib/systemd/system/pve-daily-update.timer
[Unit]
Description=Daily PVE download activities

[Timer]
OnCalendar=*-*-* 1:00
RandomizedDelaySec=5h
Persistent=true

[Install]
WantedBy=timers.target

root@**myhost**:~# systemctl cat pve-daily-update.service
# /lib/systemd/system/pve-daily-update.service
[Unit]
Description=Daily PVE download activities
After=network-online.target
Wants=network-online.target
Wants=pve-cluster.service
After=pve-cluster.service

[Service]
Type=oneshot
ExecStart=/usr/bin/pveupdate

 

[Daniel_Dog]

And in case it is important:

My server runs on the enterprise repository.
Package versions:

proxmox-ve: 8.3.0 (running kernel: 6.8.12-6-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-6
proxmox-kernel-6.8.12-6-pve-signed: 6.8.12-6
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
dnsmasq: 2.90-4~deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.4
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

 

[gfngfn256]

I don't use ACME myself, but can you check output for:

pvenode cert info

pvenode acme account list

pvenode acme account info

[redact as necessary - if you share output]

This may help you to assess Proxmox's view on these certs.

pveproxy-ssl certificate is only valid for another 18 days.

Are you sure?

 

[Daniel_Dog]

root@**My pve name**:~# pvenode cert info
┌─────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ filename │ pve-root-ca.pem │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ fingerprint │ **PVE FINGERPRINT** │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ subject │ /CN=Proxmox Virtual Environment/OU=feb47757-4d20-442b-b1e2-1b109ca59846/O=PVE Cluster Manager CA │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ issuer │ /CN=Proxmox Virtual Environment/OU=feb47757-4d20-442b-b1e2-1b109ca59846/O=PVE Cluster Manager CA │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notbefore │ 2024-12-31 03:39:24 │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notafter │ 2034-12-29 03:39:24 │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ public-key-type │ rsaEncryption │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ public-key-bits │ 4096 │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ san │ [] │
└─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ filename │ pve-ssl.pem │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ fingerprint │ **PVE FINGERPRINT** │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ subject │ /OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=pve1.danielhosting.eu │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ issuer │ /CN=Proxmox Virtual Environment/OU=feb47757-4d20-442b-b1e2-1b109ca59846/O=PVE Cluster Manager CA │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notbefore │ 2024-12-31 03:39:24 │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notafter │ 2026-12-31 03:39:24 │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ public-key-type │ rsaEncryption │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ public-key-bits │ 2048 │
├─────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ san │ - 127.0.0.1 │
│ │ - 0000:0000:0000:0000:0000:0000:0000:0001 │
│ │ - localhost │
│ │ - **My pve IP address** │
│ │ - **My pve name** │
│ │ - **My pve domain name** │
└─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────┐
│ filename │ pveproxy-ssl.pem │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ fingerprint │ **LET'S ENCRYPT SSL FINGERPRINT** │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ subject │ /CN=**My pve domain name** │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ issuer │ /C=US/O=Let's Encrypt/CN=R10 │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notbefore │ 2025-01-01 02:51:34 │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notafter │ 2025-04-01 03:51:33 │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ public-key-type │ rsaEncryption │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ public-key-bits │ 4096 │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ san │ - **My pve domain name** │
└─────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘

root@**My pve name**:~# pvenode acme account list
**My LE account name**

root@**My pve name**:~# pvenode acme account info
400 Parameter verification failed.
name: ACME account config file 'default' does not exist.
pvenode acme account info [<name>] [FORMAT_OPTIONS]

 

[Daniel_Dog]

root@**My pve name**:~# pvenode acme account info **My LE account name**
┌───────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ key │ value
╞═══════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
│ account │ contact:
│ │ - mailto:**email**
│ │ createdAt: 2025-01-01T02:49:41.790756111Z
│ │ key:
│ │ e: **removed key**
│ │ kty: RSA
│ │ n: **removed key**
│ │ use: sig
│ │ status: valid
├───────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ directory │ https://acme-v02.api.letsencrypt.org/directory
├───────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ location │ **account url**
├───────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ tos │ https://letsencrypt.org/documents/LE-SA-v1.4-April-3-2024.pdf
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

 

[Daniel_Dog]

And also when I login into Proxmox VE and go to "Datacenter > PVE Node > System > Certificates" it also show: "Using Account: **My LE account name**"

 

[gfngfn256]

│ notbefore │ 2025-01-01 02:51:34 │
├─────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┤
│ notafter │ 2025-04-01 03:51:33 │

Looks to me like it is still valid until the first of April 2025

 

[Daniel_Dog]

Well, it seems like a issue on my monitoring side and on my head math.

I noticed the issue through my monitoring software and then I just did my renewal date math wrong in my head.
In my head is is total of 3 months and renew after 2. (But it is 4 months and renewal at 3.)

Thanks for the help.