A couple of months ago I've added Let's encrypt certificates to my cluster of PVE nodes. I've followed the documentation as far as I could and everything worked fine until a little less than 30 days ago.
I suddenly got an e-mail saying my certificates were up for renewal and noticed they weren't automatically being renewed. I renewed one to make sure I would maintain access and let the other sit as I wanted to see it play out.
Low and behold my certificate expired yesterday. pve-daily-update.service is somehow not updating my certificates of both my nodes. I've manually run pve-daily-update.service a couple of times, but any useful messages on /var/log/syslog about ACME or certificates are missing on both of them.
I've looked into /usr/bin/pveupdate (the script activated by pve-daily-update.service) and extracted the following into a separate script to test the workings of autorenewal. I've noticed the only way to exit without logging anything is when if ($node_config && $node_config->{acme}) evaluates to false. So I've added
else {
syslog ('info', ' No ACME config found for node');
}
to see if it indeed evaluates to false
Now when running this script I get my self-created message "No ACME config found for node" in the syslog. I'm assuming the script is running to see whether ACME is setup and otherwise skips this part. Somehow ($node_config && $node_config->{acme}) is resolving to false in my setup. This is very strange since when working manually I can order certificates and everything looks to be working and set up correctly.
Does anyone have any clue on how to proceed debugging this issue? As mentioned I had to manually renew my other node as well. So clearly something is wrong with both my setups.
I suddenly got an e-mail saying my certificates were up for renewal and noticed they weren't automatically being renewed. I renewed one to make sure I would maintain access and let the other sit as I wanted to see it play out.
Low and behold my certificate expired yesterday. pve-daily-update.service is somehow not updating my certificates of both my nodes. I've manually run pve-daily-update.service a couple of times, but any useful messages on /var/log/syslog about ACME or certificates are missing on both of them.
I've looked into /usr/bin/pveupdate (the script activated by pve-daily-update.service) and extracted the following into a separate script to test the workings of autorenewal. I've noticed the only way to exit without logging anything is when if ($node_config && $node_config->{acme}) evaluates to false. So I've added
else {
syslog ('info', ' No ACME config found for node');
}
to see if it indeed evaluates to false
Code:
#!/usr/bin/perl
use strict;
use warnings;
use IO::File;
use File::Find;
use File::stat;
use PVE::CertHelpers;
use PVE::Certificate;
use PVE::NodeConfig;
use PVE::INotify;
use PVE::Cluster;
use PVE::APLInfo;
use PVE::SafeSyslog;
use PVE::RPCEnvironment;
use PVE::API2::Subscription;
use PVE::API2::APT;
use PVE::API2::ACME;
initlog ('acme-update', 'daemon');
die "please run as root\n" if $> != 0;
$ENV{'PATH'} = '/sbin:/bin:/usr/sbin:/usr/bin';
PVE::INotify::inotify_init();
my $rpcenv = PVE::RPCEnvironment->init('cli');
$rpcenv->init_request();
$rpcenv->set_language($ENV{LANG});
$rpcenv->set_user('root@pam');
my $nodename = PVE::INotify::nodename();
eval {
my $node_config = PVE::NodeConfig::load_config($nodename);
if ($node_config && $node_config->{acme}) {
my $cert = PVE::CertHelpers::cert_path_prefix($nodename).".pem";
if (-e $cert) {
if (PVE::Certificate::check_expiry($cert, time() + 30*24*60*60)) {
my $r=`iptables -t nat -D PREROUTING -d 1.1.1.1/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 1.1.2.2:80`;
PVE::API2::ACME->renew_certificate({ node => $nodename });
$r=`iptables -t nat -I PREROUTING -d 1.1.1.1/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 1.1.2.2:80`;
} else {
syslog ('info', 'Custom certificate does not expire soon, skipping ACME renewal.');
}
} else {
syslog ('info', 'ACME config found for node, but no custom certificate exists. Skipping ACME renewal until initial certificate has been deployed.');
}
}
else {
syslog ('info', ' No ACME config found for node');
}
};
syslog ('err', "Renewing ACME certificate failed: $@") if $@;
exit (0);Now when running this script I get my self-created message "No ACME config found for node" in the syslog. I'm assuming the script is running to see whether ACME is setup and otherwise skips this part. Somehow ($node_config && $node_config->{acme}) is resolving to false in my setup. This is very strange since when working manually I can order certificates and everything looks to be working and set up correctly.
Does anyone have any clue on how to proceed debugging this issue? As mentioned I had to manually renew my other node as well. So clearly something is wrong with both my setups.
