ACME with DNS challenge?

[Mineralwasser]

Hi,

I want to generate Let's Encrypt SSL certificates for the Proxmox host and also for each service in my VMs.

The ACME client is built into the Proxmox GUI but as far as I can see it can only use the http challenge to request certificates?
And since my server is in a local network I don't want to open and forward port 80/443 to the proxmox machine just to get the certs.

For DNS challenge it isn't required to open these ports so I'd want to use this method instead.
I've already done that on a webserver running Debian & NGINX with the following command:

acme.sh --issue --dns <provider> \
-d <domain.tld> \
--keylength 4096 \
--key-file /etc/letsencrypt/<domain.tld>/key.pem \
--ca-file /etc/letsencrypt/<domain.tld>/ca.pem \
--cert-file /etc/letsencrypt/<domain.tld>/cert.pem \
--fullchain-file /etc/letsencrypt/<domain.tld>/fullchain.pem \
--reloadcmd "sudo /bin/systemctl reload nginx.service"

However in the Wiki it says:

Certificates are managed with the Proxmox VE Node management command (see the pvenode(1) manpage).

Do not replace or manually modify the automatically generated node certificate files in /etc/pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key or the cluster CA files in /etc/pve/pve-root-ca.pem and /etc/pve/priv/pve-root-ca.key.

So how could I achieve my goal to use the ACME DNS challenge to get a certificate for Proxmox?

 

[Mineralwasser]

In this (older) Wiki article the Let's Encrypt certificates are requested with the acme.sh client.
That's exactly what I wanted to do.

But why does the newer Wiki article say that the certificates shouldn't be replaced or modified manually?
Am I allowed to do it this way or not? What can go wrong? Won't it be recognized by the GUI if I do it this way?

 

[budy]

If you want to perform your requests via a DNS challenge, you need to be able to provide a token which is served by your outside domain's DNS server. If you want to secure an internal domain, DNS challenge is out of the question, since LetsEncrypt wouldn't be able to query that.

 

[Mineralwasser]

It'll work if I use a subdomain of a real domain for my internal network.
e.g. pve.local.domain.tld

I just want to know if I run into problems if I replace the certificates manually.
Guess I'll just try...

 

[budy]

If you get a wildcard cert than it will work.

 

[Mineralwasser]

But I don't want a wildcard cert.
I'm not exactly sure what you mean.

Okay, I've just tested it and it seems to work fine.
Here's what I did. (Example with INWX DNS provider)

1. Login via SSH as root
2. Install acme.sh
   

   wget -O -  https://get.acme.sh | sh

3. Backup existing node certificates
   

   cp /etc/pve/local/pve-ssl.key /etc/pve/local/pve-ssl.key_bak && \
   cp /etc/pve/local/pve-ssl.pem /etc/pve/local/pve-ssl.pem_bak

4. Set DNS API credentials (See here)
   

   export INWX_User="<username>"
   export INWX_Password="<password>"

5. If 2FA/TOTP is required for login to dns provider:
   

   export INWX_Shared_Secret="<totp-secret>"
   apt install oathtool

6. Try to issue certificate (DRY RUN)
   

   acme.sh \
   --issue \
   --dns dns_inwx \
   --domain pve.local.domain.tld \
   --keylength 4096 \
   --key-file /etc/pve/local/pve-ssl.key \
   --fullchain-file /etc/pve/local/pve-ssl.pem \
   --reloadcmd "systemctl restart pveproxy" \
   --debug \
   --test

7. If successfull, issue the certificate (Change --test to --force)
   

   acme.sh \
   --issue \
   --dns dns_inwx \
   --domain pve.local.domain.tld \
   --keylength 4096 \
   --key-file /etc/pve/local/pve-ssl.key \
   --fullchain-file /etc/pve/local/pve-ssl.pem \
   --reloadcmd "systemctl restart pveproxy" \
   --debug \
   --force

After that the Proxmox webinterface is using the newly generated certificate.
It also shows up correctly in the GUI under node > system > certificates.

And a cronjob to renew automatically should've been installed too.
You can verify with crontab -l.

I'm still confused by the warning in the Wiki...
Let's see if it keeps working fine...

 

[budy]

Right… I see… I am running a split-domain installation, where I do have the same domain locally as well as externally. A wildcart cert is of the form *.domain.tld and would cover all hosts or subdomains of domain .tld with one cert. This is, what I am doing, so I only need to deploy one certificate for all hosts/services I run on my internal network and the ones that I also make available to the internet.

 

[fabian]

the wiki says not to replace the 'pve-ssl.pem' and 'pve-ssl.key' files, because those are managed by PVE. if you want a certificate for the GUI then you should put it into 'pveproxy-ssl.pem' and 'pveproxy-ssl.key', which is used with higher priority by pveproxy. that's why the instructions also state to copy any custom certs to those paths

note that DNS support is currently being finalized for PVE 6.x (see pve-devel list archives).

 

[Mineralwasser]

Okay so that does mean I did it wrong.

So to fix that I should do the following:

1. Restore /etc/pve/local/pve-ssl.key and /etc/pve/local/pve-ssl.pem to the default keys generated by PVE.
2. Put my Let's Encrypt certificates on the same location but with different name: /etc/pve/local/pveproxy-ssl.key and /etc/pve/local/pveproxy-ssl.pem

Correct?

Great to hear that DNS support is coming.

 

[fabian]

yes, exactly!

 

[Mineralwasser]

Yep that also works.

Thank you.

Just asking because I'm curious.
Is pve-ssl.key and pve-ssl.pem "inactive" now, or is it used by PVE for some other purpose?

 

[fabian]

from the top of my head, at the moment it's only used for the generated spice configuration (which contains the self-signed PVE CA cert of your cluster as trust-anchor), because the spice remote viewer cannot be sanely pinned to a specific certificate (and I think even then it would not support migration anymore).

 

[inxsible]

To handle SSL and also as a way to not have to remember the services ip/domains and ports, I just created a reverse proxy container within proxmox. I installed caddy2 on it and it can now handle the automatic renewal of certificates etc. No cron job needed.

I wanted to do the same for the proxmox host as well.

1. What is the recommended way to set up a SSL cert for proxmox -- only via the proxmox-->Certificates --> Acme option
2. Or can I use the same reverse proxy server to handle the SSL certs like I do for the other 8 services that I use it for?
3. Will using a container within proxmox cause an issue with SSL certs during reboots of the proxmox machine provided I make the reverse proxy container auto start on reboots and give it enough time to start up before accessing the web-ui?

I would prefer using the reverse proxy method because that way all my SSL certs can be maintained in 1 place rather than having to go on and about 3-4 different places to maintain/debug something in case of issues

Thanks,
Inxsible

 

[inxsible]

I tried setting this up similar to my other services -- via caddy2 reverse proxy server but it didn't work.

I tried using IP:8006 in the dial option for caddy reverse proxy config, but that just returned an ERR_EMPTY_RESPONSE back. I even tried using the host:localdomain:8006 and that just gave a different error and couldn't load the page

So in any case, I have not been able to use the caddy server as a reverse proxy to the proxmox login page. I might try a couple other things, but if they don't work out, I might have to use the built-in ACME.sh option.

 

[Stoiko Ivanov]

I tried using IP:8006 in the dial option for caddy reverse proxy config, but that just returned an ERR_EMPTY_RESPONSE

AFAIR this is the error you get back from pveproxy if you connect via http and not https

I hope this helps!

 

[inxsible]

AFAIR this is the error you get back from pveproxy if you connect via http and not https

I hope this helps!

I might give it another try and see if I did something wrong in the first go around.

 

[fixjunk]

@inxsible were you ever able to get caddy v2 working on the proxmox host?

I tried today with a dns challenge and I was able to get a cert for my unrouted internal domain via dns challenge (e.g. pve.dom.com reverse_proxy localhost:8006).

but it throws a very long error and the web interface doesn't work when I browse pve.dom.com from the local network.

the posts above about proxmox handling its own certs may hold the answer... caddy probably conflicts with that.

are there other ways to be able to use just a subdomain.domain.com address without port and have https?