[TUTORIAL] How-To -- Lets Encrypt and PMG

ChFin

Well-Known Member
Jan 30, 2018
51
16
48
Let's Encrypt is a free, automated and open certificate authority. The CA issues standard domain validation certificates. The certificates can be used for web servers, email servers, FTP servers and many more. Email encryption and code signing requires a different type of certificate that Let's encrypt doesn't issue.

Below are a few links that you might want to read:

You can find the documentation here:
https://letsencrypt.org/docs/

Important: What you need to know about TLS-SNI validation issues
https://community.letsencrypt.org/t...to-know-about-tls-sni-validation-issues/50811

For Let's Encrypt and DNS CAA records read this document:
https://letsencrypt.org/docs/caa/

If you don't care about CAA, you don't have to do anything but in case of errors check the CAA error section
in the document above.

If you would like to use CAA records, check out the CAA generator of:
https://sslmate.com/caa/


Proxmox Mail Gateway uses Keys and certificate to make secure connections. The application uses the keys and certificates stored at:
/etc/pmg/pmg-api.pem -- Key and certificate (combined) used be the HTTPs server (API)
/etc/pmg/pmg-tls.pem -- Key and certificate (combined) to encrypt mail traffic (TLS)

Be aware, that the application itself can overwrite some key and certificate files, e.g. when you disable & enable the TLS settings in the mail proxy configuration.

To replace the TLS certificates we request certificates from the Let's encrypt CA. We do that with the certbot application.

Preconditions
Firewall http/inbound is open.
Hostname is properly set
PMG Mail Proxy configuration has enabled TLS and TLS logging

Installation
It is recommended to use the certbot application from the stretch-backports repository. To install files from stretch-backports add the repository to your sources.list:

# vi /etc/apt/sources.list

Add the line
deb http://ftp.debian.org/debian stretch-backports main

After you edited the file run:

# apt-get update

To install certbot run:

# apt-get install certbot -t stretch-backports

Before we request the certificate we create a post-hook script in our /root directory.

# cd /root
# vi certbot-post-hook.sh


Code:
#!/bin/bash
# post-hook see renewalparams in /etc/letsencrypt/renewal/$(hostname -f).conf

# replace mail certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-tls.pem
chown root:root /etc/pmg/pmg-tls.pem
chmod 600 /etc/pmg/pmg-tls.pem

# replace http certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-api.pem
chown root:www-data /etc/pmg/pmg-api.pem
chmod 640 /etc/pmg/pmg-api.pem

systemctl restart pmgproxy

Set access permissions to your post-hook script:

# chmod 700 certbot-post-hook.sh

Now you can request the certificate:

# certbot certonly --authenticator standalone --preferred-challenges http --post-hook "/root/certbot-post-hook.sh" -d $(hostname -f)

Enter your email address, agree to the terms of service and answer the question if you would like to share your email address.

Congratulations you have requested and installed (via post-hook) your Let's encrypt certificate.

Automated renewal

The debian package comes with a cron job and a systemd timer.

The cron job won't execute the renew command when you are running systemd (if /run/systemd/system is detected). It's done via certbot.timer
For automatic renewal just make sure certbot.timer is enabled & started. Post-Hook and Preferred Challenges were stored in /etc/letsencrypt/renewal/($hostname -f).conf during certificate request.

# systemctl status certbot.timer

Status should be enabled/active (waiting).

Check your certificate in the browser and watch the TLS log output in /var/log/mail.log.

Verify your mail server tls encryption here: https://ssl-tools.net/mailservers
 
Last edited:
Certbot is recommended by Let's Encrypt and most people should start with it. If certbot does not meet your needs you are free to try a 3rd party client, but keep in mind that Let’s Encrypt does not control or review third party clients and cannot make any guarantees about their safety or reliability.
 
You are totally right. I almost always choose acme.sh for is simplicity. It almost does not have any dependencies and runs out of the box as long as you have bash available. Also the content of the whole script is available online.

As you can see here PVE uses acme.sh already for PVE setup as possible option.

Everybody choose what he/she wants. But I'd rather choose a script which access rights I can easily limit than any package I have to install.
 
Last edited:
  • Like
Reactions: DonMcCoy
Here is an alternative script when using acme.sh.

I will push the complete manual to my Github repository once I find time for that.

Code:
#!/bin/bash

# How to install cert after acme initial cert was pulled
# chmod +x /root/acme-post-hook.sh
# acme.sh --install-cert -d $(hostname -f) --key-file /root/.acme.sh/$(hostname -f)/$(hostname -f).key.pem --fullchain-file /root/.acme.sh/$(hostname -f)/fullchain.pem --reloadcmd "bash /root/acme-post-hook.sh"

# replace mail certificate
cat /root/.acme.sh/$(hostname -f)/fullchain.pem /root/.acme.sh/$(hostname -f)/$(hostname -f).key.pem >/etc/pmg/pmg-tls.pem
chown root:root /etc/pmg/pmg-tls.pem
chmod 600 /etc/pmg/pmg-tls.pem

# replace http certificate - we don't need this if you are using an external proxy and don't want to update each fingerprint every time the certificate renewal took place.
#cat /root/.acme.sh/$(hostname -f)/fullchain.pem /root/.acme.sh/$(hostname -f)/$(hostname -f).key.pem >/etc/pmg/pmg-api.pem
#chown root:www-data /etc/pmg/pmg-api.pem
#chmod 640 /etc/pmg/pmg-api.pem

systemctl restart pmgproxy
# If you are using haproxy as local reverse proxy
# systemctl restart haproxy
 
Last edited:
I started using acme.sh but in the end I've preferred certbot over acme.sh.
One nice thing of certbot is that it ships with a systemd timer so you could integrate it on the services page of the web ui and start/stop see the log.
 
I started using acme.sh but in the end I've preferred certbot over acme.sh.
One nice thing of certbot is that it ships with a systemd timer so you could integrate it on the services page of the web ui and start/stop see the log.

Creating a service (including timer) is easy. One should have proper monitoring anyways. Watching some gui entry for a renewal service that runs sometimes does not add much value for me.
 
Thanks for your How-To

I would add something for cluster and multidomain. You cannot use it out of the box but perhaps it helps some people.

I use two domains with two diffrent dns server providers. I have some domaisn for mx and the hostnames itself so I need an multidomain cert. If the cert changes cluster functiones are broken because of the changing fingerprint.

So 1. I need to update DNS records while generating the certs and 2. I have to update the fingerprints in cluster config.

I found some tips on the internet (github, AnalogJ, lexicon (Could not insert url sorry)) and added some code. Could be improved!

My solution (you need to change this for your needs!):

Code:
certbot certonly --manual --preferred-challenges dns --manual-auth-hook "/etc/letsencrypt/certbot.default.sh auth" --manual-cleanup-hook "/etc/letsencrypt/certbot.default.sh cleanup" --post-hook="/etc/letsencrypt/certbot-post-hook.sh" -d host1 -d host2 -d mx1 -d mx2

/etc/letsencrypt/certbot.default.sh
Code:
#!/usr/bin/env bash
#

set -euf -o pipefail

PROVIDER1="abc"
PROVIDER1_CREDENTIALS=("XXX")
PROVIDER2="def"
PROVIDER2_CREDENTIALS=("XXX")
PROVIDER_UPDATE_DELAY=30

# To be invoked via Certbot's --manual-auth-hook
function auth {
    if [[ "${CERTBOT_DOMAIN}" == *"domain1"* ]]; then
        lexicon "${PROVIDER1}" "${PROVIDER1_CREDENTIALS[@]}" \
        create "${CERTBOT_DOMAIN}" TXT --name "_acme-challenge.${CERTBOT_DOMAIN}" --content "${CERTBOT_VALIDATION}"
    else
        lexicon "${PROVIDER2}" "${PROVIDER2_CREDENTIALS[@]}" \
        create "${CERTBOT_DOMAIN}" TXT --name "_acme-challenge.${CERTBOT_DOMAIN}" --content "${CERTBOT_VALIDATION}"
    fi

    sleep "${PROVIDER_UPDATE_DELAY}"
}

# To be invoked via Certbot's --manual-cleanup-hook
function cleanup {
    if [[ "${CERTBOT_DOMAIN}" == *"domain1"* ]]; then
        lexicon "${PROVIDER1}" "${PROVIDER1_CREDENTIALS[@]}" \
        delete "${CERTBOT_DOMAIN}" TXT --name "_acme-challenge.${CERTBOT_DOMAIN}" --content "${CERTBOT_VALIDATION}"
    else
        lexicon "${PROVIDER2}" "${PROVIDER2_CREDENTIALS[@]}" \
        delete "${CERTBOT_DOMAIN}" TXT --name "_acme-challenge.${CERTBOT_DOMAIN}" --content "${CERTBOT_VALIDATION}"
    fi
}

HANDLER=$1; shift;
if [ -n "$(type -t $HANDLER)" ] && [ "$(type -t $HANDLER)" = function ]; then
  $HANDLER "$@"
fi

/etc/letsencrypt/certbot-post-hook.sh
Code:
#!/bin/bash

# replace mail certificate
cat /etc/letsencrypt/live/host1/fullchain.pem /etc/letsencrypt/live/host1/privkey.pem >/etc/pmg/pmg-tls.pem
chown root:root /etc/pmg/pmg-tls.pem
chmod 600 /etc/pmg/pmg-tls.pem
scp /etc/pmg/pmg-tls.pem host2:/etc/pmg/pmg-tls.pem

# replace http certificate
cat /etc/letsencrypt/live/host1/fullchain.pem /etc/letsencrypt/live/host1/privkey.pem >/etc/pmg/pmg-api.pem
chown root:www-data /etc/pmg/pmg-api.pem
chmod 640 /etc/pmg/pmg-api.pem
scp /etc/pmg/pmg-api.pem host2:/etc/pmg/pmg-api.pem

# generiere cert hash und teile mit cluster nodes
HASH="$(openssl x509 -in /etc/pmg/pmg-api.pem -noout -fingerprint -sha256 | cut -d'=' -f2)"
cp /etc/pmg/cluster.conf /etc/pmg/cluster.conf.bkp
sed "s/.*fingerprint.*/        fingerprint "$HASH"/" /etc/pmg/cluster.conf.bkp > /etc/pmg/cluster.conf
scp /etc/pmg/cluster.conf host2:/etc/pmg/cluster.conf

# starte dienste neu
systemctl restart pmgproxy
ssh root@host2 'systemctl restart pmgproxy'

Later I would add TLSA records update for DANE. Record generating is possible with
Code:
tlsa --create --selector 1 --port 25 --certificate /etc/letsencrypt/live/host1/cert.pem host1
... host2
... mx1
... mx2
Only some string magic and lexicon commands are missing.
 
Last edited:
Thanks for this one

Here is what i use to put new hash from cert per server:

Code:
#!/bin/bash
HASH="$(openssl x509 -in /etc/pmg/pmg-api.pem -noout -fingerprint -sha256 | cut -d'=' -f2)"
sed -i "0,/fingerprint/{s/.*fingerprint.*/        fingerprint ${HASH}/}" /etc/pmg/cluster.conf

It works on nodes too, just adjust the "0" to find your node side fingerprint.
 
Last edited:
here i do have a working solution for 2 gateways updating each other all the time when a new certificate will be issued:

the bashscript:
Code:
#!/bin/bash
# post-hook see renewalparams in /etc/letsencrypt/renewal/$(hostname -f).conf
# https://forum.proxmox.com/threads/how-to-lets-encrypt-and-pmg.41493/

# replace mail certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-tls.pem
chown root:root /etc/pmg/pmg-tls.pem
chmod 600 /etc/pmg/pmg-tls.pem

# replace http certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-api.pem
chown root:www-data /etc/pmg/pmg-api.pem
chmod 640 /etc/pmg/pmg-api.pem

cp /etc/pmg/cluster.conf /etc/pmg/cluster.conf.bkp
HASH="$(openssl x509 -in /etc/pmg/pmg-api.pem -noout -fingerprint -sha256 | cut -d'=' -f2)"

if [[ $HOSTNAME == mta01 ]]; then
 awk -v HASH="        fingerprint $HASH" '{ if (NR == 2) {print HASH} else {print $0}}' /etc/pmg/cluster.conf.bkp > /etc/pmg/cluster.conf
 scp  /etc/pmg/cluster.conf user@mta02:/etc/pmg/cluster.conf
 ssh user@mta02 'systemctl restart pmgproxy'
else
 awk -v HASH="        fingerprint $HASH" '{ if (NR == 10) {print HASH} else {print $0}}' /etc/pmg/cluster.conf.bkp > /etc/pmg/cluster.conf
 scp  /etc/pmg/cluster.conf user@mta01:/etc/pmg/cluster.conf
 ssh user@mta01 'systemctl restart pmgproxy'
fi

systemctl restart pmgproxy

and then you need to fire up the certbot:

Code:
certbot certonly --authenticator standalone --preferred-challenges http --post-hook "/path/to/certbot-post-hook.sh" -d $(hostname -f)
 
Let's Encrypt is a free, automated and open certificate authority. The CA issues standard domain validation certificates. The certificates can be used for web servers, email servers, FTP servers and many more. Email encryption and code signing requires a different type of certificate that Let's encrypt doesn't issue.

Below are a few links that you might want to read:

You can find the documentation here:
https://letsencrypt.org/docs/

Important: What you need to know about TLS-SNI validation issues
https://community.letsencrypt.org/t...to-know-about-tls-sni-validation-issues/50811

For Let's Encrypt and DNS CAA records read this document:
https://letsencrypt.org/docs/caa/

If you don't care about CAA, you don't have to do anything but in case of errors check the CAA error section
in the document above.

If you would like to use CAA records, check out the CAA generator of:
https://sslmate.com/caa/


Proxmox Mail Gateway uses Keys and certificate to make secure connections. The application uses the keys and certificates stored at:
/etc/pmg/pmg-api.pem -- Key and certificate (combined) used be the HTTPs server (API)
/etc/pmg/pmg-tls.pem -- Key and certificate (combined) to encrypt mail traffic (TLS)

Be aware, that the application itself can overwrite some key and certificate files, e.g. when you disable & enable the TLS settings in the mail proxy configuration.

To replace the TLS certificates we request certificates from the Let's encrypt CA. We do that with the certbot application.

Preconditions
Firewall http/inbound is open.
Hostname is properly set
PMG Mail Proxy configuration has enabled TLS and TLS logging

Installation
It is recommended to use the certbot application from the stretch-backports repository. To install files from stretch-backports add the repository to your sources.list:

# vi /etc/apt/sources.list

Add the line
deb http://ftp.debian.org/debian stretch-backports main

After you edited the file run:

# apt-get update

To install certbot run:

# apt-get install certbot -t stretch-backports

Before we request the certificate we create a post-hook script in our /root directory.

# cd /root
# vi certbot-post-hook.sh


Code:
#!/bin/bash
# post-hook see renewalparams in /etc/letsencrypt/renewal/$(hostname -f).conf

# replace mail certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-tls.pem
chown root:root /etc/pmg/pmg-tls.pem
chmod 600 /etc/pmg/pmg-tls.pem

# replace http certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-api.pem
chown root:www-data /etc/pmg/pmg-api.pem
chmod 640 /etc/pmg/pmg-api.pem

systemctl restart pmgproxy

Set access permissions to your post-hook script:

# chmod 700 certbot-post-hook.sh

Now you can request the certificate:

# certbot certonly --authenticator standalone --preferred-challenges http --post-hook "/root/certbot-post-hook.sh" -d $(hostname -f)

Enter your email address, agree to the terms of service and answer the question if you would like to share your email address.

Congratulations you have requested and installed (via post-hook) your Let's encrypt certificate.

Automated renewal

The debian package comes with a cron job and a systemd timer.

The cron job won't execute the renew command when you are running systemd (if /run/systemd/system is detected). It's done via certbot.timer
For automatic renewal just make sure certbot.timer is enabled & started. Post-Hook and Preferred Challenges were stored in /etc/letsencrypt/renewal/($hostname -f).conf during certificate request.

# systemctl status certbot.timer

Status should be enabled/active (waiting).

Check your certificate in the browser and watch the TLS log output in /var/log/mail.log.

Verify your mail server tls encryption here: https://ssl-tools.net/mailservers


Tested and working with the latest version 6.1. Thank you!
 
Why is not integrated in the gui like in PVE ?
Because this has not been implemented (yet) - actually we are planning to integrate the solution used in PVE with PMG at some point in the future - however no guarantees when this will be implemented.
 
  • Like
Reactions: DerDanilo
A first start would be a configuration option to let the PMG container access the PVE cert store.
 
here i do have a working solution for 2 gateways updating each other all the time when a new certificate will be issued:

the bashscript:
Code:
#!/bin/bash
# post-hook see renewalparams in /etc/letsencrypt/renewal/$(hostname -f).conf
# https://forum.proxmox.com/threads/how-to-lets-encrypt-and-pmg.41493/

# replace mail certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-tls.pem
chown root:root /etc/pmg/pmg-tls.pem
chmod 600 /etc/pmg/pmg-tls.pem

# replace http certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-api.pem
chown root:www-data /etc/pmg/pmg-api.pem
chmod 640 /etc/pmg/pmg-api.pem

cp /etc/pmg/cluster.conf /etc/pmg/cluster.conf.bkp
HASH="$(openssl x509 -in /etc/pmg/pmg-api.pem -noout -fingerprint -sha256 | cut -d'=' -f2)"

if [[ $HOSTNAME == mta01 ]]; then
awk -v HASH="        fingerprint $HASH" '{ if (NR == 2) {print HASH} else {print $0}}' /etc/pmg/cluster.conf.bkp > /etc/pmg/cluster.conf
scp  /etc/pmg/cluster.conf user@mta02:/etc/pmg/cluster.conf
ssh user@mta02 'systemctl restart pmgproxy'
else
awk -v HASH="        fingerprint $HASH" '{ if (NR == 10) {print HASH} else {print $0}}' /etc/pmg/cluster.conf.bkp > /etc/pmg/cluster.conf
scp  /etc/pmg/cluster.conf user@mta01:/etc/pmg/cluster.conf
ssh user@mta01 'systemctl restart pmgproxy'
fi

systemctl restart pmgproxy

and then you need to fire up the certbot:

Code:
certbot certonly --authenticator standalone --preferred-challenges http --post-hook "/path/to/certbot-post-hook.sh" -d $(hostname -f)

I've a bit updated script, this one will automatically update all nodes in the cluster.
Code:
#!/bin/bash
# post-hook see renewalparams in /etc/letsencrypt/renewal/$(hostname -f).conf

# replace mail certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-tls.pem
chown root:root /etc/pmg/pmg-tls.pem
chmod 600 /etc/pmg/pmg-tls.pem

# replace http certificate
cat /etc/letsencrypt/live/$(hostname -f)/fullchain.pem /etc/letsencrypt/live/$(hostname -f)/privkey.pem >/etc/pmg/pmg-api.pem
chown root:www-data /etc/pmg/pmg-api.pem
chmod 640 /etc/pmg/pmg-api.pem

# get old HASH from the cluster config
OLDHASH="$(grep "name $(hostname)$" /etc/pmg/cluster.conf -B4 | grep fingerprint | awk '{print $2}')"
# get certificate hash from the new hash file
NEWHASH="$(openssl x509 -in /etc/pmg/pmg-api.pem -noout -fingerprint -sha256 | cut -d'=' -f2)"

# update cluster config only when hash is updated
if [ "$OLDHASH" != "$NEWHASH" ]
then
  echo -e "The hash: \n${OLDHASH}\nwill be replaced with\n${NEWHASH}\n"
  sed -i.bak -e "s/$OLDHASH/$NEWHASH/g" /etc/pmg/cluster.conf
fi

# get own IP adress
OWNIP="$(hostname -i)"
# get all IP addresses from cluster, and remove OWNIP
OTHERS=$(grep $'\tip' /etc/pmg/cluster.conf | grep -v ${OWNIP} | awk '{print $2}')

# loop through $OTHERS and copy cluster.conf file, and then restart pmg on that host if copy went OK
for host in $OTHERS
do
  echo "Copy updated cluster.conf to ${host}."
  if scp  /etc/pmg/cluster.conf user@${host}:/etc/pmg/cluster.conf 2>&1 >/dev/null
  then
    echo "Restarting PMGproxy on ${host}"
    ssh user@${host} systemctl restart pmgproxy
  else
    echo "Copy went wrong";
  fi
done
systemctl restart pmgproxy
 
  • Like
Reactions: DerDanilo
Hello,

Is this only for a fresh install of Proxmox? I only as because I did some server hardening and installed fail2ban. When I tried to install letsencrypt, it will not let me. I am assuming it pertains to something done in post-install? I appreciate any input you can provide.

Robert
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!