Certificate Management behind proxy

[NextTime]

Hi there!

Our PMG server is running in a DMZ with limited Internet access (behind outbound proxy).
We have set this in GUI->Configuration: System->Option->HTTP Proxy:

So far, this has not been a problem.

Today I tried to set up Let’s Encrypt certificate, but stumbled upon creating an ACME account in GUI.
I was able to create an account through CLI, but in GUI it's neither possible to create a new account nor view an existing one.

CLI, failed attempt:

# pmgconfig acme account register Default administrator@mydomain.com      Directory endpoints:
0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
2) Custom
Enter selection: 0

Attempting to fetch Terms of Service from 'https://acme-v02.api.letsencrypt.org/directory'..
Acme Server API misbehaved: failed to get directory info: [7] Couldn't connect to server at /usr/share/perl5/PMG/API2/ACME.pm line 368.

CLI, successful attempt:

# https_proxy="http://proxy_IP:3128"  pmgconfig acme account register Default administrator@mydomain.com
Directory endpoints:
0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
2) Custom
Enter selection: 0

Attempting to fetch Terms of Service from 'https://acme-v02.api.letsencrypt.org/directory'..
Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Do you agree to the above terms? [y|N]: y

Attempting to register account with 'https://acme-v02.api.letsencrypt.org/directory'..
Registering new ACME account..
Registration successful, account URL: 'https://acme-v02.api.letsencrypt.org/acme/acct/xxxxxxxxx'
Task OK

GUI, failed register new account:


GUI, failed view existing account:

So I have stopped further actions for now.

Is this the expected behaviour?
How can we force certificate management in GUI to use proxy as well?

Best,
Aleksei

 

[NextTime]

Continuing my previous post.
Considering the fact that I managed to register an ACME account through a proxy using a small workaround in the CLI, I hoped to order the SSL certificate for our server in the same way. But unfortunately this time the same workaround doesn't work.
The script does not even try to use a proxy to connect server via API (there is nothing in the proxy logs)

# pmgconfig acme cert order api --force
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/xxxxxxxxx/xxxxxxxxxx

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxx'
The validation for mail.mydomain.com is pending!
[Mon May 24 12:00:46 EEST 2021] Using zone.ee dns api
[Mon May 24 12:02:55 EEST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 7
[Mon May 24 12:02:55 EEST 2021] error dns/mail.mydomain.com
[Mon May 24 12:02:55 EEST 2021] invalid domain
[Mon May 24 12:02:55 EEST 2021] Error add txt for domain:_acme-challenge.mail.mydomain.com
command 'setpriv --reuid nobody --regid nogroup --clear-groups --reset-env -- /bin/bash /usr/share/proxmox-acme/proxmox-acme setup zone mail.mydomain.com' failed: exit code 1
Task command 'setpriv --reuid nobody --regid nogroup --clear-groups --reset-env -- /bin/bash /usr/share/proxmox-acme/proxmox-acme setup zone mail.mydomain.com' failed: exit code 1

# printenv |grep -i proxy
https_proxy=http://proxy_IP:3128
http_proxy=http://proxy_IP:3128

#  https_proxy="http://proxy_IP:3128" pmgconfig acme cert order api --force
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/xxxxxxxxx/xxxxxxxxxx

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxx'
The validation for mail.mydomain.com is pending!
[Mon May 24 12:05:13 EEST 2021] Using zone.ee dns api
[Mon May 24 12:07:23 EEST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 7
[Mon May 24 12:07:23 EEST 2021] error dns/mail.mydomain.com
[Mon May 24 12:07:23 EEST 2021] invalid domain
[Mon May 24 12:07:23 EEST 2021] Error add txt for domain:_acme-challenge.mail.mydomain.com
command 'setpriv --reuid nobody --regid nogroup --clear-groups --reset-env -- /bin/bash /usr/share/proxmox-acme/proxmox-acme setup zone mail.mydomain.com' failed: exit code 1
Task command 'setpriv --reuid nobody --regid nogroup --clear-groups --reset-env -- /bin/bash /usr/share/proxmox-acme/proxmox-acme setup zone mail.mydomain.com' failed: exit code 1

Can anyone advise on how to get it to work through a proxy?
Or it could be considered as a bug and I have to create a bugreport at the https://bugzilla.proxmox.com?

Thanks in advance,
Aleksei

 

[NextTime]

Hi guys!

I'd like to inform that I managed to make it work (in case anyone else is interested in it).
I slightly changed three files and now certificate management through a proxy works both in CLI and GUI.
And nothing seems to have broken
I'm not sure if this is done in the most elegant way, but it works.
And I hope that someone from the PROXMOX team will pay attention to this and add it to the main code, so that I and others like me do not have to make these changes again and again after each system update.

Now to the changes themselves.
Somewhere at the beginning of the file /usr/share/proxmox-acme/proxmox-acme add the following line:

PROXY_ADDRESS="$(pmgconfig dump| grep pmg.admin.http_proxy| awk {'print $3'})"

Then replace this original code (in two places, for _get and _post methods):

  _CURL="curl -L --silent --dump-header $HTTP_HEADER -g "
  if [ "$HTTPS_INSECURE" ]; then
    _CURL="$_CURL --insecure  "
  fi

to the following code:

  _CURL="curl -L --silent --dump-header $HTTP_HEADER -g "
  # use proxy if needed
  if [ -n "$PROXY_ADDRESS" ]; then
    _CURL="$_CURL --proxy $PROXY_ADDRESS "
  fi
  if [ "$HTTPS_INSECURE" ]; then
    _CURL="$_CURL --insecure  "
  fi

At the beginning of the /usr/share/perl5/PMG/API2/ACME.pm and /usr/share/perl5/PMG/API2/Certificates.pm add the following code
(identical in both files):

# setup proxy env
my $pmg_cfg = PMG::Config->new();
if (my $http_proxy = $pmg_cfg->get('admin', 'http_proxy')) {
$ENV{http_proxy} = $http_proxy;
$ENV{https_proxy} = $http_proxy;
}

That's all folks!

 

[Stoiko Ivanov]

Thanks for reporting (and fixing the issue)!
Since the code for LE certificates is shared with Proxmox VE - I'll try to come up with a version which is not PMG specific.

From a first look I think the modification of

/usr/share/proxmox-acme/proxmox-acme

should not be necessary? (curl honors the http_proxy and https_proxy variables)

 

[NextTime]

Hi Stoiko!
Thank you, I hope this helps someone else.

As for the variables http_proxy and https_proxy.
To keep the server configuration as controllable as possible, I only change configuration via web GUI if possible.
So I didn't add manually the proxy address to the file /etc/environment.
And as far as I know, these variables are not propagated from the PGM settings Configuration -> Options -> HTTP proxy
This means that these variables are undefined:

root@server:~# env|grep -i proxy
root@server:~#

My changes ensure that certificate management continues to work if you change the proxy settings through the GUI.

Thanks and regards,
Aleksei

 

[NextTime]

One more observation.
I do not know if this is somehow related to the change of the certificate,
but today I could not install updates from the GUI.
In the command prompt everything works as expected:

#apt update
#apt full-upgrade

But if I click the "Upgrade" button I get this error:



The following appears in tasks:

client connection: V4(127.0.0.1:51446)
invalid authentication, code 403
TASK ERROR: command '/usr/bin/termproxy 5900 --path /nodes/outmail -- pmgupgrade --shell' failed: exit code 1

Does anyone have any idea what's wrong with the xterms.js?

 

[Stoiko Ivanov]

Does anyone have any idea what's wrong with the xterms.js?

managed to reproduce it (it does not happen without your modifications):
* setting http_proxy in the environment of pmgdaemon/pmgproxy (which the patch to /usr/share/perl5/PMG/API2/ACME.pm and /usr/share/perl5/PMG/API2/Certificates.pm does) - causes the termproxy-connection to try to send the request to localhost:5900 (used for the shell-connection) to also get sent to the proxy (which rejects it - since it's for 'localhost')

Thanks for the report! - this means that we need to be more careful in setting this in general (I'll report here once I have a somewhat working patch)

 

[NextTime]

Thanks mate!

One extra line of code in both files solved this problem:

$ENV{no_proxy} = "localhost";

So now this code looks like this:

# setup proxy env
my $pmg_cfg = PMG::Config->new();
if (my $http_proxy = $pmg_cfg->get('admin', 'http_proxy')) {
$ENV{http_proxy} = $http_proxy;
$ENV{https_proxy} = $http_proxy;
$ENV{no_proxy} = "localhost";
}

I will explore further ...

Thanks and regards

 

[NextTime]

Hi everyone!

It's me again with the issues with proxy and ACME certificates.
It turned out that my fixes still don't work 100% automatically.
To obtain HTTPS certificates, the domains must be validated and the validation is valid for some time.
After validation expired you need to validate domains again.
In the process of finding solution to the problem, I made many attempts to obtain a certificate manually. One of them probably ended with a successful domain validation and further attempts to obtain a certificate (including automatic ones) were also successful.
Now, two months later, when my certificate started to expire, the acmerenew task tried to automatically renew the certificate.
And then the following facts became clear:

• domain validation is now required again
• validation must go through a proxy
• the validation process is launched by user nobody, who can neither launch pmgconfig, nor has access to the environment variables as http_proxy or https_proxy

The message "please run as root" refers to the command:

pmgconfig dump| grep pmg.admin.http_proxy| awk {'print $3'}

which works as expected from the command line.
As a temporary workaround, I have hardcoded the proxy server address in the /usr/share/proxmox-acme/proxmox-acme file.
But is there any more elegant solution (I'd rather use the settings from the PMG server GUI)?

BR,
Aleksei