[SOLVED] lets-encrypt-connection-refused-status-400: acme via dns-plugin

Jul 16, 2021
11
1
3
31
I'm having an issue with the same error message as reported here , but we're using the DNS-validation plugin. This week we also had an issue with acme.sh, but this is fixed in the development branch, see the Issue on gitlab.


Does anyone have an idea where this might fail?


Bash:
Jun 15 05:18:08 mailgw-01 systemd[1]: Starting Daily Proxmox Mail Gateway activities...
Jun 15 05:18:09 mailgw-01 pmg-daily[3123776]: cleanup removed 647 entries from statistic database
Jun 15 05:18:10 mailgw-01 pmg-daily[3123776]: starting task UPID:mailgw-01:002FAA51:082A8E40:62A94F71:aptupdate::root@pam:
Jun 15 05:18:12 mailgw-01 pmg-daily[3123793]: update new package list: /var/lib/pmg/pkgupdates
Jun 15 05:18:17 mailgw-01 pmg-daily[3123776]: end task UPID:mailgw-01:002FAA51:082A8E40:62A94F71:aptupdate::root@pam: OK
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: Custom 'api' certificate does not expire soon, skipping ACME renewal.
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: starting task UPID:mailgw-01:002FABE5:082A94A7:62A94F82:acmerenew::root@pam:
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: Loading ACME account details
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: Placing ACME order
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: Order URL: https://acme-v02.api.letsencrypt.org/acme/order/391969880/96650890216
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/118334122216'
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: The validation for mailgw-01.domain.tld is pending!
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022] +---------------------------------------------+
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022] | Adding DNS TXT entry to your cyon.ch domain |
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022] +---------------------------------------------+
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]   * Full Domain: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]   * TXT Value:   3_UZ3CT4aDbbkL77bDcISgoht59uUwd3pp902jYrkv8
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]   - Logging in...
Jun 15 05:18:28 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:28 CEST 2022]     success
Jun 15 05:18:29 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:29 CEST 2022]
Jun 15 05:18:29 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:29 CEST 2022]   - Changing domain environment...
Jun 15 05:18:30 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:30 CEST 2022]     success
Jun 15 05:18:30 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:30 CEST 2022]
Jun 15 05:18:30 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:30 CEST 2022]   - Adding DNS TXT entry...
Jun 15 05:18:32 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:32 CEST 2022]     success (TXT|_acme-challenge.mailgw-01.domain.tld.|3_UZ3CT4aDbbkL77bDcISgoht59uUwd3pp902jYrkv8)
Jun 15 05:18:32 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:32 CEST 2022]
Jun 15 05:18:32 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:32 CEST 2022]   - Logging out...
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:33 CEST 2022]     success
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:33 CEST 2022]
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: Add TXT record: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: Sleeping 300 seconds to wait for TXT record propagation
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: Triggering validation
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022] +-------------------------------------------------+
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022] | Deleting DNS TXT entry from your cyon.ch domain |
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022] +-------------------------------------------------+
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]   * Full Domain: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]   - Logging in...
Jun 15 05:23:34 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:34 CEST 2022]     success
Jun 15 05:23:35 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:35 CEST 2022]
Jun 15 05:23:35 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:35 CEST 2022]   - Changing domain environment...
Jun 15 05:23:36 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:36 CEST 2022]     success
Jun 15 05:23:36 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:36 CEST 2022]
Jun 15 05:23:36 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:36 CEST 2022]   - Deleting DNS TXT entry...
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]     success (TXT|_acme-challenge.mailgw-01.domain.tld.|3_UZ3CT4aDbbkL77bDcISgoht59uUwd3pp902jYrkv8)
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]     done
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]   - Logging out...
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:40 CEST 2022]     success
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:40 CEST 2022]
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: Remove TXT record: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:23:40 mailgw-01 pmg-daily[3124197]: failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: status code 400
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: status code 400
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: end task UPID:mailgw-01:002FABE5:082A94A7:62A94F82:acmerenew::root@pam: failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: status code 400
Jun 15 05:23:40 mailgw-01 systemd[1]: pmg-daily.service: Succeeded.
Jun 15 05:23:40 mailgw-01 systemd[1]: Finished Daily Proxmox Mail Gateway activities.
Jun 15 05:23:40 mailgw-01 systemd[1]: pmg-daily.service: Consumed 16.834s CPU time.

[/ICODE]
 
I'm having an issue with the same error message as reported here , but we're using the DNS-validation plugin. This week we also had an issue with acme.sh, but this is fixed in the development branch, see the Issue on gitlab.


Does anyone have an idea where this might fail?


Bash:
Jun 15 05:18:08 mailgw-01 systemd[1]: Starting Daily Proxmox Mail Gateway activities...
Jun 15 05:18:09 mailgw-01 pmg-daily[3123776]: cleanup removed 647 entries from statistic database
Jun 15 05:18:10 mailgw-01 pmg-daily[3123776]: starting task UPID:mailgw-01:002FAA51:082A8E40:62A94F71:aptupdate::root@pam:
Jun 15 05:18:12 mailgw-01 pmg-daily[3123793]: update new package list: /var/lib/pmg/pkgupdates
Jun 15 05:18:17 mailgw-01 pmg-daily[3123776]: end task UPID:mailgw-01:002FAA51:082A8E40:62A94F71:aptupdate::root@pam: OK
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: Custom 'api' certificate does not expire soon, skipping ACME renewal.
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: starting task UPID:mailgw-01:002FABE5:082A94A7:62A94F82:acmerenew::root@pam:
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: Loading ACME account details
Jun 15 05:18:26 mailgw-01 pmg-daily[3123776]: Placing ACME order
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: Order URL: https://acme-v02.api.letsencrypt.org/acme/order/391969880/96650890216
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/118334122216'
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: The validation for mailgw-01.domain.tld is pending!
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022] +---------------------------------------------+
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022] | Adding DNS TXT entry to your cyon.ch domain |
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022] +---------------------------------------------+
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]   * Full Domain: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]   * TXT Value:   3_UZ3CT4aDbbkL77bDcISgoht59uUwd3pp902jYrkv8
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]
Jun 15 05:18:27 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:27 CEST 2022]   - Logging in...
Jun 15 05:18:28 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:28 CEST 2022]     success
Jun 15 05:18:29 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:29 CEST 2022]
Jun 15 05:18:29 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:29 CEST 2022]   - Changing domain environment...
Jun 15 05:18:30 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:30 CEST 2022]     success
Jun 15 05:18:30 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:30 CEST 2022]
Jun 15 05:18:30 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:30 CEST 2022]   - Adding DNS TXT entry...
Jun 15 05:18:32 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:32 CEST 2022]     success (TXT|_acme-challenge.mailgw-01.domain.tld.|3_UZ3CT4aDbbkL77bDcISgoht59uUwd3pp902jYrkv8)
Jun 15 05:18:32 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:32 CEST 2022]
Jun 15 05:18:32 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:32 CEST 2022]   - Logging out...
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:33 CEST 2022]     success
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:18:33 CEST 2022]
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: Add TXT record: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:18:33 mailgw-01 pmg-daily[3123776]: Sleeping 300 seconds to wait for TXT record propagation
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: Triggering validation
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022] +-------------------------------------------------+
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022] | Deleting DNS TXT entry from your cyon.ch domain |
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022] +-------------------------------------------------+
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]   * Full Domain: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]
Jun 15 05:23:33 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:33 CEST 2022]   - Logging in...
Jun 15 05:23:34 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:34 CEST 2022]     success
Jun 15 05:23:35 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:35 CEST 2022]
Jun 15 05:23:35 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:35 CEST 2022]   - Changing domain environment...
Jun 15 05:23:36 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:36 CEST 2022]     success
Jun 15 05:23:36 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:36 CEST 2022]
Jun 15 05:23:36 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:36 CEST 2022]   - Deleting DNS TXT entry...
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]     success (TXT|_acme-challenge.mailgw-01.domain.tld.|3_UZ3CT4aDbbkL77bDcISgoht59uUwd3pp902jYrkv8)
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]     done
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]
Jun 15 05:23:39 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:39 CEST 2022]   - Logging out...
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:40 CEST 2022]     success
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: [Wed Jun 15 05:23:40 CEST 2022]
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: Remove TXT record: _acme-challenge.mailgw-01.domain.tld
Jun 15 05:23:40 mailgw-01 pmg-daily[3124197]: failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: status code 400
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: status code 400
Jun 15 05:23:40 mailgw-01 pmg-daily[3123776]: end task UPID:mailgw-01:002FABE5:082A94A7:62A94F82:acmerenew::root@pam: failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/118334122216/UUCSDg: status code 400
Jun 15 05:23:40 mailgw-01 systemd[1]: pmg-daily.service: Succeeded.
Jun 15 05:23:40 mailgw-01 systemd[1]: Finished Daily Proxmox Mail Gateway activities.
Jun 15 05:23:40 mailgw-01 systemd[1]: pmg-daily.service: Consumed 16.834s CPU time.

[/ICODE]
And as an addendum: The same happens on our second mail gateway on another location.
 
Same happened here with the netcup DNS plugin
Bash:
/usr/lib/pmg/bin/pmg-daily
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/469366890/100795733826

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/123353208216'
The validation for uhura.petersen20.de is pending!
Add TXT record: _acme-challenge.uhura.petersen20.de
Sleeping 300 seconds to wait for TXT record propagation
Triggering validation
Remove TXT record: _acme-challenge.uhura.petersen20.de
failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/123353208216/aiuoFA: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/123353208216/aiuoFA: status code 400
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/469366890/100795733826

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/123353208216'
The validation for uhura.petersen20.de is pending!
Add TXT record: _acme-challenge.uhura.petersen20.de
Sleeping 300 seconds to wait for TXT record propagation
Triggering validation
Remove TXT record: _acme-challenge.uhura.petersen20.de
failed to execute POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/123353208216/aiuoFA: http request failed: https://acme-v02.api.letsencrypt.org/acme/chall-v3/123353208216/aiuoFA: status code 400
 
We have a similar case in our enterprise support currently (different dns-plugin) - and in that case the issue could not be reproduced here

My guess currently is, that something is in your path to letsencrypt.org - which changes the request (IDS/IPS/firewall/content-inspection/proxy)

The recent change to the acme-implementation of pmg was switching over from a curl-based http-client to rust's ureq

Maybe your network provider can provide some insight?

I hope this helps!
 
Hi Stoiko,

I don't fully understand your answer. You have a similar case but can't reproduce the issue? What do you mean by "similar case"?


I don't think this is related to the network:

* It happens with different providers
* It works with the acme.sh script
* We do not have anything in place between the server and the internet which alters connections, except a packet filtering firewall (but without any IPS,IDS or DPI features enabled for this connection). There I checked the logs and everything seems normal.


Is there any how a possibility to get more detailed logs?
 
You have a similar case but can't reproduce the issue? What do you mean by "similar case"?
We have one case in our enterprise support portal - where a different user has the same error (POST to letsencrypt.org to validate the challenge fails with 400) the difference is that they use a different dns-plugin neither netcup, nor cyon .
However - when I tried creating a certificate with their plugin and a set of test-credentials I could successfully do so.
Therefore I don't think this is a general bug in PMG.

* It happens with different providers
yes it does - because it's not the DNS-plugins that fail to do their job, but the communication between PMG and letsencrypt.org

* It works with the acme.sh script
yes - as said - acme.sh uses curl (or wget) as http-client - PMG recently switched from a curl-based rust-implementation to one based on rust's ureq.
This is where I think the issue is rooted - and since ureq works in many cases (it does from my workstation here) I think it could be something on the path between your PMG and letsencrypt.org

Is there any how a possibility to get more detailed logs?
not really easily - since this part is in the rust-code it would require you to setup a build-environment to compile the necessary code.


I hope this explains it
 
We have a similar case in our enterprise support currently (different dns-plugin) - and in that case the issue could not be reproduced here

My guess currently is, that something is in your path to letsencrypt.org - which changes the request (IDS/IPS/firewall/content-inspection/proxy)

The recent change to the acme-implementation of pmg was switching over from a curl-based http-client to rust's ureq

Maybe your network provider can provide some insight?

I hope this helps!
I don't think that the provider can help, because exactly the same configuration works with the proxmox backup server: I can renew a certificate with the netcup DNS plugin without any problems from the web gui of the PBS.
So in my opinion there is something broken in the proxmox mail gateway implementation.
 
I don't think that the provider can help, because exactly the same configuration works with the proxmox backup server:
This is something that might even confirm my guess:
* all three Proxmox products (for now, we're working on changing that) use 3 different implementations under the hood:
** PVE uses perl's LWP as http-client
** PBS uses a http-client based on Rust's hyper
** PMG uses a http-client based on Rust's ureq - and it used to use one based on a curl-wrapper in Rust - this change is what I think caused the recent issues. - and since the ureq implementation works in some cases - I do assume it's not a general issue with the ACME implementation

I hope this explains it
 
What will be the next steps now for the PMG now? What could the network provider do as you mentioned?
Regarding PMG - we're considering to add a bit of debug output there - so there is a chance to see what gets returned- I cannot give you a date when this will be done though (it's not too high on our prio list) ...

Regarding the provider - I would suggest you contact the people responsible for the network - and perimeter firewalls in your organisation (or if you are this person - contact your ISP) - and say that you are running into issues with a POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/123353208216/aiuoFA

which only happens when done with a ureq-based http-client - but not with one based on hyper - maybe they have an idea where it's going wrong.

I hope this helps!
 
Hi all,

Have the same issue using the prod ACME directory, however it's working just fine using the ACME Staging directory. Any idea why this is happening. Thanks.
 
Regarding PMG - we're considering to add a bit of debug output there - so there is a chance to see what gets returned- I cannot give you a date when this will be done though (it's not too high on our prio list) ...

Regarding the provider - I would suggest you contact the people responsible for the network - and perimeter firewalls in your organisation (or if you are this person - contact your ISP) - and say that you are running into issues with a POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/123353208216/aiuoFA

which only happens when done with a ureq-based http-client - but not with one based on hyper - maybe they have an idea where it's going wrong.

I hope this helps!
I ask my provider for support, but there is nothing he can do.
Perhaps any news from debugging?
 
This is something that might even confirm my guess:
* all three Proxmox products (for now, we're working on changing that) use 3 different implementations under the hood:
** PVE uses perl's LWP as http-client
** PBS uses a http-client based on Rust's hyper
** PMG uses a http-client based on Rust's ureq - and it used to use one based on a curl-wrapper in Rust - this change is what I think caused the recent issues. - and since the ureq implementation works in some cases - I do assume it's not a general issue with the ACME implementation

I hope this explains it
Hi @Stoiko Ivanov - any idea why this fails only on Prod Let's Encrypt and not on Staging? I would expect it to fail on both if a network/firewall issue.


What I noticed is that on staging it detects that the status is still pending

Code:
Sleeping 120 seconds to wait for TXT record propagation
Triggering validation
Sleeping for 5 seconds
Status is still 'pending', trying again in 10 seconds
Status is 'valid', domain 'xxx' OK!


On prod it fails immediately after Triggering validation an it goes to teardown.


Thanks.
 
Hello there. We are facing the same problem. Could someone please test, if this is time related issue? We originally set the Validation Delay in the DNS plugin (ACTIVE24) to 300 seconds. With this setup we get a 400 status error every time. After reducing this timeout to 160 seconds (minimum possible delay for our DNS provider), we are able to order the certificate successfully.

Thank you.
 
  • Like
Reactions: agdsign and Alain_
Hello there. We are facing the same problem. Could someone please test, if this is time related issue? We originally set the Validation Delay in the DNS plugin (ACTIVE24) to 300 seconds. With this setup we get a 400 status error every time. After reducing this timeout to 160 seconds (minimum possible delay for our DNS provider), we are able to order the certificate successfully.

Thank you.
Hey radim.smehlik

Yes, I could reproduce this behavior. We had 300 seconds as validation delay, too. I changed it now to 160 seconds as you did and was also able to order the certificate. Thank you very much!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!