[SOLVED] Network connection fails on api.letsencrypt.org

[ScrewItFix]

Hi Members,

I'm a System Administrator (Linux) and use Proxmox for private testing and development.
After my last "apt upgrade" I tried to renew my letsencrypt certificates and got a network connection error for the subdomains of letsencrypt.org.

I'm not using letsencrypt with Proxmox directly, I have on container running HAproxy and that container is handling the letsencrypt certificates.

Test on the host system (Proxmox VE 5.4-13) :

>> route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 host public IP 0.0.0.0 UG 0 0 0 ens3
host public IP 0.0.0.0 255.255.255.0 U 0 0 0 ens3
172.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 vmbr0

>> traceroute acme-v01.api.letsencrypt.org
traceroute to acme-v01.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 3050.228 ms !H 3050.044 ms !H 3050.018 ms !H


>> traceroute api.letsencrypt.org
traceroute to api.letsencrypt.org (host public IP), 30 hops max, 60 byte packets
1 svr01.2-4-h.net (host public IP) 0.080 ms 0.028 ms 0.026 ms

>> traceroute letsencrypt.org
traceroute to letsencrypt.org (167.99.129.42), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 edge3.ffm2.php-friends.de (176.96.136.1) 0.290 ms 0.281 ms 0.264 ms
4 fra1-edge1.digitalocean.com (80.81.193.141) 0.492 ms 0.460 ms 0.423 ms
5 * * *
6 167.99.129.42 (167.99.129.42) 4.013 ms !X 3.960 ms !X 3.959 ms !X

Tests on the container (HAproxy 1.8.8-1ubuntu0.9)

>> route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.17.0.1 0.0.0.0 UG 0 0 0 eth0
172.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0

>> traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 gtw.app (172.17.0.2) 3060.695 ms !H 3060.563 ms !H 3060.538 ms !H

>> traceroute api.letsencrypt.org
traceroute to api.letsencrypt.org (host public IP), 30 hops max, 60 byte packets
1 2-4-h.com (host public IP) 0.101 ms 0.029 ms 0.032 ms

>> traceroute letsencrypt.org
traceroute to letsencrypt.org (142.93.108.123), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 0.056 ms 0.019 ms 0.010 ms
2 2.59.132.1 (2.59.132.1) 17.943 ms 17.923 ms 17.900 ms
3 176.96.136.81 (176.96.136.81) 17.850 ms 17.807 ms 12.167 ms
4 edge3.ffm2.php-friends.de (176.96.136.1) 0.869 ms 0.807 ms 0.216 ms
5 fra1-edge1.digitalocean.com (80.81.193.141) 5.498 ms 5.461 ms 5.426 ms
6 138.197.250.173 (138.197.250.173) 0.706 ms 0.550 ms 0.854 ms
7 142.93.108.123 (142.93.108.123) 0.892 ms !X 0.830 ms !X 0.849 ms !X

The subdomain's are somehow blocked. The only thing that I can think of is some block within Proxmox.

Can any of you help me with his issue ?

 

[t.lamprecht]

traceroute acme-v01.api.letsencrypt.org

Slightly off-topic FYI: the v1 API has been discontinued, new registrations are not possible since November 2019 and renewals will start to fail occasionally in 2020. https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430

Anyway, Proxmox VE does not blocks those, we even have integration to directly use it so that does really not makes sense for us to have by default.
You probably already checked, so just to be sure: Do you have firewalling activated? Quickest way to see if any rules are active would be iptables-save

Are you in a hosted enviroment?

 

[ScrewItFix]

Thank you for the update and reply. I was not aware about the end of life for the V1 API, however the issue occurs on the V2 API endpoint in the same way with the same results.

Are you in a hosted service?

Yes - I checked with my Hosting Provider (PHP-Friends) and they replied that the block must be on "my side", most likely my bridge configuration, and they were half right!

My internal network uses the IP address range of 172.17.0.0/8 and this configuration was working for more than 12 months without any problem, so why that stopped working? As most times, the solution was staring in my face:
Traceroute to acme-v02.api.letsencrypt.org actually pointed to 172.65.32.248 -- the same realm (172.0.0.0) that I used for my internal network and the netmask 255.0.0.0 caused that this 172.65.32.248 was treated as a local IP.
I guess that LetsEncrypt changed their IP address for their API endpoints recently.

In order to fix the issue I only needed to change my internal network to the 10.0.0.0 range

Danke Thomas für deine Antwort.