ACME certificate HTTP method not opening port 80 for IPv4

[Ian Warburton]

I am setting up a new server for automatic certificate renewal but the process only starts the HTTP listener on an IPV6 address, I need this to be on a V4 address
There is no service running on port 80.
version proxmox 8.03
netstat -tulpn | grep LISTEN
tcp 0 0 127.0.0.1:85 0.0.0.0:* LISTEN 1398/pvedaemon
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3693793/master
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3695231/sshd: /usr/
tcp6 0 0 ::1:25 :::* LISTEN 3693793/master
tcp6 0 0 :::8006 :::* LISTEN 1407/pveproxy
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::80 :::* LISTEN 3763419/task UPID:e
tcp6 0 0 :::22 :::* LISTEN 3695231/sshd: /usr/
tcp6 0 0 :::3128 :::* LISTEN 1414/spiceproxy

Does anyone know how to configure the service to use ipv4 ?

 

[Superfish1000]

I have been having exactly the same issue on one of my servers for a bit now. Just had the time to figure out what was causing it.
Did you ever end up getting this resolved?

The other thread I found talking about this just ended how many of them seem to, descending into numerous requests for logs that lost sight of the issue and then the OP just nuking and reinstalling.

 

[fabian]

which PVE versions are you using? by default, the standalone plugin will bind to "::" without IPV6Only (which might be displayed as ipv6 depending on which tool you are using to check it out), or if that fails, to "0.0.0.0".

 

[Ian Warburton]

Hi No didn't find a solution and have had other things to sort out - I'm on 8.0. 4 - It would be nice to sort this out - @fabian I am assuming you mean the acme plugin ?

 

[Ian Warburton]

OK there are no plugins configures so it must be using the default so where is this configured ?

 

[fabian]

the standalone challenge handler doesn't require any plugins. the source is here: https://git.proxmox.com/?p=proxmox-...4c5a9e003b3bb43e9b7eca9769d91dad779f3;hb=HEAD

 

[Ian Warburton]

It would appear that line 42 my $server = HTTP:daemon->new( LocalHost => '::', V6Only => 0, %sockopts) //
is not operating as anticipated - V6Only => 0 does not seem to be working as intended . I will investigate further.

 

[Ian Warburton]

OK this script does work however it's not ideal - the full line is
my $server = HTTP:daemon->new( LocalHost => '::', V6Only => 1, %sockopts) // HTTP:daemon->new( LocalHost => '0.0.0.0', %sockopts)
so if the first part is successful then the second part is not executed. If the host is configured to use IPV6 then port 80 will never appear on the V4 address.
It is possible that a host may be using IPv6 but also require the ACME service to operate on a V4 port - currently the code does not support this condition.

Short term options are to

1 modify the code, this should work
my $server = HTTP:daemon->new( LocalHost => '0.0.0.0', %sockopts)
or
2 disable IPV6
In ubuntu 23.10 this had the desired outcome
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1

A longer term option would be to allow this to be a configurable for the ACME service

 

[Superfish1000]

I haven't had a lot of time to mess with this, but my situation is that I have 4 identical servers and one that is running different physical hardware, but is otherwise configured identical that are all working with the exception of this one node.

I've dug through the logs for a few hours now but nothing I have found has been remotely useful, so I don't really know where it's dying.

A snippet of the error I'm getting from LetsEncrypt is as follows.

"status": "invalid",
"expires": "2023-12-17T09:56:24Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietfarams:acme:error:connection",
"detail": "XXX.XXX.XXX.XXX: Fetching http://node-1.xxx.com/.well-known/acme-challenge/XXXXXXXXXXXXXXXXXXXX: Timeout during connect (likely firewall problem)",
"status": 400
},

I can confirm that there is not a firewall problem as I have fully disabled it a few times while testing and got the same result.
Additionally, the other 4 working servers all have the same firewall configuration.

I have tried reinstalling the acme components, though I may have done it wrong, and I have tried to regenerate the node certificates.

I've also removed and re-added the account as well as setting up a staging account since I have hit rate limiting several times trying to diagnose the issue.
At this point I'm really out of ideas where to go next.

I did notice that while the server should be running I can also open HTTP to the server and will be greeted with a 404 Not Found page, so it seems like HTTP is working anyway.

I guess my question at this point is is there another log that I can look at to get some indication what is wrong?

UPDATE: I now feel very stupid, but I just realized that somehow one of my secondary interfaces on the node that was having issues was set up with a NAT gateway IP.
I'm not really sure why this caused this failure, but I noticed this because I tried to access the servers web control panel from the internet instead of over the LAN and wasn't able to get the same HTTP 404 error.

Out of curiosity, the second gateway shouldn't have existed to begin with but is there a reason why this still failed when the server should have been accessible? It seems like it was only accessible on the LAN interface for some reason.

 

[fabian]

there can only ever be one gateway (well, you can write more than one into the config, but it's not valid ). I am not sure where exactly your "wrong" setup went wrong.. by default, the standlone challenge responder should bind to all local interfaces both on IPv4 and IPv6