PVE 5.2 Lets Encrypt: TASK ERROR: validating challenge failed

[Gunnar]

I was trying to figure out the new letsencrypt integration, and couldn't get it to work...

Then i noticed this:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
task\x20U 9499 root 3u IPv4 76078 0t0 TCP *:80 (LISTEN)
task\x20U 9504 root 3u IPv4 76078 0t0 TCP *:80 (LISTEN)

So that's not going to work for me, proxmox is on an IPv6 only network and that's not changing, so i need that task to listen on both...

 

[fabian]

I was trying to figure out the new letsencrypt integration, and couldn't get it to work...

Then i noticed this:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
task\x20U 9499 root 3u IPv4 76078 0t0 TCP *:80 (LISTEN)
task\x20U 9504 root 3u IPv4 76078 0t0 TCP *:80 (LISTEN)

So that's not going to work for me, proxmox is on an IPv6 only network and that's not changing, so i need that task to listen on both...

please file a bug on https://bugzilla.proxmox.com for libpve-common-perl. thank you!

 

[dotsam]

I see there's a bug open for this https://bugzilla.proxmox.com/show_bug.cgi?id=2061 and that there are even some recent commits that look to be getting this implemented, but I needed to get this working now, so as a workaround, I set up a quick socat listener to forward connections from IPv6 to IPv4

/usr/bin/socat TCP6-LISTEN:80,fork,ipv6only=1,reuseaddr TCP4:127.0.0.1:80

 

[fabian]

see http://download.proxmox.com/debian/...ry-amd64/libhttp-daemon-perl_6.01-2.changelog

 

[dotsam]

see http://download.proxmox.com/debian/...ry-amd64/libhttp-daemon-perl_6.01-2.changelog

Strange, I have that version installed:

root@imbrem:~# apt-cache policy libhttp-daemon-perl
libhttp-daemon-perl:
  Installed: 6.01-2
  Candidate: 6.01-2
  Version table:
 *** 6.01-2 500
        500 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
        100 /var/lib/dpkg/status

Yet when I was attempting the certificate order yesterday, I only saw a listener opening on 0.0.0.0:80, not [::]:80, and the verification failed. Using the socat redirect worked right away though, so there were no other issues with the firewall or anything else.

Does the HTTP server constructor in PVE/ACME/StandAlone.pm need to be changed to ensure it's listening on IPv6 as well perhaps?

 

[Ronny Aasen]

I see the same issue as dotsam.

the socat trick works, but i assume it will fail on renewals ?

 

[dotsam]

I see the same issue as dotsam.

the socat trick works, but i assume it will fail on renewals ?

I just made a simple systemd unit to keep socat running:

cat /etc/systemd/system/pve-ipv6-cert-proxy.service
[Unit]
Description=Forward IPv6 port 80 to proxomox acme validation server

[Service]
ExecStart=/usr/bin/socat TCP6-LISTEN:80,fork,ipv6only=1,reuseaddr TCP4:127.0.0.1:80

[Install]
WantedBy=multi-user.target

 

[Ronny Aasen]

I just made a simple systemd unit to keep socat running:

cat /etc/systemd/system/pve-ipv6-cert-proxy.service
[Unit]
Description=Forward IPv6 port 80 to proxomox acme validation server

[Service]
ExecStart=/usr/bin/socat TCP6-LISTEN:80,fork,ipv6only=1,reuseaddr TCP4:127.0.0.1:80

[Install]
WantedBy=multi-user.target

thank you !

it is a crutch until they fix the bug

Ronny

 

[fabian]

could you re-test with http://download.proxmox.com/debian/...ry-amd64/libhttp-daemon-perl_6.01-2.1_all.deb ?

SHA256: dfcc4b4d03857c10c2f2dbef42ffd2921d9c478207d2fd8f6f0fbddfa0f7d939

 

[Ronny Aasen]

could you re-test with http://download.proxmox.com/debian/...ry-amd64/libhttp-daemon-perl_6.01-2.1_all.deb ?
SHA256: dfcc4b4d03857c10c2f2dbef42ffd2921d9c478207d2fd8f6f0fbddfa0f7d939

seems to fail on pve 5.4

pveversion
pve-manager/5.4-11/6df3d8d0 (running kernel: 4.15.18-18-pve)

root@node-b:~# sha256sum libhttp-daemon-perl_6.01-2.1_all.deb
dfcc4b4d03857c10c2f2dbef42ffd2921d9c478207d2fd8f6f0fbddfa0f7d939 libhttp-daemon-perl_6.01-2.1_all.deb

root@node-b:~# dpkg -i libhttp-daemon-perl_6.01-2.1_all.deb
(Reading database ... 46732 files and directories currently installed.)
Preparing to unpack libhttp-daemon-perl_6.01-2.1_all.deb ...
Unpacking libhttp-daemon-perl (6.01-2.1) over (6.01-2.1) ...
Setting up libhttp-daemon-perl (6.01-2.1) ...
Processing triggers for man-db (2.7.6.1-2) ...

root@node-b:~# pvenode acme cert order --force
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/61564363/784090232

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz/nlCtHL-W5hCbDcVk17JsgD1lcgtUJVkNTaqPTvfQILg'
... pending!
Setting up webserver
Triggering validation
Sleeping for 5 seconds
validating challenge 'https://acme-v02.api.letsencrypt.org/acme/authz/nlCtHL-W5hCbDcVk17JsgD1lcgtUJVkNTaqPTvfQILg' failed
Task validating challenge 'https://acme-v02.api.letsencrypt.org/acme/authz/nlCtHL-W5hCbDcVk17JsgD1lcgtUJVkNTaqPTvfQILg' failed

same error in the web user interface.

when running netstat in a different session show it is only listening on 0.0.0.0:80

kind regards
Ronny Aasen

 

[fabian]

could you include your /etc/hosts and /etc/gai.conf ?

 

[dotsam]

Just to bump this thread, I upgraded from Proxmox 5 to Proxmox 6 and this issue still persists. I still need the socat workaround to get the internal certificate generator to work.

My /etc/gai.conf is stock, with everything commented out.

My /etc/hosts looks like this, with relevant info redacted:

127.0.0.1 localhost.localdomain localhost
2001:470:x:xxx::xxx HOSTNAME.fqdn.tld HOSTNAME HOSTNAMElocalhost
10.0.5.xxx HOSTNAME.fqdn.tld HOSTNAME HOSTNAMElocalhost

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts