[SOLVED] ACME issue with Openbao ACME issuer

[kellen]

Hello! I am attempting to use an Openbao (fork of Vault) ACME issuer with PVE 8.2. When I try to register the ACME account on PVE I get this error:

Generating ACME account key..
Registering ACME account..
TASK ERROR: Registration failed: Error: GET to http://openbao.example.com/v1/tls/acme/new-nonce

It doesn't tell me what went wrong. Other non-PVE certbot clients are able to use the Openbao ACME without issue. Does anyone know where I might be able to get more detailed debug information?

Thanks,
Kellen

 

[billy999]

Proxmox should be printing the status code and response body of the request as well, not sure why its missing here. Did you redact the URL or is that the actual log you received?

 

[kellen]

Aside from redacting the URL to "example.com" that is the entire log I receive. Good to know there should be more, it does seem a bit cut off.

 

[cipherboy]

\o hey kellen -- if you enable raw audit logs, do you see any difference between the request proxmox sends and any other ACME client?

Something like:

$ bao audit enable file file_path=/tmp/acme-log log_raw=true # note that this is unsafe for general usage

IIRC, one of the differences in ACME clients was whether or not they'd use the directory's nonce or whether they'd explicitly fetch their own nonce... It could be that proxmox isn't using the nonce returned on the directory and we have a bug in that endpoint.

 

[kellen]

@cipherboy Thanks for the debug command! It revealed something interesting!

A request from PVE comes in for tls/acme/directory, Openbao replies with a HTTP 200. Nothing further.

A request from Certbot comes in for tls/acme/directory, Openbao replies with a HTTP 200. Certbot then requests tls/acme/new-nonce and the issuance proceeds to a successful conclusion.

Here is the PVE convo:

{"time":"2024-10-18T19:55:14.4596296Z","type":"request","auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"id":"2591986f-502e-e73b-144c-0455923b07c0","operation":"read","mount_point":"tls/","mount_type":"pki","mount_running_version":"v2.0.2+builtin.bao","mount_class":"secret","namespace":{"id":"root"},"path":"tls/acme/directory","remote_address":"10.0.20.38","remote_port":56220}}

{"time":"2024-10-18T19:55:14.459965416Z","type":"response","auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"id":"2591986f-502e-e73b-144c-0455923b07c0","operation":"read","mount_point":"tls/","mount_type":"pki","mount_accessor":"pki_0edd96c9","mount_running_version":"v2.0.2+builtin.bao","mount_class":"secret","namespace":{"id":"root"},"path":"tls/acme/directory","remote_address":"10.0.20.38","remote_port":56220},"response":{"mount_point":"tls/","mount_type":"pki","mount_accessor":"pki_0edd96c9","mount_running_plugin_version":"v2.0.2+builtin.bao","mount_class":"secret","data":{"http_content_type":"application/json","http_raw_body":"eyJrZXlDaGFuZ2UiOiJodHRwOi8vb3BlbmJhby4wLnR1cy51cy5ibHVlcXVhcnR6Lnh5ei92MS90bHMvYWNtZS9rZXktY2hhbmdlIiwibWV0YSI6eyJleHRlcm5hbEFjY291bnRSZXF1aXJlZCI6ZmFsc2V9LCJuZXdBY2NvdW50IjoiaHR0cDovL29wZW5iYW8uMC50dXMudXMuYmx1ZXF1YXJ0ei54eXovdjEvdGxzL2FjbWUvbmV3LWFjY291bnQiLCJuZXdOb25jZSI6Imh0dHA6Ly9vcGVuYmFvLjAudHVzLnVzLmJsdWVxdWFydHoueHl6L3YxL3Rscy9hY21lL25ldy1ub25jZSIsIm5ld09yZGVyIjoiaHR0cDovL29wZW5iYW8uMC50dXMudXMuYmx1ZXF1YXJ0ei54eXovdjEvdGxzL2FjbWUvbmV3LW9yZGVyIiwicmV2b2tlQ2VydCI6Imh0dHA6Ly9vcGVuYmFvLjAudHVzLnVzLmJsdWVxdWFydHoueHl6L3YxL3Rscy9hY21lL3Jldm9rZS1jZXJ0In0=","http_status_code":200}}}

And the Certbot convo, up to the request for tls/acme/new-nonce:

{"time":"2024-10-18T20:00:06.240419817Z","type":"request","auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"id":"2f7a31ba-ed8c-95db-1a92-e6cc8335affa","operation":"read","mount_point":"tls/","mount_type":"pki","mount_running_version":"v2.0.2+builtin.bao","mount_class":"secret","namespace":{"id":"root"},"path":"tls/acme/directory","remote_address":"10.0.21.253","remote_port":51052}}

{"time":"2024-10-18T20:00:06.240796275Z","type":"response","auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"id":"2f7a31ba-ed8c-95db-1a92-e6cc8335affa","operation":"read","mount_point":"tls/","mount_type":"pki","mount_accessor":"pki_0edd96c9","mount_running_version":"v2.0.2+builtin.bao","mount_class":"secret","namespace":{"id":"root"},"path":"tls/acme/directory","remote_address":"10.0.21.253","remote_port":51052},"response":{"mount_point":"tls/","mount_type":"pki","mount_accessor":"pki_0edd96c9","mount_running_plugin_version":"v2.0.2+builtin.bao","mount_class":"secret","data":{"http_content_type":"application/json","http_raw_body":"eyJrZXlDaGFuZ2UiOiJodHRwOi8vb3BlbmJhby4wLnR1cy51cy5ibHVlcXVhcnR6Lnh5ei92MS90bHMvYWNtZS9rZXktY2hhbmdlIiwibWV0YSI6eyJleHRlcm5hbEFjY291bnRSZXF1aXJlZCI6ZmFsc2V9LCJuZXdBY2NvdW50IjoiaHR0cDovL29wZW5iYW8uMC50dXMudXMuYmx1ZXF1YXJ0ei54eXovdjEvdGxzL2FjbWUvbmV3LWFjY291bnQiLCJuZXdOb25jZSI6Imh0dHA6Ly9vcGVuYmFvLjAudHVzLnVzLmJsdWVxdWFydHoueHl6L3YxL3Rscy9hY21lL25ldy1ub25jZSIsIm5ld09yZGVyIjoiaHR0cDovL29wZW5iYW8uMC50dXMudXMuYmx1ZXF1YXJ0ei54eXovdjEvdGxzL2FjbWUvbmV3LW9yZGVyIiwicmV2b2tlQ2VydCI6Imh0dHA6Ly9vcGVuYmFvLjAudHVzLnVzLmJsdWVxdWFydHoueHl6L3YxL3Rscy9hY21lL3Jldm9rZS1jZXJ0In0=","http_status_code":200}}}

{"time":"2024-10-18T20:00:06.257127808Z","type":"request","auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"id":"a5ae1ba0-0651-61fd-7ea9-f24764834ffe","operation":"header","mount_point":"tls/","mount_type":"pki","mount_running_version":"v2.0.2+builtin.bao","mount_class":"secret","namespace":{"id":"root"},"path":"tls/acme/new-nonce","remote_address":"10.0.20.38","remote_port":57708}}

{"time":"2024-10-18T20:00:06.257465843Z","type":"response","auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"id":"a5ae1ba0-0651-61fd-7ea9-f24764834ffe","operation":"header","mount_point":"tls/","mount_type":"pki","mount_accessor":"pki_0edd96c9","mount_running_version":"v2.0.2+builtin.bao","mount_class":"secret","namespace":{"id":"root"},"path":"tls/acme/new-nonce","remote_address":"10.0.20.38","remote_port":57708},"response":{"mount_point":"tls/","mount_type":"pki","mount_accessor":"pki_0edd96c9","mount_running_plugin_version":"v2.0.2+builtin.bao","mount_class":"secret","data":{"http_content_type":"","http_status_code":200},"headers":{"Link":["\u003chttp://openbao.0.tus.us.bluequartz.xyz/v1/tls/acme/directory\u003e;rel=\"index\""],"Replay-Nonce":["dmF1bHQw0QJt6foy1LgzVJYrly00oJJ824NEpFkwNa8JmSKhmfYh0Vn7ER1cEw"]}}}

This looks to me like PVE is having some trouble with the initial response from Openbao, but I don't have a clue what it could be.

 

[cipherboy]

So it looks like it uses this custom client: https://github.com/proxmox/proxmox-...265f7b48aca0db6d57cfb/proxmox-acme/Cargo.toml

I believe we're calling from https://github.com/proxmox/proxmox-backup/blob/master/src/acme/client.rs#L624 -> https://github.com/proxmox/proxmox-backup/blob/master/src/acme/client.rs#L202-L207 -> https://github.com/proxmox/proxmox-backup/blob/master/src/acme/client.rs#L644-L672 but the interesting hint is this:

> TASK ERROR: Registration failed: Error: GET to http://openbao.example.com/v1/tls/acme/new-nonce

This is weird, because the above code seems to suggest we should be issuing a HEAD request. Notably, while OpenBao does support GET requests, we reply with 204 rather than 200. This is in line with the RFC 8555.

I'd almost bet that something is expecting a non-204 (edit: expecting a 200 OK) response to GET...?

That said, this other part of the client https://github.com/proxmox/proxmox-...b6d57cfb/proxmox-acme/src/client.rs#L232-L235 seems to also be alright and uses HEAD not GET... So I'm rather confused where this GET request is coming from!

(I did check, and Let's Encrypt also returns a directory without a Nonce header, so I don't think this is a difference there.)

 

[cipherboy]

One other thought I had: what request and response headers have you enabled? (https://openbao.org/api-docs/secret/pki/#acme-required-headers) -- I almost wonder if Certbot can work around a missing header that the Proxmox client can't...?

 

[kellen]

Hmmm, I made sure those are enabled on the Openbao mount, and retested. No joy there.

I have noticed that Openbao is returning HTTP URLs in the body of the directory call. Which is a bit weird since I think it should all be HTTPS.

In my setup I have a reverse proxy in front of openbao, which performs a HTTPS call to Openbao:
client --HTTPS--> RP:443 ---HTTPS--> Openbao:8200

The reverse proxy is sending HTTP 308 Redirects on port 80. Didn't work when I turned that off.

Attempting to use the HTTP directory URL causes a 500 error similar to the one this is about during registration.

Should Openbao be sending HTTP URLs?

 

[kellen]

To answer my question: No, it should not.

The registration works now, I had to correct the Openbao configuration for the cluster to use HTTPS URLs instead of HTTP ones. I guess that was leftover from my initial experiments and Certbot was willing to accept the redirects.

Thanks @cipherboy and @billy999 for helping me out!