webhook notification to non-public target (self-signed root CA)

[morik_proxmox]

Hello,
Thanks to the PVE developers for introducing webhook notifications. I was wondering if there's a way to post to a non-public target. By that I mean, a target for which the certificate is self-issued and not via well-known provider. My webhook notifications are implemented on a self-hosted web server with a self-signed root CA on a made-up domain which is valid only in my LAN. Other similar webhook notification facilities provide such a facility. The curl equivalent, I believe, would be usage of

(curl -k) OR (curl --insecure)

I'm running pve-manager/8.3.2/3e76eec21c4a14a7 (running kernel: 6.8.12-5-pve). I tried adding few fields to /etc/pve/notifications.cfg like mode inseucr and such, but it didn't work

 

[wahmed]

If I am understanding correctly what you are trying to achieve, you can configure Proxmox webhook notifications to post to a non-public target with a self-signed certificate by ensuring that PVE recognizes your self-signed certificate as trusted.
1. Add the self-signed cert to PVE trusted store:
Copy the self-signed root cert to a directory on your PVE node:

scp rootCA.crt root@pve_node:/etc/ssl/certs/

2. Update system's CA cert

update-ca-certificates

3. test connection:

curl -v https://yourdomain.com -k

If you see SSL certificate problem: unable to get local issuer certificate, ensure the CA was added correctly and restart any services using SSL (systemctl restart pveproxy).
If it connects successfully, PVE should work as expected.

4. Configure the webhook pointing to your LAN target.

You could avoid all the extra configuration by adding --insecure flag to disable SSL verification. But that is not recommended at all from security perspective.

 

[morik_proxmox]

@wahmed
Thank you for much for your guidance.

If I am understanding correctly what you are trying to achieve, you can configure Proxmox webhook notifications to post to a non-public target with a self-signed certificate by ensuring that PVE recognizes your self-signed certificate as trusted.

Yes. As the communication is within my heavily locked down VLAN, I was hoping to take the short-cut by allowing `insecure` flag (for web notification).

I did follow your recommendations in steps 1-4 above.

/usr/local/share/ca-certificates# update-ca-certificates
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
2 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

/usr/local/share/ca-certificates# ssrest pveproxy
root@pve-dell-xr12-2:/usr/local/share/ca-certificates# sstatus pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Sun 2024-12-29 17:26:06 PST; 2min 13s ago
    Process: 1486342 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 1486347 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 1486350 (pveproxy)
      Tasks: 4 (limit: 629145)
     Memory: 176.2M
        CPU: 1.651s
     CGroup: /system.slice/pveproxy.service
             ├─1486350 pveproxy
             ├─1486351 "pveproxy worker"
             ├─1486352 "pveproxy worker"
             └─1486353 "pveproxy worker"

Dec 29 17:26:05 pve-dell-xr12-2 systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
Dec 29 17:26:06 pve-dell-xr12-2 pveproxy[1486350]: starting server
Dec 29 17:26:06 pve-dell-xr12-2 pveproxy[1486350]: starting 3 worker(s)
Dec 29 17:26:06 pve-dell-xr12-2 pveproxy[1486350]: worker 1486351 started
Dec 29 17:26:06 pve-dell-xr12-2 pveproxy[1486350]: worker 1486352 started
Dec 29 17:26:06 pve-dell-xr12-2 pveproxy[1486350]: worker 1486353 started
Dec 29 17:26:06 pve-dell-xr12-2 systemd[1]: Started pveproxy.service - PVE API Proxy Server.

/usr/local/share/ca-certificates# curl -v https://homeassistant.esco.ghaar
*   Trying 192.168.0.58:443...
* Connected to homeassistant.esco.ghaar (192.168.0.58) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=homeassistant.esco.ghaar
*  start date: Dec 27 01:34:09 2023 GMT
*  expire date: Nov 29 01:35:09 2033 GMT
*  subjectAltName: host "homeassistant.esco.ghaar" matched cert's "homeassistant.esco.ghaar"
*  issuer: O=Escoghaar; CN=Escoghaar Intermediate CA
*  SSL certificate verify ok.
* using HTTP/2

The above were performed on both nodes in the cluster. Web notification configuration is:

root@pve-dell-xr12-2:/usr/local/share/ca-certificates# cat /etc/pve/notifications.cfg
webhook: esco-ha-webhook
    body ewogICJib2R5IjogInt7IGVzY2FwZSBtZXNzYWdlIH19Igp9
    disable true
    header name=Content-Type,value=YXBwbGljYXRpb24vanNvbg==
    method post
    url https://homeassistant.esco.ghaar/api/webhook/webhooks-ZLFIK4W_QNG2CfiUeIkwPUAK

But testing the notification fails with a 500.

Could not test target: https://homeassistant.esco.ghaar/api/webhook/webhooks-ZLFIK4W_QNG2CfiUeIkwPUAK: Connection Failed: tls connection init failed: invalid peer certificate: UnknownIssuer (500)

But, it works just fine whilst posting to webhook via curl command

root@pve-dell-xr12-2:/usr/local/share/ca-certificates# curl -vi -H "Content-Type: application/json" \
> -d '{"message": "test message"}' \
> https://homeassistant.esco.ghaar/api/webhook/webhooks-ZLFIK4W_QNG2CfiUeIkwPUAK
*   Trying 192.168.0.58:443...
* Connected to homeassistant.esco.ghaar (192.168.0.58) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=homeassistant.esco.ghaar
*  start date: Dec 27 01:34:09 2023 GMT
*  expire date: Nov 29 01:35:09 2033 GMT
*  subjectAltName: host "homeassistant.esco.ghaar" matched cert's "homeassistant.esco.ghaar"
*  issuer: O=Escoghaar; CN=Escoghaar Intermediate CA
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: POST]
* h2h3 [:path: /api/webhook/webhooks-ZLFIK4W_QNG2CfiUeIkwPUAK]
* h2h3 [:scheme: https]
* h2h3 [:authority: homeassistant.esco.ghaar]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 27]
* Using Stream ID: 1 (easy handle 0x5d6d1aeabce0)
> POST /api/webhook/webhooks-ZLFIK4W_QNG2CfiUeIkwPUAK HTTP/2
> Host: homeassistant.esco.ghaar
> user-agent: curl/7.88.1
> accept: */*
> content-type: application/json
> content-length: 27
>
* We are completely uploaded and fine
< HTTP/2 200
HTTP/2 200
< server: nginx
server: nginx
< date: Mon, 30 Dec 2024 01:39:01 GMT
date: Mon, 30 Dec 2024 01:39:01 GMT
< content-length: 0
content-length: 0
< referrer-policy: no-referrer
referrer-policy: no-referrer
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000; includeSubDomains
strict-transport-security: max-age=31536000; includeSubDomains

<
* Connection #0 to host homeassistant.esco.ghaar left intact

Any further guidance would be much appreciated.