Metrics server - Error 404

M

mii

New Member
Proxmox Subscriber
Sep 30, 2024
11
2
3
I am trying to Proxmox to report Metrics via InfluxDB.

I have an LXC container, on the Proxmox machine, running Telegraf acting as an InfluxDB listener on:
Code: 
https://influx.domain.com:8186/
. (the V1 API would also be available).

This works fine, as I can send to it from my PC, and can contact it from Proxmox using either the hostname, or the IP address (without certificate verification).

From Proxmox:
Code: 
curl -i -XPOST  'https://influx.domain.com:8186/api/v2/write' --data-binary 'cpu_load_short,host=s
erver01,region=us-west value=1.13' --header 'Authorization: Token TokenGoesHereInClearText'
HTTP/1.1 204 No Content
Date: Tue, 22 Oct 2024 11:26:09 GMT

What my configuration looks like:
Code: 
root@pma:~# cat /etc/pve/status.cfg
influxdb: Influx
        disable
        port 8186
        server influx.domain.com
        influxdbproto https

Whenever I try to enable it in the GUI, I get this error:
Code: 
404 (Not found)
.

It is contacting the server though - I can, on the server, get the following in the Telegraf logs when the host is contacted via IP, but enabled certificate verification:
Code: 
2024-10-22T11:46:37Z I! http: TLS handshake error from 10.xx.x.20:39330: local error: tls: bad record MAC

I have tried with various combinations of bucket / organization, which should not be necessary. Other hosts can send to this endpoint without problems. I can disable the certificate verification, but this does not help.

I have put various variations / parts of
Code: 
/api/v2/write
in API path prefix, without success - I keep getting a 404.

Where can I get more information on what Proxmox is doing? The Telegraf instance isn't showing anything useful..
 
I found the issue with sending metrics to telegraf to be that Proxmox was requesting a /health endpoint to check for InfluxDB, and would fail with a 404 error if it couldn't find it. I ended up setting up a little nginx reverse proxy in front of telegraf that would return 200 for /health to work around the issue. Honestly though I think the metrics from pve-exporter are better, and plan on using that moving forward.

Code: 
server {
        listen 8087 default_server;
        listen [::]:8087 default_server;
        root /var/www/html;
        index index.html index.htm index.nginx-debian.html;
        location /health {
                return 200;
        }

        location / {
                proxy_pass http://127.0.0.1:8086;
        }
}
 
  • Like
Reactions: mii
Just ran into this as well. Confirmed using a quick network capture, indeed proxmox tries the /health endpoint first, which telegraf in my case responds to with 404.
I am using a telegraf influx input plugin to ingest telemetry and then forward through kafka to influx. The telegraf influxv2 input plugin I am using for this does not expose a /health endpoint.
Any way to turn off healthcheck? Can I just change the config file to enable telemetry sending?
 
Last edited:
@mcfly9 thanks for the input.

The v2 API specifies a `/health` endpoint (https://docs.influxdata.com/influxdb/v2/api/#operation/GetHealth).

However, the v2 endpoint does not serve this, see https://github.com/influxdata/teleg...uxdb_v2_listener/influxdb_v2_listener.go#L224

There is another one served, however: `/api/v2/ready`, which is also a kind of healthcheck.

So:
- Proxmox could disable the healthcheck
- Proxmox could use the ready endpoint
- Telegraf could add the /health endpoint, as specified in the docs..

At the moment, I do not see any other workaround than the method suggested by @wavyline, but I don't really want to put more software in front of my metrics endpoint..
 
You must log in or register to reply here.
Top Bottom