[SOLVED] Proxmox prometheus exporter questions

[jfmanamtr]

I'm not sure if this is the right place to ask this or not. I have a 2 node proxmox cluster. I'm running Prometheus and Grafana on a vm. I want to use this to monitor the proxmox systems and report to me through slack.

I have the system setup and am trying to get the proxmox exporter exposing those metrics so I can scrape them with the server. When I try to look at the http://ip:9221/pve, I get the following: Couldn't authenticate user: prometheus@pve to https://localhost:8006/api2/json/access/ticket

I think this is something I'm doing wrong as far as the api. But I'm not sure. Has anyone seen anything like this before or can you point me in the right direction?

Thanks

 

[Dominic]

Hi!

Did you verify with /etc/pve/user.cfg if your users and realm (see documentation for details) are really correct? Have you had any success doing the authentication manually (using curl), as explained in the Wiki?

If you don't mind some dependencies, there are tools/guides available that might help. For example this article.

 

[jfmanamtr]

Hi!

Did you verify with /etc/pve/user.cfg if your users and realm (see documentation for details) are really correct? Have you had any success doing the authentication manually (using curl), as explained in the Wiki?

If you don't mind some dependencies, there are tools/guides available that might help. For example this article.

I am at work but will triple check and post here when I get home later. Thanks. I didn't try it manually yet.

 

[jfmanamtr]

ok. so, I think I have it all set right.

root@node-a:~# cat /etc/pve/user.cfg       

user:prometheus@pve:1:0::::::
token:prometheus@pve!ab27.....031e:0:0::
user:root@pam:1:0:::::

when I go to http://ip:9221/pve?target=ip (prometheus exporter), I get the following:

Couldn't authenticate user: prometheus@pve to https://ip:8006/api2/json/access/ticket

then when I go to https://ip:8006/api2/json/access/ticket, I get the following error:

{"data":null}

Any ideas? I will post anything you need to help me. Just let me know. Thanks

 

[Dominic]

then when I go to https://ip:8006/api2/json/access/ticket, I get the following error:

By "I got to", do you mean entering this into the browser? Try using the curl command from the Wiki instead.

 

[jfmanamtr]

Apologies. I forgot to add that part to my last reply.

root@node-a:~# curl -k -d "username=prometheus@pve&password=<redacted>"  https://node-a:8006/api2/json/access/ticket                       

{"data":{"username":"prometheus@pve","cap":{"nodes":{},"access":{},"dc":{},"vms":{},"storage":{}},"ticket":"PVE:prometheus@pve:5F4697F2::tZdQenzHv5dZ2nMEI0fHPdCtj8ZcJPEg6WDA9xPdUb7rwyEPp1nUk5vXzUgU+fkMMpITMH7fIuhJplD9FgY4c/2e+ihw27hAtCajYN/OhLC+hGALaDHv7gpFZz2fBLd0d0CLvK
InvlEFyzD3ogWf4M2Vu8PB5Esb8rvwk2gyF8qnmAZhpju5AebK3DgAnxQGKnVCzGa7iVRIJZDWqX2vmkJdu3HNDmlXTXew8oNNTyORf3M4c59Lu245HXG11/28yAxnwnn3ygWhdzFgAanqoQcW0HC0G6ifcY/LYZ9VFEcTdNTsypASjf7k9L1ZXQyViP9O3VwDNyASCTgzcKih1g==","CSRFPreventionToken":"5F4697F2:aeUa65cbg3rEncOHtX7ArpCjbA5

I am assuming this means that the API is working as expected then?

 

[jfmanamtr]

I think I got this figured out. I had to set permissions on the cluster. I believe I have this working now.

Give me a little time and I'll put together a synopsis of what I did cause the instructions I followed are severely lacking.

 

[jfmanamtr]

ok. So, I followed these instructions, but they are a little incomplete. Specifically the section marked:

Install proxmox-pve-exporter

There are some steps you have to do inside of Proxmox to make Prometheus pve exporter work correctly.

You have to create a read only user under the datacenter > users section, click add.



Screen looks like this. Fill in the user name, set realm to Proxmox VE authentication & give it your password & confirm password. Finally click add



Move to the api tokens section. select the user you just created & give it a token id. then finally click add.



lastly go back to the permissions. Set the path to / & select your user.



Under role, make sure to select PVE Auditor.


Then follow those steps for the promoetheus-pve-exporter & it should work.

 

[Dominic]

Thank you for posting the update!

 

[sshaikh]

@jfmanamtr this was great. Just out of interest, why do you create a token? It doesn't look like the pve exporter supports them? I had to use the user/pass.

 

[sshaikh]

From https://github.com/prometheus-pve/prometheus-pve-exporter/issues/43, you can use tokens by using a slightly different config:

default:
  user: "..."
  token_name: "..."
  token_value: "..."

 

[jfmanamtr]

From https://github.com/prometheus-pve/prometheus-pve-exporter/issues/43, you can use tokens by using a slightly different config:

default:
  user: "..."
  token_name: "..."
  token_value: "..."

Ah. I honestly just used the token cause it worked. No special reason for it over any other way.

 

[thantos71]

Thanks for this as it really helped!

I used the token method and I still had an issue with a 403 error.
The key was to untick the privilege separation box in the 4th screenshot where you add the token and then everything works great

 

[fabian]

Thanks for this as it really helped!

I used the token method and I still had an issue with a 403 error.
The key was to untick the privilege separation box in the 4th screenshot where you add the token and then everything works great

alternatively you could give the token the permissions it needs

 

[kbaniszack]

@jfmanamtr
It's a great solution ! thank you for posting it ! Helps a lot.

Has anyone managed to configure the exporter to use https?
Thank you in advance.

 

[kbaniszack]

I found the solution:

/opt/prometheus-pve-exporter/bin/pve_exporter --config.file /opt/prometheus-pve-exporter/bin/pve.yml --server.keyfile /etc/pve/nodes/[Node_name]/pveproxy-ssl.key --server.certfile /etc/pve/nodes/[Node_name]/pveproxy-ssl.pem

 

[georgelza]

hi all, before i do something strange... looking at the original doc, listed higher up, it shows shell commands, assume those are to be executed on every proxmox node. - "ok. So, I followed these instructions, but they are a little incomplete. Specifically the section marked:"

followed by the additional steps ?

Would have been great if prometheus exporting was default capability, and simply a tick box to enable/disable it for a machine or cluster...

G

 

[georgelza]

... actually... has anyone written a ansible script to get this done ?
G

 

[georgelza]

... battling this side... followed the above to point....
getting a user authentication error...

root@pmox1:/etc/prometheus# systemctl status prometheus-pve-exporter
× prometheus-pve-exporter.service - Prometheus exporter for Proxmox VE
     Loaded: loaded (/etc/systemd/system/prometheus-pve-exporter.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sun 2024-11-17 19:25:47 SAST; 11s ago
   Duration: 796us
       Docs: https://github.com/znerol/prometheus-pve-exporter
   Main PID: 6810 (code=exited, status=217/USER)
        CPU: 415us

Nov 17 19:25:47 pmox1 systemd[1]: prometheus-pve-exporter.service: Scheduled restart job, restart counter is at 5.
Nov 17 19:25:47 pmox1 systemd[1]: Stopped prometheus-pve-exporter.service - Prometheus exporter for Proxmox VE.
Nov 17 19:25:47 pmox1 systemd[1]: prometheus-pve-exporter.service: Start request repeated too quickly.
Nov 17 19:25:47 pmox1 systemd[1]: prometheus-pve-exporter.service: Failed with result 'exit-code'.
Nov 17 19:25:47 pmox1 systemd[1]: Failed to start prometheus-pve-exporter.service - Prometheus exporter for Proxmox VE.

 

[kbaniszack]

Please, post your /etc/systemd/system/prometheus-pve-exporter.service file