Zabbix Template - Proxmox VE by HTTP - not working to provide data

[greavette]

Hello Forum,

I'm hoping someone can assist or point me in the right direction on how to get monitoring from Zabbix to my Proxmox servers working.

I found this post in the Proxmox forums related to what I'm doing. The solution was to re-issue the API key. I've tried this but I still can't get monitoring to work:

https://forum.proxmox.com/threads/monitoring-with-zabbix-not-working.112477/

I've opened this post in the Zabbix forums but no one has replied yet with suggestions. I'm using Zabbix 6.0 and my Proxmox has been updated to 7.2-11.

Linux by Zabbix agent
Template Proxmox VE by HTTP
https://git.zabbix.com/projects/ZBX/...Frelease%2F6.0

Many of the items being monitored using this Promox Template have errors in Latest Data:

Proxmox: Node []: Status - Response code "403" did not match any of the required status codes "200"

I see this error on many items being monitored.

Zabbix 1 image.

More information regarding my issue:

I'm looking at the Item - Proxmox: Get cluster resources.

looking at the configuration for this item I've run a Tes and I can see data coming back from the Result of my cluster. It's accurate in that it shows which hosts are online or offline.

Zabbix 2 image.

My test does show that the details are there but for some reason the latest data is showing blank and no history or graph details. Is this template specific to the version of Proxmox?

Any advice or direction you may have would be greatly appreciated.

Thank you.

 

[greavette]

I don't think it's a permission issue although it could be a problem with how the template can connect to the Proxmox API that is giving the 403 error.

I have done a lot more digging. I found this post that outlines - https://i12bretro.github.io/tutorials/0572.html. Where it provides very good detail on how to create the REST API in Proxmox and how to test connect to Proxmox through Postman. I've done this and found that some of the commands work like version and nodes but status didn't work for from postman.

Was there a change between how the Rest API works between Proxmox version 6 and I'm using Proxmox version 7?

Can anyone provide some details or point me to where I can learn more about how to connect to the Proxmox API and get the data I need in my Zabbix Template?

How are others monitoring their Proxmox cluster?

Thank you.

 

[Dunuin]

Not really a help how to get your template running, but I use this alternative template and that works fine with Zabbix 6 LTS and PVE 7.2: https://github.com/ismvru/zbx-tmplt-pve

 

[greavette]

Not really a help how to get your template running, but I use this alternative template and that works fine with Zabbix 6 LTS and PVE 7.2: https://github.com/ismvru/zbx-tmplt-pve

Thanks for the reply @Dunuin . I wasn't aware of this earlier version of the same template. I tried importing this template you've linked into my Zabbix 6 but I receiving an error for an "Invalid parameter"

• Invalid parameter "/12/params": incorrect expression starting from "last(pve.storage.used[{#NODE},{#NAME}])/last(pve.storage.total[{#NODE},{#NAME}])".

Did you receive this error and if so what did you do to get past it in Zabbix 6?

Thank you.

 

[Dunuin]

No errors here. Maybe @vashvampir can help. He created that template.

 

[greavette]

Thanks for the reply @Dunuin , I found my issue. I'm using the Proxmox PVE HTTP template that comes with Zabbix 6. Did @vashvampir originally create this template? If so Wow! Nice work.

My issue was related to the permissions of the API token. In addition to creating the token I also had to create a new role which I named API admin. I then added all privileges available to this new role. Then in Permissions, I added this API Admin role to my new API token. Once I added the necessary token information to the macro in Zabbix template all my permission issues were gone and now I'm getting so much information from my Proxmox cluster it's amazing! Hopefully, this helps anyone else trying to do the same.

 

[vashvampir]

Thanks for the reply @Dunuin , I found my issue. I'm using the Proxmox PVE HTTP template that comes with Zabbix 6. Did @vashvampir originally create this template? If so Wow! Nice work.

My issue was related to the permissions of the API token. In addition to creating the token I also had to create a new role which I named API admin. I then added all privileges available to this new role. Then in Permissions, I added this API Admin role to my new API token. Once I added the necessary token information to the macro in Zabbix template all my permission issues were gone and now I'm getting so much information from my Proxmox cluster it's amazing! Hopefully, this helps anyone else trying to do the same.

Hello! Replied to Issue on GitHub, copyng my answer here.

Most likely, the user under which the template goes to the PBS API does not have enough permissions...

Minimal needed permissions for user:

/datastore - Audit
/system/status - Audit
/system/tasks - Audit

 

[Dunuin]

Did you also find out what permissions are minimum needed for your PVE template? I'm still running that token with full admin privileges which is of cause not ideal because of security.
I think I asked that already before but you weren't sure back then.

 

[vashvampir]

Did you also find out what permissions are minimum needed for your PVE template? I'm still running that token with full admin privileges which is of cause not ideal because of security.
I think I asked that already before but you weren't sure back then.

No, unfortunately there was no time to do this, and with the advent of the official template from the Zabbix team, I stopped developing my template for PVE.

You can try setting the following privileges for the token:
/ - PVEAuditor

 

[greavette]

Hello! Replied to Issue on GitHub, copyng my answer here.

Most likely, the user under which the template goes to the PBS API does not have enough permissions...

Minimal needed permissions for user:

/datastore - Audit
/system/status - Audit
/system/tasks - Audit

Hello @vashvampir ,

Thanks very much for your reply to my post and github post...I really appreciate your input and for providing this template for all to use.

I've added more than what was required to the permissions on my PBS server. I've added Admin in fact to each of the 3 Path's you've mentioned and even added Admin to "/" for my API Token user I created.




Unfortunately my Zabbix still has the same issues with authentication errors. For example item "PBS: Node status" still has the following error:
Response code "403" did not match any of the required status codes "200"

I've confirmed that the PBS.Node.Name and PBS.Token.ID, PBS.Token.Secret and Host.Conn are all correct. But when I test I see the above error:





I do receive some values from the PBS template so my token must be working in some respects. But for example if I look at the item "PBS Datastore: pbs-ds-01 total" does return a value for me but there is no way to Test and see what macro's are being used?

Do you know what in the Template enables the Test button on an item from Latest Data?




Or another example, the item "PBS: Datastores usage" does return data from my PBS server when I test:

I've confirmed that the values in PBS.Token.ID, PBS.Token.Secret and Host.Code are the same in this Item test to the above item test. I notice in the above screenshot there is an extra macro used PBS.Node.Name. Not sure why one item test works and the other has an error.







Thank you!

 

[vashvampir]

Hello!
Testing items in this template is possible only for items with the HTTP Agent type:

* `pbs.ping`
* `pbs.status.datastore-usage`
* `pbs.node.status`
* `pbs.tasks.failed`
* `pbs.version`


The remaining items are taken from them by parsing the above items.

If it's not difficult, can you throw off the exact text of the errors from the Zabbix server log, the version of Zabbix itself and the PBS version? I'll test it and see what could go wrong.

 

[greavette]

Hello @vashvampir

No need to apologize. You've provided this for us all free of charge and are still supporting my questions. I really appreciate you doing that for me and for everyone else's benefit. Thank you!

Thank you for your explanation as to why some of the items are testable from the latest data > configuration of the item. I think another reason is because of the error I'm receiving on some items and there is a dependency from an item where I'm receiving the access error.

I'd be happy to share the information you requested.

From my /var/log/zabbix/zabbix_server.log.1 file I see the following error when Zabbix is trying to connect to my PBS server swaspbs1:

 677:20221025:125915.340 error reason for "swaspbs1:pve.cluster.status" changed: Response code "403" did not match any of the required status codes "200"

My Zabbix server is version 6.0.9

My PBS Server is version 2.2-6

Thank you!

 

[garbled]

I was just fiddling with this. Turns out, if you add the permissions to the USER, it doesn't work. Instead I had to add the perms to the API Token. I just added SysAudit to / on the api token, and suddenly I was pulling down data.

 

[greavette]

Hello @garbled , appreciate your input. Please confirm though what you did. Here is what I've done:

On my PBS Server I have done the following:
Created a user called pbs-user@pbs.
Created an API token for this pbs-user.
In Permissions I added an API Token Permission for pbs-user.

The first permission I tried was Audit for / but that did not fix my issue in Zabbix.
I then changed the permission for pbs-user token for / to be Admin Role. But once again my Zabbix Latest Data does not show all the details I need.
Specifically, I have the following errors for these items:

1. PBS: Failed Tasks - Response code "400" did not match any of the required status codes "200"
2. PBS Node: Status - Response code "403" did not match any of the required status codes "200"
3. PBS Node: memory used percent - Cannot evaluate function: not enough data at "last(//pbs.node.memory.used) / last(//pbs.node.memory.total) * 100".
4. PBS Node: root used percent - Cannot evaluate function: not enough data at "last(//pbs.node.root.used) / last(//pbs.node.root.total) * 100"
5. PBS Node: swap used percent - Cannot evaluate function: not enough data at "last(//pbs.node.swap.used) / last(//pbs.node.swap.total) * 100".

You will notice on most of my items from the PBS template there is no data collected in my Zabbix. Only some of the items have data collected:



Please let me know what values you see in your PBS Tempate on your Zabbix when you adjusted your permissions.

Thank you!

 

[garbled]

OK, to be clear, I am using the proxmox VE by HTTP template, from the zabbix site, on zabbix 6.0, pve 7.2.

I'm not using it on PBS. Is it supposed to work on PBS? I don't see any PBS stuff in the template I pulled down, which one are you using? (I also ask because I really want a pbs monitor)

But for clarity:

created user zabbix auth realm PVE
created api token for zabbix
assigned PVEAudit to user zabbix
assigned PVEAudit to api token for zabbix

template works. tested on 2 different pve clusters.

 

[greavette]

Hello @garbled , I appreciate you confirming what you did. I also have had success in adding the PVE by HTTP template to my Proxmox Servers (all 9 are working great!). I also have Zabbix 6 LTS and my Proxmox is version 7.2.11.

There is a PBS template as well created by @vashvampir who has been kind enough to provide insight into both templates for us. As you can see from the screenshot I provided above, my Zabbix can now provide some details from my two PBS servers. I'm very excited to be able to see my datastore size now. But the problem I'm having with the PBS template is that I have some errors from some items related to seeing the status of my PBS tasks. It appears to be a permission issue and it might be related to the version of PBS I'm using 2.2-6. I'm hoping vashvampir can see the reason for these errors so we can get real value from this template. In the meantime you can add his template to your Zabbix and you'll at least see the data I'm seeing from my screenshot above.

https://github.com/ismvru/zbx-tmplt-pbs

I have to apologize as well...my initial reason for this post was for the PVE by HTTP template but I also began using @vashvampir with questions about the PBS template as well since he created both.

 

[greavette]

Hello @vashvampir , have you made any progress on why the PBS template is missing some data in my latest data? I still think there is a permission problem in the template being able to retrieve the data it is looking for.

Thank you.

 

[foka_lopes]

Hi guys!
I read and reread this topic several times. But a still couldn't receive the metrics.
I followed this https://i12bretro.github.io/tutorials/0572.html to setup API Token, User and permissions.
And I setup my macros like this: https://www.zabbix.com/integrations/proxmox#proxmox
I don't know if is matter, but I'm using zabbix proxy to monitor this PVE in the remote location.
As you can see,
"API service status" is "OK (200)"

So, the others itens isn't receiving any data.
For example the "Proxmox: Node [pve]: Status" item is:
"Preprocessing failed for: Error getting data
1. Failed: cannot extract value from json by path "$.data.[?(@.name == 'pve' && @.type == 'node')].online.first()": cannot parse as a valid JSON object: invalid object format, expected opening character '{' or '[' at: 'Error getting data'

What is missing?

 

[jialine]

I have solved this problem！
Please type this command at zabbix server:

setsebool -P zabbix_can_network 1

 

[complexplaster27]

I have solved this problem！
Please type this command at zabbix server:

setsebool -P zabbix_can_network 1

Legend this is what finally got it all working, FYI I've been trying to troubleshoot this issue for hours using a Rocky Linux Install.