Proxmox monitoring using PDM & ServiceNow Event Management

[KarelBP]

Hello everyone,
I've been playing with PVE monitoring and ServiceNow integration with PDM as the central point to collect metrics that are already available, instead of using a complex monitoring stack. The result is a small experiment that polls the RRD data from PVE nodes and sends events to ServiceNow Event Management when configured thresholds are exceeded for a certain period of time.

This is a proof of concept, not a ready-made product. The code is available on GitHub, background story in an article. The point is to demonstrate how the integration could work. It sends the events according to Pushing events to the instance using web service API.

The repository is small, so you may like to download the whole thing as a ZIP and feed it to your favorite LLM before running it in your lab. It could probably be adapted for other ITOM platforms, such as BMC Helix (some documentation here).

I'd be curious to hear what you think about it.

Thanks,
Karel

 

[Bu66as]

Nice idea to use PDM as the single data source instead of putting another exporter on every node. Haven't looked at the code in detail yet. How do you handle a node that goes down completely? If the RRD data just stops coming, a threshold check never fires. From my experience the "no data for X minutes" case is the alert you actually want, and derived metrics are a bad down signal. Do you also send a clear event (severity 0) with the same message_key once the value drops back below the threshold, so the alert in ServiceNow closes by itself and doesn't need manual cleanup? And if a whole node is down, do the events for all of its guests come through separately? That can flood the event queue pretty quickly if there's no grouping or suppression on one side or the other.

 

[KarelBP]

The poller remembers the last state. In case a host goes down, it sends a critical event (severity 1) with node_status key. At the same time it sends guest_unreachable for every VM/CT which was on the node. This might send a lot of data to the event queue, but that's what it's there for. Events trigger alerts, but those can be correlated, so in the end you may get only one high priority incident ticket. The module in SN is built for this, it really depends on how you set up your processing rules. The poller also sends clear events (severity 0), so once an alarm is not there anymore, it will announce it. You can also put things into "maintenance mode", that is silence certain alarms; they will be registered by the poller, but it won't send a beep until you remove the message key from the maintenance.txt file.

See the OPERATIONS.md and ACCEPTANCE.md files in the repo, there's a whole lot of explanation how it works/behaves and what I tested.