Push per-node Prometheus metrics (no duplicates)

[LRademakers]

I don't know if this is the right thread, please let me know. At the moment I have a 3 node cluster with Prometheus PVE-Exporter and Grafana Alloy installed. For now I have the Prometheus PVE-Exporter and Grafana Alloy (directly) installed on the first node, and use this dashboard, which shows all metrics from all nodes. My question is if it is possible to push the metrics separately from each node, so there will no duplicates and also shows the metrics separately in the Grafana dashboard?

 

[LRademakers]

Maybe I'm answering my own question here, but as I see it this isn't possible because the Proxmox API exposes all the metrics (for "The cluster node name (for types 'node', 'storage', 'qemu', and 'lxc')" ) on the /cluster/resources/node path, so it will always return all related metrics regardless of which node they are on.

Edit: If my assumption is confirmed, does that mean my setup, installing Prometheus PVE-Exporter + Grafana Alloy on every node, isn't possible because the data is cluster‑wide?

 

[chuck7k]

https://github.com/Starttoaster/proxmox-exporter ??

Or from pve-exporter doc:

Note on scraping large clusters:


It is adviced to setup separate jobs to collect cluster metrics andnode metrics in larger deployments. Scraping any node in a cluster with theurl params set to cluster=1&node=0 results in the same set of metrics. Hencecluster metrics can be scraped efficiently from a single node or from a subsetof cluster nodes (e.g., a different node selected on every scrape viaround-robin DNS).


Node metrics can only be scraped from a given node. In order to compile acomplete set of node metrics it is necessary to scrape every node in a clusterwith url params set to cluster=0&node=1.

 

[chuck7k]

# Job 1 — cluster metrics (scrape once via round-robin DNS)
- job_name: 'pve_cluster'
  static_configs:
    - targets:
      - pve-cluster.home.arpa   # round-robin between 192.168.1.5 and 192.168.1.6
  metrics_path: /pve
  params:
    module: [default]
    cluster: ['1']
    node: ['0']           # ← cluster only, no node metrics
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 'prometheus-pve-exporter:9221'

# Job 2 — node metrics (scrape every node individually)
- job_name: 'pve_nodes'
  static_configs:
    - targets:
      - 192.168.1.5       # pve2
      - 192.168.1.6       # pve3
  metrics_path: /pve
  params:
    module: [default]
    cluster: ['0']        # ← node only, no cluster metrics
    node: ['1']
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 'prometheus-pve-exporter:9221'