Isolating resources within LXC containers with node_exporter on Proxmox

[freak12techno]

I have a Proxmox server and 4 containers inside. This server has 7 disks connected to it.
I am running node_exporter on all of the containers to get an overview on what's happening inside each container, however there are some metrics related to hardware not used by the container. For example, this is the output of running curl localhost:9100/metrics:

# HELP node_disk_writes_completed_total The total number of writes completed successfully.
# TYPE node_disk_writes_completed_total counter
node_disk_writes_completed_total{device="dm-0"} 7.170327e+06
node_disk_writes_completed_total{device="dm-1"} 768088
node_disk_writes_completed_total{device="dm-2"} 437969
node_disk_writes_completed_total{device="dm-3"} 719625
node_disk_writes_completed_total{device="nvme0n1"} 3.193654e+06
node_disk_writes_completed_total{device="sda"} 98545
node_disk_writes_completed_total{device="sdb"} 99308
node_disk_writes_completed_total{device="sdc"} 314856
node_disk_writes_completed_total{device="sdd"} 343948
node_disk_writes_completed_total{device="sde"} 276031
node_disk_writes_completed_total{device="sdf"} 2138

It includes information on every disk I have installed on an underlying server, despite the container using only one of these disks (nvme0n1 here).

The issue is that I have node_exporter in all of my containers and Proxmox server as well, so if one disk is slow, I will get 5 alerts from Prometheus, with the same device but for each of these containers.

Is there any way to limit node_exporter to only return data about the hardware that is actually used by the container?

 

[Deripa]

you can limit the metrics exposed by node_exporter to only include the hardware that is relevant to each container. This can be achieved by configuring node_exporter to use the appropriate Prometheus text file collector directory for each container.

Here's a step-by-step guide to configure node_exporter for each container:

Determine the device(s) that are relevant to each container. In your example, the container is using the "nvme0n1" device.

Create a directory for each container in the Prometheus text file collector directory. By default, this directory is located at /var/lib/node_exporter/textfile_collector/.
bash
sudo mkdir -p /var/lib/node_exporter/textfile_collector/container1

Within each container's directory, create a text file to define the metrics for that container. Name the file something descriptive, like container1_metrics.prom.sudo nano /var/lib/node_exporter/textfile_collector/container1/container1_metrics.prom

In the text file, define the metrics that should be exposed for the container. In this case, you want to include only the metrics related to the "nvme0n1" device. Copy the relevant lines from the original node_exporter metrics output and paste them into the text file.

# HELP node_disk_writes_completed_total The total number of writes completed successfully.
# TYPE node_disk_writes_completed_total counternode_disk_writes_completed_total{device="nvme0n1"} 3.193654e+06

Save and close the text file.

Repeat steps 2-5 for each container, creating a separate directory and text file for each container and including only the relevant metrics.

Restart the node_exporter service to apply the changes.sudo systemctl restart node_exporter

By following these steps, each container's node_exporter will only expose the metrics defined in the respective text file, limiting the metrics to the hardware relevant to each container. This will prevent unnecessary alerts for disks that are not being used by a particular container.If I guessed wrong then maybe you should contact an expert in software development https://tech-stack.com/services/custom-software-development, maybe they can help you