Metric Server (Influxdbv3 / Grafana) VM IOPs count and latency graphs?
- Thread starter rahman
- Start date
The built-in metric server sends only `diskread` and `diskwrite` per VM, as cumulative byte counters. A non-negative derivative in Grafana turns those into throughput, but the export has no operation counts or latency.
For IOPS and latency, run `qm status <vmid> --verbose` on the node and look at the `blockstat` section for each disk. `rd_operations` and `wr_operations` are cumulative operation counts, and `rd_total_time_ns` and `wr_total_time_ns` are the cumulative time spent on them.
Collect those on a schedule with a Telegraf `exec` input that runs the command and writes to InfluxDB. Apply the same non-negative derivative to the counts for IOPS, and divide the change in `rd_total_time_ns` by the change in `rd_operations` over the same interval for average read latency (the same for writes).
Drafted with AI.
For IOPS and latency, run `qm status <vmid> --verbose` on the node and look at the `blockstat` section for each disk. `rd_operations` and `wr_operations` are cumulative operation counts, and `rd_total_time_ns` and `wr_total_time_ns` are the cumulative time spent on them.
Collect those on a schedule with a Telegraf `exec` input that runs the command and writes to InfluxDB. Apply the same non-negative derivative to the counts for IOPS, and divide the change in `rd_total_time_ns` by the change in `rd_operations` over the same interval for average read latency (the same for writes).
Drafted with AI.
This is definitely a missing metric. having to run a schedule tasks to pull this information from each and every host/VM in our environment just seems broken to me.
IMO the metrics that are available via the API are extremely lacking when compared to the competition. Hopefully there will be some updates to this in the future.
Right now we are using the OTEL to send the metrics to Alloy then onto Mimir/Grafana. It works really well with the metrics available but it can be improved with things like IOPS.
I would also love to see an easier way to setup logging to a remote location as well ( right now we are using alloy sending to Loki )
IMO the metrics that are available via the API are extremely lacking when compared to the competition. Hopefully there will be some updates to this in the future.
Right now we are using the OTEL to send the metrics to Alloy then onto Mimir/Grafana. It works really well with the metrics available but it can be improved with things like IOPS.
I would also love to see an easier way to setup logging to a remote location as well ( right now we are using alloy sending to Loki )
A feature request on bugzilla.proxmox.com is the way to get per-disk operation counts added to the metric server export.
Until then, one systemd timer per node covers every VM on it. Have it loop over `qm list`, read each disk's `blockstat` from `qm status <vmid> --verbose`, and write the counters to a `.prom` file.
Alloy's `prometheus.exporter.unix` component reads that file through its `textfile` block, so the counters reach Mimir with the rest of your metrics.
Drafted with AI.
Until then, one systemd timer per node covers every VM on it. Have it loop over `qm list`, read each disk's `blockstat` from `qm status <vmid> --verbose`, and write the counters to a `.prom` file.
Alloy's `prometheus.exporter.unix` component reads that file through its `textfile` block, so the counters reach Mimir with the rest of your metrics.
Drafted with AI.
If I can get the metrics for IOPs/latency via pvesh/qm commands then I wonder why they are not available via builtin metric exporter :/
Filed a feature request: https://bugzilla.proxmox.com/show_bug.cgi?id=8208
Filed a feature request: https://bugzilla.proxmox.com/show_bug.cgi?id=8208