Ceph missing Prometheus stats

comfreak

Member
May 20, 2020
46
12
13
34
Hi,

I am not quite sure wether it belongs here or not, as I have the issue since the last proxmox update - otherwise I am sorry!

I updated Proxmox last weekend to 6.4-5 (I think from 6.3-3 or so) and after that, in the ceph prometheus module, some stats are missing. I am showing the exported stats in grafana.

At least most of the PG stats are missing and are not exported, which has worked before. I used before the update ceph octopus and I am still using it (probably a newer minor version...).

From http://pve:9283/metrics:

Code:
# HELP ceph_pg_active PG active per pool
# TYPE ceph_pg_active gauge
ceph_pg_active{pool_id="1"} 512.0
ceph_pg_active{pool_id="3"} 1.0
# HELP ceph_pg_clean PG clean per pool
# TYPE ceph_pg_clean gauge
ceph_pg_clean{pool_id="1"} 512.0
ceph_pg_clean{pool_id="3"} 1.0
# HELP ceph_pg_down PG down per pool
# TYPE ceph_pg_down gauge
# HELP ceph_pg_recovery_unfound PG recovery_unfound per pool
# TYPE ceph_pg_recovery_unfound gauge
# HELP ceph_pg_backfill_unfound PG backfill_unfound per pool
# TYPE ceph_pg_backfill_unfound gauge
# HELP ceph_pg_scrubbing PG scrubbing per pool
# TYPE ceph_pg_scrubbing gauge
# HELP ceph_pg_degraded PG degraded per pool
# TYPE ceph_pg_degraded gauge
# HELP ceph_pg_inconsistent PG inconsistent per pool
# TYPE ceph_pg_inconsistent gauge
# HELP ceph_pg_peering PG peering per pool
# TYPE ceph_pg_peering gauge
# HELP ceph_pg_repair PG repair per pool
# TYPE ceph_pg_repair gauge
# HELP ceph_pg_recovering PG recovering per pool
# TYPE ceph_pg_recovering gauge
# HELP ceph_pg_forced_recovery PG forced_recovery per pool
# TYPE ceph_pg_forced_recovery gauge
# HELP ceph_pg_backfill_wait PG backfill_wait per pool
# TYPE ceph_pg_backfill_wait gauge
# HELP ceph_pg_incomplete PG incomplete per pool
# TYPE ceph_pg_incomplete gauge
# HELP ceph_pg_stale PG stale per pool
# TYPE ceph_pg_stale gauge
# HELP ceph_pg_remapped PG remapped per pool
# TYPE ceph_pg_remapped gauge
# HELP ceph_pg_deep PG deep per pool
# TYPE ceph_pg_deep gauge
# HELP ceph_pg_backfilling PG backfilling per pool
# TYPE ceph_pg_backfilling gauge
# HELP ceph_pg_forced_backfill PG forced_backfill per pool
# TYPE ceph_pg_forced_backfill gauge
# HELP ceph_pg_backfill_toofull PG backfill_toofull per pool
# TYPE ceph_pg_backfill_toofull gauge
# HELP ceph_pg_recovery_wait PG recovery_wait per pool
# TYPE ceph_pg_recovery_wait gauge
# HELP ceph_pg_recovery_toofull PG recovery_toofull per pool
# TYPE ceph_pg_recovery_toofull gauge
# HELP ceph_pg_undersized PG undersized per pool
# TYPE ceph_pg_undersized gauge
# HELP ceph_pg_activating PG activating per pool
# TYPE ceph_pg_activating gauge
# HELP ceph_pg_peered PG peered per pool
# TYPE ceph_pg_peered gauge
# HELP ceph_pg_snaptrim PG snaptrim per pool
# TYPE ceph_pg_snaptrim gauge
# HELP ceph_pg_snaptrim_wait PG snaptrim_wait per pool
# TYPE ceph_pg_snaptrim_wait gauge
# HELP ceph_pg_snaptrim_error PG snaptrim_error per pool
# TYPE ceph_pg_snaptrim_error gauge
# HELP ceph_pg_creating PG creating per pool
# TYPE ceph_pg_creating gauge
# HELP ceph_pg_unknown PG unknown per pool
# TYPE ceph_pg_unknown gauge
# HELP ceph_pg_premerge PG premerge per pool
# TYPE ceph_pg_premerge gauge
# HELP ceph_pg_failed_repair PG failed_repair per pool
# TYPE ceph_pg_failed_repair gauge
# HELP ceph_pg_laggy PG laggy per pool
# TYPE ceph_pg_laggy gauge
# HELP ceph_pg_wait PG wait per pool
# TYPE ceph_pg_wait gauge

Grafana shows the Active and Clean correctly (as they are exported). The Scrub and Deep-Scrubs are the last non Zero value. They show 1, because now it is not exported. The 1 comes from the scrub from last night (timeframe from 6-now).


1620289236793.png

Is this new and correct? Will Ceph only export data, when there is a non-zero value like the scrubs as they are running at night and therefore are non-zero?

Am I missing a new configuration or something else?

Thanks!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!