[TUTORIAL] Bypassing Broken Ceph/PVE REST API Monitoring with Zabbix Agent 2 (Proxmox VE 9.2 / Ceph 19 / Zabbix 7.4)

D

Disciplus

Renowned Member
Apr 25, 2018
4
1
68
46

Context & Problem​

When upgrading to Proxmox VE 9.2 running Ceph 19 (Squid), native RESTful API monitoring endpoints can break or become unstable depending on your Zabbix / PVE API configuration.

Rather than relying on external HTTPS REST requests or heavy API polling, a lightweight, highly efficient local wrapper combined with Zabbix Agent 2, Dependent Items, and JSONPath Preprocessing provides an alternative.

Architecture Overview​

To keep system overhead minimal, we do not run ceph status for every single item.

  1. Master Item (custom.ceph.status[status]): Fetches the entire cluster status JSON in a single local CLI call every 60 seconds.
  2. Dependent Items: Extract global health, OSD counts, and pool capacity in-memory using JSONPath without additional execution costs.
  3. Low-Level Discovery (LLD): Uses custom.ceph.status[osd-df] to automatically discover OSDs and parse per-OSD utilization.

Step-by-Step Configuration​

1. Create the Shell Wrapper Script​

On the Proxmox node, create /usr/local/bin/zabbix_ceph_status.sh:

Code: 
echo "zabbix ALL=(ALL) NOPASSWD: /usr/local/bin/zabbix_ceph_status.sh" > /etc/sudoers.d/zabbix_ceph
chmod 440 /etc/sudoers.d/zabbix_ceph

2. Grant Sudo Privileges​

Allow the zabbix system user to execute the script without password prompting:
Code: 
echo "zabbix ALL=(ALL) NOPASSWD: /usr/local/bin/zabbix_ceph_status.sh" > /etc/sudoers.d/zabbix_ceph
chmod 440 /etc/sudoers.d/zabbix_ceph

3. Configure Zabbix Agent 2​

Note: Using a custom key prefix like custom.ceph.status[*] avoids conflicts with the native zabbix-agent2-plugin-ceph reserved keys.

Create /etc/zabbix/zabbix_agent2.d/ceph_script.conf:

Code: 
UserParameter=custom.ceph.status[*],sudo /usr/local/bin/zabbix_ceph_status.sh "$1"

Restart the agent:

Code: 
systemctl restart zabbix-agent2

Validate local execution:

Code: 
zabbix_agent2 -t "custom.ceph.status[status]"

Zabbix Frontend Setup (7.4 / 7.0 LTS)​

Master Items​

  • Ceph: Get Cluster Status Raw
    • Type: Zabbix agent
    • Key: custom.ceph.status[status]
    • Type of information: Text
    • Update Interval: 1m
  • Ceph: Get OSD DF Raw
    • Type: Zabbix agent
    • Key: custom.ceph.status[osd-df]
    • Type of information: Text
    • Update Interval: 5m

Dependent Items (Parsing JSONPath)​

  • Global Health:
    • Type: Dependent item | Master: Ceph: Get Cluster Status Raw
    • Key: ceph.health.status
    • JSONPath: $.health.status
    • Trigger Expression: last(/Template/ceph.health.status)<>"HEALTH_OK"
  • Total OSDs:
    • Key: ceph.osdmap.num_osds | JSONPath: $.osdmap.num_osds
  • Up OSDs:
    • Key: ceph.osdmap.num_up_osds | JSONPath: $.osdmap.num_up_osds
    • Trigger Expression: last(/Template/ceph.osdmap.num_up_osds)<last(/Template/ceph.osdmap.num_osds)
  • Used Capacity:
    • Key: ceph.pgmap.bytes_used | JSONPath: $.pgmap.bytes_used (Units: B)

Low-Level Discovery (LLD) Rules​

  • Name: Ceph OSD Discovery
  • Type: Dependent item | Master: Ceph: Get OSD DF Raw
  • LLD Macros:
    • {#OSD_ID} = $.nodes
    • .id
    • {#OSD_NAME} = $.nodes
    • .name
  • Item Prototype:
    • Name: OSD {#OSD_NAME}: Utilization
    • Type: Dependent item | Master: Ceph: Get OSD DF Raw
    • Key: ceph.osd.utilization[{#OSD_ID}]
    • JSONPath Preprocessing: $.nodes[?(@.id == {#OSD_ID})].utilization.first()

Key Results​

  • Minimal CPU & I/O Footprint: Reduced from N execution calls down to 1 single CLI call per cycle for all cluster metrics.
  • Alerting Latency: Mean Time To Detect (MTTD) under 60 seconds for HEALTH_WARN or OSD DOWN states.
  • Zero API Dependency: Independent from API tokens, expired certificates, or REST module breakage.

you can import this xml file in ZABBIX if you want :


Code: 
<?xml version="1.0" encoding="UTF-8"?>
<zabbix_export>
    <version>7.0</version>
    <template_groups>
        <template_group>
            <uuid>7df9672f09124172b7d02820d8697265</uuid>
            <name>Templates/Storage</name>
        </template_group>
    </template_groups>
    <templates>
        <template>
            <uuid>a89c23f1e9484029b32e18d61234abcd</uuid>
            <template>Ceph Cluster by Custom Script</template>
            <name>Ceph Cluster by Custom Script</name>
            <description>Monitoring Ceph Squid sur Proxmox VE via script wrapper Zabbix Agent 2 local.</description>
            <groups>
                <group>
                    <name>Templates/Storage</name>
                </group>
            </groups>
            <items>
                <item>
                    <uuid>b1111111111141118111111111111111</uuid>
                    <name>Ceph: Get Cluster Status Raw</name>
                    <key>custom.ceph.status[status]</key>
                    <delay>1m</delay>
                    <history>1d</history>
                    <trends>0</trends>
                    <value_type>TEXT</value_type>
                    <description>Récupération brute du JSON ceph status.</description>
                </item>
                <item>
                    <uuid>b2222222222242228222222222222222</uuid>
                    <name>Ceph: Get OSD DF Raw</name>
                    <key>custom.ceph.status[osd-df]</key>
                    <delay>5m</delay>
                    <history>1d</history>
                    <trends>0</trends>
                    <value_type>TEXT</value_type>
                    <description>Récupération brute du JSON ceph osd df.</description>
                </item>
                <item>
                    <uuid>b3333333333343338333333333333333</uuid>
                    <name>Ceph: Health Status</name>
                    <type>DEPENDENT</type>
                    <key>ceph.health.status</key>
                    <delay>0</delay>
                    <history>31d</history>
                    <trends>0</trends>
                    <value_type>CHAR</value_type>
                    <master_item>
                        <key>custom.ceph.status[status]</key>
                    </master_item>
                    <preprocessing>
                        <step>
                            <type>JSONPATH</type>
                            <parameters>
                                <parameter>$.health.status</parameter>
                            </parameters>
                        </step>
                    </preprocessing>
                    <triggers>
                        <trigger>
                            <uuid>c1111111111141118111111111111111</uuid>
                            <expression>last(/Ceph Cluster by Custom Script/ceph.health.status)&lt;&gt;"HEALTH_OK"</expression>
                            <name>Ceph cluster health is degraded ({ITEM.LASTVALUE1})</name>
                            <opdata>Current status: {ITEM.LASTVALUE1}</opdata>
                            <priority>HIGH</priority>
                        </trigger>
                    </triggers>
                </item>
                <item>
                    <uuid>b4444444444444448444444444444444</uuid>
                    <name>Ceph: OSDs Total</name>
                    <type>DEPENDENT</type>
                    <key>ceph.osdmap.num_osds</key>
                    <delay>0</delay>
                    <history>31d</history>
                    <master_item>
                        <key>custom.ceph.status[status]</key>
                    </master_item>
                    <preprocessing>
                        <step>
                            <type>JSONPATH</type>
                            <parameters>
                                <parameter>$.osdmap.num_osds</parameter>
                            </parameters>
                        </step>
                    </preprocessing>
                </item>
                <item>
                    <uuid>b5555555555545558555555555555555</uuid>
                    <name>Ceph: OSDs Up</name>
                    <type>DEPENDENT</type>
                    <key>ceph.osdmap.num_up_osds</key>
                    <delay>0</delay>
                    <history>31d</history>
                    <master_item>
                        <key>custom.ceph.status[status]</key>
                    </master_item>
                    <preprocessing>
                        <step>
                            <type>JSONPATH</type>
                            <parameters>
                                <parameter>$.osdmap.num_up_osds</parameter>
                            </parameters>
                        </step>
                    </preprocessing>
                    <triggers>
                        <trigger>
                            <uuid>c2222222222242228222222222222222</uuid>
                            <expression>last(/Ceph Cluster by Custom Script/ceph.osdmap.num_up_osds)&lt;last(/Ceph Cluster by Custom Script/ceph.osdmap.num_osds)</expression>
                            <name>One or more Ceph OSDs are DOWN</name>
                            <priority>HIGH</priority>
                        </trigger>
                    </triggers>
                </item>
                <item>
                    <uuid>b6666666666646668666666666666666</uuid>
                    <name>Ceph: Storage Total Capacity</name>
                    <type>DEPENDENT</type>
                    <key>ceph.pgmap.bytes_total</key>
                    <delay>0</delay>
                    <history>31d</history>
                    <units>B</units>
                    <master_item>
                        <key>custom.ceph.status[status]</key>
                    </master_item>
                    <preprocessing>
                        <step>
                            <type>JSONPATH</type>
                            <parameters>
                                <parameter>$.pgmap.bytes_total</parameter>
                            </parameters>
                        </step>
                    </preprocessing>
                </item>
                <item>
                    <uuid>b7777777777747778777777777777777</uuid>
                    <name>Ceph: Storage Used</name>
                    <type>DEPENDENT</type>
                    <key>ceph.pgmap.bytes_used</key>
                    <delay>0</delay>
                    <history>31d</history>
                    <units>B</units>
                    <master_item>
                        <key>custom.ceph.status[status]</key>
                    </master_item>
                    <preprocessing>
                        <step>
                            <type>JSONPATH</type>
                            <parameters>
                                <parameter>$.pgmap.bytes_used</parameter>
                            </parameters>
                        </step>
                    </preprocessing>
                </item>
            </items>
            <discovery_rules>
                <discovery_rule>
                    <uuid>d1111111111141118111111111111111</uuid>
                    <name>Ceph OSD Discovery</name>
                    <type>DEPENDENT</type>
                    <key>ceph.osd.discovery</key>
                    <delay>0</delay>
                    <master_item>
                        <key>custom.ceph.status[osd-df]</key>
                    </master_item>
                    <lld_macro_paths>
                        <lld_macro_path>
                            <lld_macro>{#OSD_ID}</lld_macro>
                            <path>$.nodes[*].id</path>
                        </lld_macro_path>
                        <lld_macro_path>
                            <lld_macro>{#OSD_NAME}</lld_macro>
                            <path>$.nodes[*].name</path>
                        </lld_macro_path>
                    </lld_macro_paths>
                    <item_prototypes>
                        <item_prototype>
                            <uuid>e1111111111141118111111111111111</uuid>
                            <name>OSD {#OSD_NAME}: Utilization</name>
                            <type>DEPENDENT</type>
                            <key>ceph.osd.utilization[{#OSD_ID}]</key>
                            <delay>0</delay>
                            <history>31d</history>
                            <value_type>FLOAT</value_type>
                            <units>%</units>
                            <master_item>
                                <key>custom.ceph.status[osd-df]</key>
                            </master_item>
                            <preprocessing>
                                <step>
                                    <type>JSONPATH</type>
                                    <parameters>
                                        <parameter>$.nodes[?(@.id == {#OSD_ID})].utilization.first()</parameter>
                                    </parameters>
                                </step>
                            </preprocessing>
                        </item_prototype>
                    </item_prototypes>
                </discovery_rule>
            </discovery_rules>
        </template>
    </templates>
</zabbix_export>

;)
 
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom
Back