[SOLVED] ceph zabbix send Fehler nach Update auf PVE 6.1

[Ingo S]

Hallo zusammen

Wir haben Zabbix als Monitoring und bislang erfolgreich auch die Ceph Metriken via dem zabbix Modul des Ceph MGR an unser Monitoring versendet. Gestern haben wir von PVE 6.0 auf 6.1 upgedated und seitdem meldet Ceph Failed to send data to Zabbix

Manuell erhalte ich folgende Meldungen auf der Konsole und im Log:

root@vm-2:~# ceph zabbix send
Failed to send data to Zabbix
root@vm-2:~#
-----------
parallel dazu:
root@vm-2:~#journalctl -f
Dec 19 11:11:31 vm-2 ceph-mgr[2825]: 2019-12-19 11:11:31.870 7f2024255700 -1 mgr.server reply reply (1) Operation not permitted

Jemand ne Idee? Es wurde nichts an der Konfiguration geändert, wir haben nur die aktuellen Updates eingespielt.

 

[Alwin]

Might be the template change.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LTX6IRH3JJ2JVUKB3PQGMLZCYQBR46YN/

 

[Ingo S]

Indeed, that solved the problem. I changed the Key ceph.num_pg_wait_backfill to ceph.num_pg_backfill_wait in the Template. A few minutes later the error message was gone.
Thx

 

[sunkroot]

Thank you for opening this thread. Unfortunately, i still have a problem even with right syntax in the zabbix_template.xml
any help please?


root@1:/usr/share/ceph/mgr/zabbix# grep ceph.num_pg_wait_backfill *.xml
root@1:/usr/share/ceph/mgr/zabbix# grep ceph.num_pg_backfill_wait *
zabbix_template.xml: <key>ceph.num_pg_backfill_wait</key>
root@1:/usr/share/ceph/mgr/zabbix#
ceph zabbix send -vv
ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable)

 

[Ingo S]

Hi sunkroot

on our server the search for "ceph.num_pg_wait_backfill" gives the exact same result. ceph.num_pg_backfill_wait * is present in zabbix_template.xml.
ceph zabbix send is working just fine and returns with "ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable)"

So, i don't know what the exact problem is on your side.

 

[sunkroot]

Hi sunkroot

on our server the search for "ceph.num_pg_wait_backfill" gives the exact same result. ceph.num_pg_backfill_wait * is present in zabbix_template.xml.
ceph zabbix send is working just fine and returns with "ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable)"

So, i don't know what the exact problem is on your side.

Thank you for looking into this issue. Unfortunately nothing from ceph app is send to zabbix server. I'm not sure where to begin..

root@pve1:/var/log/ceph# ceph zabbix send
Failed to send data to Zabbix
root@pve1:/var/log/ceph#

root@pve1:/var/log/ceph# ceph -v
ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable)
root@pve1:/var/log/ceph#

ceph-mgr.pve1.log:RuntimeError: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [1994910]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000449"}]
ceph-mgr.pve1.log:2020-04-27 10:29:05.598 7f7ea9b44700 0 mgr[zabbix] Failed to send.
ceph-mgr.pve1.log: File "/usr/share/ceph/mgr/zabbix/module.py", line 303, in send
ceph-mgr.pve1.log: File "/usr/share/ceph/mgr/zabbix/module.py", line 46, in send
ceph-mgr.pve1.log:RuntimeError: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [2012205]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000417"}]
root@pve1:/var/log/ceph#

 

[Ingo S]

Jepp, unfortunately ceph zabbix send isn't quite talkative when it comes to tracing errors. I remember having to set up some settings in Zabbix to make it accept the messages from ceph zabbix send. Maybe thats the part that is missing.

I will look it up, what it was and post it here. But this will take some time, because I'm out of office tomorrow.

 

[sunkroot]

Traceback (most recent call last):
File "/usr/share/ceph/mgr/zabbix/module.py", line 303, in send
zabbix.send(identifier, data)
File "/usr/share/ceph/mgr/zabbix/module.py", line 46, in send
stderr))
RuntimeError: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [1925062]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000416"}]

I've attached module.py and renamed it to module.py.txt so that i can upload it.