[SOLVED] RRD Graph on one cluster node is empty

[udo]

Hi,
on one cluster node the RRD-Graph is empty - for the host and all VMs.

The RRD-Files are actual but there isn't an output

root@proxmox2:~# ls -lsa /var/lib/rrdcached/db/pve2-vm/201
72 -rw-r----- 1 root root 67752 Dec 12 17:00 /var/lib/rrdcached/db/pve2-vm/201
file /var/lib/rrdcached/db/pve2-vm/201
/var/lib/rrdcached/db/pve2-vm/201: RRDTool DB version 0003

restart of pvestatd, and pveproxy don't help.

Any hints?

All nodes are running pve-manager/3.3-1/a06c9f73

Udo

 

[dietmar]

Is the system time in sync with other nodes?

 

[udo]

Hi Dietmar,
yes - must be! Because three of them are ceph-mons and the node with the missing graphs are also an ceph-mon.

Udo

 

[dietmar]

And rrdcached is running?

Or is that a newly added node?

 

[udo]

And rrdcached is running?

yes - I have also restarted the daemon

root      746430  0.0  0.0 300424  2116 ?        Ssl  16:45   0:01 /usr/bin/rrdcached -l unix:/var/run/rrdcached.sock -j /var/lib/rrdcached/journal/ -F -b /var/lib/rrdcached/db/ -B -p /var/run/rrdcached.pid

Or is that a newly added node?

No, it's proxmox2

pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M  14696   2014-10-15 18:14:27  proxmox3
   2   M  14688   2014-10-12 17:11:43  proxmox2
   3   M  14688   2014-10-12 17:11:43  proxmox1
   4   M  14688   2014-10-12 17:11:43  proxmox4
   5   M  14704   2014-11-08 07:10:53  proxmox5

Udo

 

[dietmar]

And there is no hint/error in syslog? (I get out of ideas now).

 

[udo]

Hi,
perhaps - the only one which looks different is

Dec 12 16:45:16 proxmox2 pmxcfs[731336]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/proxmox4: -3

But an remove of tis file don't help.
But I have found, that it can be happens during an lab-test: we cloned an fileserver (restore) and put ceph-storage from another VM on this VM.
e.g. in 297.conf add following:

virtio1: testceph:vm-245-disk-1,backup=no,size=3072G

The VMs are now clean - both use only the own storage.
I have deleted the rrd-data from both VMs (rm /var/lib/rrdcached/db/pve2-vm/297; rm /var/lib/rrdcached/db/pve2-vm/245) but still no graphs...

Udo

 

[wahmed]

Did you try to flush all rrd cache data and see if it works.
#rrdcached -P FLUSH

or

#rrdcached -P FLUSHALL

 

[udo]

Did you try to flush all rrd cache data and see if it works.
#rrdcached -P FLUSH

or

#rrdcached -P FLUSHALL

Hi Wasim,
great! After that (and restarted pvestatd+pveproxy but don't know if neccessary) I got graphs again!

Thanks!

Udo