RRDTool does not update the containers' graphs. Physical server graphs are ok...

[pieterpap]

Hey guys,

I have a physical server (standalone - no cluster) and around 80 containers.

I can see the monitor graphs created by RRDTool for the physical server, but I have some issues with those on the each individual container: Even though the graph is created, it is not updated. In one case, I can not even see the graph itself, the browser shows the broken image icon.

This is my pveversion -v output:

pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-17-pve
pve-kernel-2.6.32-13-pve: 2.6.32-72
pve-kernel-2.6.32-17-pve: 2.6.32-83
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-34
qemu-server: 2.0-39
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-4
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3

A quick look at tail /var/log/syslog gives me the result below:

Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/111
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/9
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/10111
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/10128
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/98
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/10245
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/10109
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/1000
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/10140
Jun  6 19:38:24 slot03 pmxcfs[89659]: [status] crit: RRD update error: unknown/wrong key pve2.3-vm/10009

I have already tried to restart the pvestatd and pvemanager services with no luck.

Thanks for guiding,

Petros

 

[dietmar]

Does it help if you reboot the server?

 

[pieterpap]

Does it help if you reboot the server?

It is a production server and I try to keep it as the last resort.

Actually, the last time I have done it I had to update to the latest pve-firmware, as apparently the kernel -17 was unable to locate the NetXtreme firmware. It was quite a mess, that's why I try to leave it last.

Is the proxmox kernel tied to a specific RRDTool version? I can try to upgrade it...

Thanks again,

Petros

 

[dietmar]

Try:

# service pve-cluster restart
# service pvestatd restart
# service pvedaemon restart
# service apache2 restart

 

[pieterpap]

Hi dietmar,

Firstly, thanks for having a look at the issue

Apparently, the server is a stand-alone one and not a part of a cluster.
service pve-cluster restart
/var/log# service pve-cluster restart
Restarting pve cluster filesystem: pve-cluster[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)
(warning).

The rest restarted OK, but still no graph update on the containers.

R there any specific daemons that need to run on the containers for the graphs to update?

Thanks,

Petros

 

[dietmar]

OK, seems you do not use latest available packages. You need pve-cluster >= 1.0-36