PVE 5.3-7: Graphs stopped arbitrarily

[cloudguy]

Hello,

I have a 3 node cluster running PVE 5.3-7. One of the nodes stopped showing telemetry (CPU, memory, etc.) graphs for the node and VMs. I tried to restart pvestatd, pveproxy processes however that didn't help. Remaining two nodes are showing graphs without issue.

Nothing was done on the node leading up to the event. I can't find anything notable in messages, syslog or journalctl. The only thing I can see is a PVE backup job running from 00:01 -> 01:59 (scheduled daily).



Can someone suggest how best to troubleshoot this?

# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-7 (running version: 5.3-7/e8ed1e22)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-19
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-35
libqb0: 1.0.3-1~bop
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-1
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-33
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-44
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

 

[Stoiko Ivanov]

Did the graphs continue eventually?

Check the logs during the time when the graph was not written - look for messages from pvestatd, pmxcfs and rrdcached

It might be that pvestatd (which writes the graphs) does not finish its run, since your storage-network is overloaded during the backup

hope this helps

 

[cloudguy]

Unfortunately no. I ended up rebuilding the machine. Fine now but very strange that it just stopped graphing at that time.