RRD update error

[Hai]

Hello,

Started getting this error on my cluster recently, whats the best way to fix?
In addition i get the 'Invalid PVE Ticket' error message and get kicked out of the GUI.
Would be nice to get a solution that doesn't require downtime or doesn't break cluster/nodes.

Jan 26 15:17:49 examplenode pmxcfs[2808]: [status] notice: rJan 26 15:17:49 examplenode pmxcfs[2808]: [status] notice: received log
Jan 26 15:17:50 examplenode pmxcfs[2808]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/examplenode: -1
Jan 26 15:17:50 examplenode pmxcfs[2808]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/examplenode: /var/lib/rrdcached/db/pve2-node/examplenode: illegal attempt to update using time 1769433470 when last update time is 4045247863 (minimum one second step)eceived log

 

[mfederanko]

You somehow managed to create log entries for the year 2098, which the time series db rightly complains about. They should be safe to just delete, you can optionally install rrdtool apt install rrdtool and create a dump of the db/remove the offending line. See man 1 rrdtool.

Regarding the invalid ticket, that also seems like a system time issue. It might be worth checking the time of your node(s) with (date -I) and verify if that is correct.

 

[Hai]

You somehow managed to create log entries for the year 2098, which the time series db rightly complains about. They should be safe to just delete, you can optionally install rrdtool apt install rrdtool and create a dump of the db/remove the offending line. See man 1 rrdtool.

Regarding the invalid ticket, that also seems like a system time issue. It might be worth checking the time of your node(s) with (date -I) and verify if that is correct.

Thanks,


I ran theese commands:

authkey.pub was also somehow stamped at 2098, this command did reset its date to current one. This fixed the loggin out of GUI (Invalid PVE ticket).

touch /etc/pve/authkey.pub

This fixed the log entry errors:

systemctl stop pvestatd rrdcached

mv /var/lib/rrdcached/db /var/lib/rrdcached/db.bak.$(date +%F_%H%M%S)
mkdir -p /var/lib/rrdcached/db
chown root:root /var/lib/rrdcached/db

systemctl start rrdcached pvestatd

All without downtime or anything messing up, safe to use on running cluster with live VMs