RRDC and RRD update errors

[chudak]

I had to replace my NUC, used the same RAM and SSD, proxmox booted up ok and VMs seem fine too, but I am seeing errors/warnings as below.
My node is "pve" but in the log points to "pve2-storage" (??)

I tried:
"hwclock --systohc"
and
systemctl stop rrdcached.service
rrdcached -P FLUSHALL
systemctl start rrdcached.service
but so far unsuccessfully.

Any help appreciated!
Thx

Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/VMs: -1
Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/VMs: /var/lib/rrdcached/db/pve2-storage/pve/VMs: illegal attempt to update using time 1600617319 when last update time is 1600639262 (minimum one second step)
Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: -1
Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: illegal attempt to update using time 1600617319 when last update time is 1600639262 (minimum one second step)

 

[chudak]

Warnings do seem to be gone without anything !!!
Never seen anything similar ...

Keeping this post for benefits to other users.

 

[Stoiko Ivanov]

These error messages usually happen if the system clock was off - if it was in the future at some point the rrd is written with this timestamp - once your clock gets synchronized to the current time the rrd refuses to updated (since the timestamp is now older than the one from when the clock was ahead) - once your system reaches the future timestamp rrds get updated normally again - and this error vanishes

I hope this explains it.

 

[Joris L.]

These error messages usually happen if the system clock was off - if it was in the future at some point the rrd is written with this timestamp - once your clock gets synchronized to the current time the rrd refuses to updated (since the timestamp is now older than the one from when the clock was ahead) - once your system reaches the future timestamp rrds get updated normally again - and this error vanishes

I hope this explains it.

i continuously have this issue, there is no fix in sight, the clock is in sync continuously, even for VM which are stopped

 

[Stoiko Ivanov]

i continuously have this issue, there is no fix in sight, the clock is in sync continuously, even for VM which are stopped

could you post a few loglines with the error-messages?

 

[Joris L.]

could you post a few loglines with the error-messages?

Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/138: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/134: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/137: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/136: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/135: -1

because i noticed people mention this, this is also a ZFS based system

 

[Stoiko Ivanov]

Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/135: -1

is a different error - (no mention of the timestamps )...

depending on how important the historical data is - I would probably move the cache directory to a safe place and restart the services (that way recording should start fresh):

cd /var/lib/rrdcached/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster

I hope this helps!

 

[Joris L.]

this appears to have resolved the issue, i hate solutions like this but this time it didn't matter too much, rrd data can be harvested regardless

 

[pcready.cl]

is a different error - (no mention of the timestamps )...

depending on how important the historical data is - I would probably move the cache directory to a safe place and restart the services (that way recording should start fresh):

cd /var/lib/rrdcached/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster

I hope this helps!

There is an error in the commands, the folder path does not exist, please use these in Proxmox v6.2-15:

cd /var/lib/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster

 

[Elfy]

Worked for me. Thanks @chudak and @pcready.cl!
I was also having a related issue where Proxmox GUI was logging me out every 30 seconds due to clocks being out of sync.

 

[marcuspoon]

Worked for me as well, thanks!! @chudak and @pcready.cl!
My version: 7.2.3.

 

[YoIm]

Worked for me too. Thx all.

 

[DocMAX]

doesnt work

 

[Joris L.]

no solutions presented in this thread work and keep working
this seems to be an issue which is not addressed as it only seems to happen on non-cluster proxmox ?

 

[DocMAX]

no solutions presented in this thread work and keep working
this seems to be an issue which is not addressed as it only seems to happen on non-cluster proxmox ?

fixed this. i had to do it on the node which presents the web gui.

 

[Joris L.]

unless you did something not documented here and something I've not tried yet, the issue will resurface

my system is a single-node Proxmox set-up, which I think is part of the issue resurfacing

 

[Joris L.]

There is an error in the commands, the folder path does not exist, please use these in Proxmox v6.2-15:

cd /var/lib/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster

thanks, did work for some time but the issue is back and resurfaces after multiple repeats of this sequence

 

[manzke89]

I have the same problem using 8.1.3, it doesn't display the graphs, so I'm storing the time column in pvesh get nodes/pve4/rrddata --timeframe hour

 

[DocMAX]

my mistake was to delete the rrd database on the node, and not the node which displays the web gui.

 

[bialy7]

On my Proxmox Version 8.1.4

Attention: This operation will clear all your chart history.

systemctl stop rrdcached
mv /var/lib/rrdcached/db /var/lib/rrdcached/db.bck
systemctl start rrdcached
systemctl restart pve-cluster

If all its ok you can remove db.bck

rm -rf /var/lib/rrdcached/db.bck