RRDC and RRD update errors

chudak

Well-Known Member
May 11, 2019
322
16
58
I had to replace my NUC, used the same RAM and SSD, proxmox booted up ok and VMs seem fine too, but I am seeing errors/warnings as below.
My node is "pve" but in the log points to "pve2-storage" (??)

I tried:
"hwclock --systohc"
and
systemctl stop rrdcached.service
rrdcached -P FLUSHALL
systemctl start rrdcached.service
but so far unsuccessfully.

Any help appreciated!
Thx

Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/VMs: -1
Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/VMs: /var/lib/rrdcached/db/pve2-storage/pve/VMs: illegal attempt to update using time 1600617319 when last update time is 1600639262 (minimum one second step)
Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: -1
Sep 20 08:55:19 pve pmxcfs[1108]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: illegal attempt to update using time 1600617319 when last update time is 1600639262 (minimum one second step)
 
These error messages usually happen if the system clock was off - if it was in the future at some point the rrd is written with this timestamp - once your clock gets synchronized to the current time the rrd refuses to updated (since the timestamp is now older than the one from when the clock was ahead) - once your system reaches the future timestamp rrds get updated normally again - and this error vanishes :)

I hope this explains it.
 
  • Like
Reactions: jammin and chudak
These error messages usually happen if the system clock was off - if it was in the future at some point the rrd is written with this timestamp - once your clock gets synchronized to the current time the rrd refuses to updated (since the timestamp is now older than the one from when the clock was ahead) - once your system reaches the future timestamp rrds get updated normally again - and this error vanishes :)

I hope this explains it.
i continuously have this issue, there is no fix in sight, the clock is in sync continuously, even for VM which are stopped
 
Last edited:
i continuously have this issue, there is no fix in sight, the clock is in sync continuously, even for VM which are stopped
could you post a few loglines with the error-messages?
 
could you post a few loglines with the error-messages?
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/138: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/134: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/137: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/136: -1
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/135: -1

because i noticed people mention this, this is also a ZFS based system
 
Last edited:
Oct 22 18:03:01 pmx pmxcfs[386322]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/135: -1
is a different error - (no mention of the timestamps )...

depending on how important the historical data is - I would probably move the cache directory to a safe place and restart the services (that way recording should start fresh):
Code:
cd /var/lib/rrdcached/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster

I hope this helps!
 
is a different error - (no mention of the timestamps )...

depending on how important the historical data is - I would probably move the cache directory to a safe place and restart the services (that way recording should start fresh):
Code:
cd /var/lib/rrdcached/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster

I hope this helps!

There is an error in the commands, the folder path does not exist, please use these in Proxmox v6.2-15:

Bash:
cd /var/lib/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster
 
Worked for me. Thanks @chudak and @pcready.cl!
I was also having a related issue where Proxmox GUI was logging me out every 30 seconds due to clocks being out of sync.
 
There is an error in the commands, the folder path does not exist, please use these in Proxmox v6.2-15:

Bash:
cd /var/lib/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster
thanks, did work for some time but the issue is back and resurfaces after multiple repeats of this sequence
 
I have the same problem using 8.1.3, it doesn't display the graphs, so I'm storing the time column in pvesh get nodes/pve4/rrddata --timeframe hour
 
On my Proxmox Version 8.1.4

Attention: This operation will clear all your chart history.
Bash:
systemctl stop rrdcached
mv /var/lib/rrdcached/db /var/lib/rrdcached/db.bck
systemctl start rrdcached
systemctl restart pve-cluster

If all its ok you can remove db.bck
Bash:
rm -rf /var/lib/rrdcached/db.bck
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!