RRD update errors

K

kevdog

Active Member
Feb 28, 2019
6
0
41
54
Hi - Thanks for any help

I just installed proxmox -- yeah.
I haven'f configured anything other than setting up the OS. No virtual machines or anything as such.
My logs are being littered with this:

Code: 
Feb 28 15:28:08 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1551389288 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:08 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1551389288 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:08 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: /var/lib/rrdcached/db/pve2-storage/pve/local: illegal attempt to update using time 1551389288 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:18 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1551389298 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:18 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1551389298 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:18 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: /var/lib/rrdcached/db/pve2-storage/pve/local: illegal attempt to update using time 1551389298 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:28 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1551389308 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:28 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: /var/lib/rrdcached/db/pve2-storage/pve/local: illegal attempt to update using time 1551389308 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:28 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1551389308 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:38 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1551389318 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:38 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1551389318 when last update time is 1551406335 (minimum one second step)
Feb 28 15:28:38 pve pmxcfs[1475]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: /var/lib/rrdcached/db/pve2-storage/pve/local: illegal attempt to update using time 1551389318 when last update time is 1551406335 (minimum one second step)


My network time is up to date and I've verified the ntpd is working and querying. I've seen other threads about this topic in the past. How do I resolved this issue.
 
did you change the time at any point?

you can try to wipe the rrd database, or you can wait until the time catches up with the timestamp in the rrd database
 
Note -- I'm not sure if I wiped the rrd database since I tried a bunch of commands. Eventually hours later the error went away and I suspect the time caught up to the timestamp in the database. The only time I changed the system time was on the motherboard prior to loading proxmox. System was new and time listed on MB was not correct as unit was new. Other than this step, I never changed the actual time on the system. Anyway I'm going to declare the issue as resolved. Thanks for responding.
 
Hi
I have the same problem.
Fresh Install (proxmox-ve_5.4-1.iso) and I have the following errors

Code: 
May 15 21:50:07 srv-proxmox pmxcfs[1324]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local: /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local: illegal attempt to update using time 1557953407 when last update time is 1557957600 (minimum one second step)
May 15 21:50:17 srv-proxmox pmxcfs[1324]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/srv-proxmox: -1
May 15 21:50:17 srv-proxmox pmxcfs[1324]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/srv-proxmox: /var/lib/rrdcached/db/pve2-node/srv-proxmox: illegal attempt to update using time 1557953417 when last update time is 1557957600 (minimum one second step)
May 15 21:50:17 srv-proxmox pmxcfs[1324]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local-lvm: -1
May 15 21:50:17 srv-proxmox pmxcfs[1324]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local-lvm: /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local-lvm: illegal attempt to update using time 1557953417 when last update time is 1557957600 (minimum one second step)
May 15 21:50:17 srv-proxmox pmxcfs[1324]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local: -1
May 15 21:50:17 srv-proxmox pmxcfs[1324]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local: /var/lib/rrdcached/db/pve2-storage/srv-proxmox/local: illegal attempt to update using time 1557953417 when last update time is 1557957600 (minimum one second step)

1557953417 = 15/5/2019 à 20:50:17
1557957600 = 15/5/2019 à 22:00:00

What did you do ?
 
We had the same issue on 2 clusters, on all newly fresh installed nodes.
As above stated, wait until the time catches up with the timestamp in the rrd database and the errors will be gone
 
Same problem
Fresh install
Please help. Thanks.

Code: 
proxmox-ve: 6.0-2 (running kernel: 5.0.18-1-pve)
pve-manager: 6.0-5 (running version: 6.0-5/f8a710d7)
pve-kernel-5.0: 6.0-6
pve-kernel-helper: 6.0-6
pve-kernel-5.0.18-1-pve: 5.0.18-2
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-3
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-7
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-6
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

Code: 
Aug 11 00:31:01 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Aug 11 00:31:01 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1565454661 when last update time is 1565482387 (minimum one second step)
Aug 11 00:31:01 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: -1
Aug 11 00:31:01 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1565454661 when last update time is 1565464331 (minimum one second step)
Aug 11 00:31:01 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local: -1
Aug 11 00:31:12 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Aug 11 00:31:12 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1565454671 when last update time is 1565482387 (minimum one second step)
Aug 11 00:31:12 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local: -1
Aug 11 00:31:12 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: -1
Aug 11 00:31:12 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1565454672 when last update time is 1565464331 (minimum one second step)
Aug 11 00:31:21 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Aug 11 00:31:21 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1565454681 when last update time is 1565482387 (minimum one second step)
Aug 11 00:31:21 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: -1
Aug 11 00:31:21 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1565454681 when last update time is 1565464331 (minimum one second step)
Aug 11 00:31:21 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local: -1
Aug 11 00:31:31 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Aug 11 00:31:31 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1565454691 when last update time is 1565482387 (minimum one second step)
Aug 11 00:31:31 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local: -1
Aug 11 00:31:31 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: -1
Aug 11 00:31:31 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1565454691 when last update time is 1565464331 (minimum one second step)
Aug 11 00:31:41 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Aug 11 00:31:41 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1565454701 when last update time is 1565482387 (minimum one second step)
Aug 11 00:31:41 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local: -1
Aug 11 00:31:41 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: -1
Aug 11 00:31:41 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1565454701 when last update time is 1565464331 (minimum one second step)
Aug 11 00:31:51 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Aug 11 00:31:51 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: /var/lib/rrdcached/db/pve2-node/pve: illegal attempt to update using time 1565454711 when last update time is 1565482387 (minimum one second step)
Aug 11 00:31:51 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: -1
Aug 11 00:31:51 pve pmxcfs[1587]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve/local-zfs: illegal attempt to update using time 1565454711 when last update time is 1565464331 (minimum one second step)
Aug 11 00:31:51 pve pmxcfs[1587]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/local: -1

Code: 
root@pve:~# timedatectl
               Local time: Sun 2019-08-11 00:33:36 CST
           Universal time: Sat 2019-08-10 16:33:36 UTC
                 RTC time: Sat 2019-08-10 16:33:36
                Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
 
seems your machine had a clock jump:
`Sat Aug 10 18:31:31 2019` (time the node tries to write the rrd)
`Sun Aug 11 02:13:07 2019` (time the rrd was last written)

this can have multiple reasons:
* the hw-clock is completely off and after booting the system uses it for reference?
* usually the hw-clock on linux systems like PVE is expected to be in UTC and not localtime

* compare the output of `hwclock` and `date`

hope this helps!
 
Solved Edit:
I left it like this for the night and went to bed. It resolved on its own after about 3 hours. I rebooted to check and didn't see any errors. I think rrdcached takes a little longer to detect time and fix it in memory.
Thank you.

Sep 04 06:27:25 pve1 pmxcfs[1113]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Sep 04 06:27:25 pve1 pmxcfs[1113]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local: /var/lib/rrdcached/db/pve2-storage/pve1/local: illegal attempt to update using time 1599190045 when last update time is 1599190046 (minimum one second step)
Sep 04 06:28:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Sep 04 06:28:01 pve1 systemd[1]: pvesr.service: Succeeded.
Sep 04 06:28:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Sep 04 06:28:01 pve1 pvedaemon[18215]: <root@pam> successful auth for user 'root@pam'
Sep 04 06:28:26 pve1 pvedaemon[17918]: worker exit
Sep 04 06:28:26 pve1 pvedaemon[1290]: worker 17918 finished
Sep 04 06:28:26 pve1 pvedaemon[1290]: starting 1 worker(s)
Sep 04 06:28:26 pve1 pvedaemon[1290]: worker 22178 started
Sep 04 06:29:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Sep 04 06:29:01 pve1 systemd[1]: pvesr.service: Succeeded.

---
Hi @Stoiko Ivanov ,
I am having the same problem. How can we solve it?

1599178084173.png
1599178111890.png
root@pve1:~# timedatectl
Local time: Fri 2020-09-04 03:07:40 +03
Universal time: Fri 2020-09-04 00:07:40 UTC
RTC time: Fri 2020-09-04 00:07:40
Time zone: Europe/Istanbul (+03, +0300)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no

root@pve1:~# hwclock
2020-09-04 03:10:09.436958+03:00
root@pve1:~# date
Fri 04 Sep 2020 03:10:12 AM +03

All hours are the same, I wrote the pictures and articles a little late
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom