[Solved] RRDC update error

S

snowman66

Active Member
Dec 1, 2010
254
1
38
I had to replace mainboard+cpu (because of unexpected server reboots) and now this errors shows up and pvestatd does not work:
Code: 
Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/node-pve1: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/node-pve1: /var/lib/rrdcached/db/pve2-node/node-pve1: illegal attempt to update using time 1380727729 when last update time is 1380730627 (minimum one second step) Oct 2 17:28:49 node-pve1 pvestatd[21060]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future Oct 2 17:28:49 node-pve1 pvestatd[21060]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/104: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/104: /var/lib/rrdcached/db/pve2-vm/104: illegal attempt to update using time 1380727729 when last update time is 1380730388 (minimum one second step) Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/103: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/103: /var/lib/rrdcached/db/pve2-vm/103: illegal attempt to update using time 1380727729 when last update time is 1380730388 (minimum one second step) Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/106: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/106: /var/lib/rrdcached/db/pve2-vm/106: illegal attempt to update using time 1380727729 when last update time is 1380730388 (minimum one second step) Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/102: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/102: /var/lib/rrdcached/db/pve2-vm/102: illegal attempt to update using time 1380727729 when last update time is 1380730627 (minimum one second step) Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/101: /var/lib/rrdcached/db/pve2-vm/101: illegal attempt to update using time 1380727729 when last update time is 1380730388 (minimum one second step) Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/105: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/105: /var/lib/rrdcached/db/pve2-vm/105: illegal attempt to update using time 1380727729 when last update time is 1380730388 (minimum one second step) Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1 Oct 2 17:28:49 node-pve1 pmxcfs[2262]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/100: /var/lib/rrdcached/db/pve2-vm/100: illegal attempt to update using time 1380727729 when last update time is 1380730627 (minimum one second step)
pve version:
Code: 
proxmox-ve-2.6.32: 3.1-111 (running kernel: 2.6.32-24-pve)pve-manager: 3.1-14 (running version: 3.1-14/d914b943)pve-kernel-2.6.32-20-pve: 2.6.32-100pve-kernel-2.6.32-24-pve: 2.6.32-111pve-kernel-2.6.32-22-pve: 2.6.32-107pve-kernel-2.6.32-23-pve: 2.6.32-109lvm2: 2.02.98-pve4clvm: 2.02.98-pve4corosync-pve: 1.4.5-1openais-pve: 1.1.4-3libqb0: 0.11.1-2redhat-cluster-pve: 3.2.0-2resource-agents-pve: 3.9.2-4fence-agents-pve: 4.0.0-2pve-cluster: 3.0-7qemu-server: 3.1-4pve-firmware: 1.0-23libpve-common-perl: 3.0-6libpve-access-control: 3.0-6libpve-storage-perl: 3.0-13pve-libspice-server1: 0.12.4-2vncterm: 1.1-4vzctl: 4.0-1pve3vzprocps: 2.0.11-2vzquota: 3.1-2pve-qemu-kvm: 1.4-17ksm-control-daemon: 1.1-1glusterfs-client: 3.4.0-2
any idea?
 
Last edited:
Re: RRDC update error

seems that running service rrdcached restart helps but some errors still exists:


Code: 
Oct 2 18:00:36 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future
Oct 2 18:00:36 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future
Oct 2 18:00:46 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future
Oct 2 18:00:46 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future
Oct 2 18:00:56 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future
Oct 2 18:00:56 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future
Oct 2 18:01:06 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future
Oct 2 18:01:06 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future
Oct 2 18:01:16 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future
Oct 2 18:01:16 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future
Oct 2 18:01:26 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future
Oct 2 18:01:26 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future
Oct 2 18:01:36 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/101.conf' modified in future
Oct 2 18:01:36 node-pve1 pvestatd[23151]: WARNING: file '/etc/pve/nodes/node-pve1/qemu-server/103.conf' modified in future
Oct 2 18:01:36 node-pve1 rrdcached[22803]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/node-pve1/backupNAS) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/node-pve1/backupNAS: illegal attempt to update using time 1380729396 when last update time is 1380730627 (minimum one second step))
Oct 2 18:01:36 node-pve1 rrdcached[22803]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/node-pve1/local) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/node-pve1/local: illegal attempt to update using time 1380729396 when last update time is 1380730627 (minimum one second step))
Oct 2 18:01:39 node-pve1 pveproxy[2533]: worker 20535 finished
 
Re: RRDC update error

After stop/start VM 101 & VM 103 errors went away or it was just a luck - not sure.

Code: 
Oct  2 18:25:10 node-pve1 kernel: device tap103i0 entered promiscuous mode
Oct  2 18:25:10 node-pve1 kernel: vmbr0: port 3(tap103i0) entering forwarding state
Oct  2 18:25:11 node-pve1 pvedaemon[23844]: <root@pam> end task UPID:node-pve1:0000629B:0....8E6:qmstart:103:root@pam: OK
Oct  2 18:25:13 node-pve1 ntpd[22582]: Listen normally on 13 tap103i0 fe80::3ce1:c9ff:fe9b:bafd UDP 123
Oct  2 18:25:13 node-pve1 ntpd[22582]: peers refreshed

What is the meaning of the last two lines?
 
Re: RRDC update error

snowman66 said:
After stop/start VM 101 & VM 103 errors went away or it was just a luck - not sure.
Click to expand...

The time difference (1380730388 - 1380727729 => 2659) is less than one hour, so I guess the error simply vanished after that time. I assume the system time on the old node was wrong (in future).
 
Hi,

just have the same issue here (after a mainboard replacement).
Managed the RRD problem with a rrdcached restart.
But the logs like this are still flooding syslog:
Code: 
Aug 29 19:22:51 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future
Aug 29 19:23:01 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future
Aug 29 19:23:11 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future
Aug 29 19:23:21 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future
Aug 29 19:23:31 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future
Aug 29 19:23:41 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future
Aug 29 19:23:51 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future

The fun trick is that:
Code: 
root@hv-co-01-pareq6:~# tail -n1 /var/log/syslog ; echo ; date ; echo ; stat /etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf
Aug 29 19:28:21 hv-co-01-pareq6 pvestatd[6231]: file '/etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf' modified in future

Tue Aug 29 19:28:26 CEST 2017

  File: /etc/pve/nodes/hv-co-01-pareq6/qemu-server/100.conf
  Size: 360       	Blocks: 1          IO Block: 4096   regular file
Device: 2dh/45d	Inode: 24          Links: 1
Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (   33/www-data)
Access: 2017-08-29 19:04:50.000000000 +0200
Modify: 2017-08-29 19:04:50.000000000 +0200
Change: 2017-08-29 19:04:50.000000000 +0200
 Birth: -

So, no, it's not "modified in future".
About the time, the server is NTP synced ; it's possible that it was not synced when it booted (systemd's timedatectl mysteries) since the mainboard has just been replaced, but it is now.

How to get rid of that ? (restarting pvestatd doesn't do anything).
 
Symbol said:
About the time, the server is NTP synced ; it's possible that it was not synced when it booted (systemd's timedatectl mysteries) since the mainboard has just been replaced, but it is now.
Click to expand...
OK, it's because of that: pve daemons were started in "Aug 30", and only after systemd decided to properly set the date:
Code: 
Aug 30 16:20:08 hv-co-01-pareq6 systemd[1]: Started PVE VM Manager.
[...]
Aug 30 16:20:17 hv-co-01-pareq6 systemd[1]: Started User Manager for UID 0.
Aug 29 16:18:48 hv-co-01-pareq6 systemd[1]: Time has been changed
Aug 29 16:18:48 hv-co-01-pareq6 systemd-timesyncd[1036]: Synchronized to time server
I understand that it will spontaneously fix by itself tomorrow.
However, what to do in the meanwhile, and isn't there something weird here, since the file as shown by ls isn't "in the future" ?

For a server, systemd stuff clearly lacks an ntpdate at bootime.
 
I found similar problem with rrd updates.
I inspect /var/lib/rrdcached/ for long-time not-modified files and delete this:

find /var/lib/rrdcached/ -type f -mtime +5 -delete;

After this new files was created automatically, and logs stay clear; no any restarts need;

With last update to pve 5.4-14 this problem also affect one container with UT99 game server:
https://aminux.wordpress.com/2020/05/09/proxmox-rrdcache-issue/

game clients suspended and stuck while this rrd-update logs appear;
fix this issue also solve starnge problem with game server;
 
  • Like
Reactions: devilkin
Aminux said:
I found similar problem with rrd updates.
I inspect /var/lib/rrdcached/ for long-time not-modified files and delete this:

find /var/lib/rrdcached/ -type f -mtime +5 -delete;

After this new files was created automatically, and logs stay clear; no any restarts need;

With last update to pve 5.4-14 this problem also affect one container with UT99 game server:
https://aminux.wordpress.com/2020/05/09/proxmox-rrdcache-issue/

game clients suspended and stuck while this rrd-update logs appear;
fix this issue also solve starnge problem with game server;
Click to expand...

Thanks, that helped. Somewhat.

The issue keeps on resurfacing for cpu and cgroups.
 
Hello there
I ll add to your issues mine too which happened today after changing ram dimms. I also set the bios the time correctly (probably needs battery replacement)

My logs are as follows
Code: 
Nov 23 11:15:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 11:15:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 11:15:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/100: /var/lib/rrdcached/db/pve2-vm/100: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/200: /var/lib/rrdcached/db/pve2-vm/200: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/101: /var/lib/rrdcached/db/pve2-vm/101: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local: /var/lib/rrdcached/db/pve2-storage/pve1/local: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/100: /var/lib/rrdcached/db/pve2-vm/100: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/200: /var/lib/rrdcached/db/pve2-vm/200: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/101: /var/lib/rrdcached/db/pve2-vm/101: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local: /var/lib/rrdcached/db/pve2-storage/pve1/local: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)

I already tried starting and shutting down my one Vm and 2 lxcs and running service rrdcached restart. Also find /var/lib/rrdcached/ -type f -mtime +5 -delete;

As for time/ntp under pve1 time option the only thing you can set is Europe/your_country nothing else

Nothing seems to help to stop that flooding sys log. Also after a while it logs me out with the << connection error 401 permission denied - invalid pve ticket>>

Also I can t find out why it mentions pve2-vm/200: -1 / pve2-storage/pve1/local: Where did it find pve2 in a one node only system
 
Last edited:
ieronymous said:
Hello there
I ll add to your issues mine too which happened today after changing ram dimms. I also set the bios the time correctly (probably needs battery replacement)

My logs are as follows
Code: 
Nov 23 11:15:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 11:15:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 11:15:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/100: /var/lib/rrdcached/db/pve2-vm/100: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/200: /var/lib/rrdcached/db/pve2-vm/200: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/101: /var/lib/rrdcached/db/pve2-vm/101: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 11:15:05 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local: /var/lib/rrdcached/db/pve2-storage/pve1/local: illegal attempt to update using time 1606122905 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/100: /var/lib/rrdcached/db/pve2-vm/100: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/200: /var/lib/rrdcached/db/pve2-vm/200: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/101: /var/lib/rrdcached/db/pve2-vm/101: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 11:15:15 pve1 pmxcfs[1463]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve1/local: /var/lib/rrdcached/db/pve2-storage/pve1/local: illegal attempt to update using time 1606122915 when last update time is 1606128399 (minimum one second step)

I already tried starting and shutting down my one Vm and 2 lxcs and running service rrdcached restart. Also find /var/lib/rrdcached/ -type f -mtime +5 -delete;

Nothing seems to help to stop that flooding sys log. Also after a while it logs me out with the << connection error 401 permission denied - invalid pve ticket>>

Also I can t find out why it mentions pve2-vm/200: -1 / pve2-storage/pve1/local: Where did it find pve2 in a one node only system
Click to expand...
Maybe this post can help https://forum.proxmox.com/threads/rrdc-and-rrd-update-errors.76219/post-345702
 
Joris L. said:
Maybe this post can help https://forum.proxmox.com/threads/rrdc-and-rrd-update-errors.76219/post-345702
Click to expand...
Thank you for your quick reply. Well after issuing the commands this is what i get afterwards

Code: 
Nov 23 11:59:24 pve1 systemd[1]: Starting LSB: start or stop rrdcached...
Nov 23 11:59:24 pve1 rrdcached[13250]: rrdcached started.
Nov 23 11:59:24 pve1 systemd[1]: Started LSB: start or stop rrdcached.
Nov 23 11:59:39 pve1 systemd[1]: Stopping The Proxmox VE cluster filesystem...
Nov 23 11:59:39 pve1 pmxcfs[1375]: [main] notice: teardown filesystem
Nov 23 11:59:39 pve1 systemd[6633]: etc-pve.mount: Succeeded.
Nov 23 11:59:39 pve1 systemd[1]: etc-pve.mount: Succeeded.
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: State 'stop-sigterm' timed out. Killing.
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: Killing process 1375 (pmxcfs) with signal SIGKILL.
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: Main process exited, code=killed, status=9/KILL
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: Failed with result 'timeout'.
Nov 23 11:59:49 pve1 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Nov 23 11:59:49 pve1 systemd[1]: Starting The Proxmox VE cluster filesystem...
Nov 23 11:59:51 pve1 pve-ha-lrm[1692]: updating service status from manager failed: Connection refused
Nov 23 11:59:51 pve1 pve-firewall[1503]: status update error: Connection refused
Nov 23 11:59:51 pve1 pve-firewall[1503]: firewall update time (10.014 seconds)
Nov 23 11:59:51 pve1 pve-firewall[1503]: status update error: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[1] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[2] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[3] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[4] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: status update error: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: status update time (10.016 seconds)
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[1] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[2] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[3] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[4] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: status update error: Connection refused
Nov 23 11:59:51 pve1 systemd[1]: Started The Proxmox VE cluster filesystem.
Nov 23 11:59:51 pve1 systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Nov 23 12:00:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:00:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:00:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:00:50 pve1 pveproxy[1685]: worker exit
Nov 23 12:00:50 pve1 pveproxy[1683]: worker 1685 finished
Nov 23 12:00:50 pve1 pveproxy[1683]: starting 1 worker(s)
Nov 23 12:00:50 pve1 pveproxy[1683]: worker 14618 started
Nov 23 12:01:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:01:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:01:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:02:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:02:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:02:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:03:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:03:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:03:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:03:19 pve1 systemd[1]: Starting Cleanup of Temporary Directories...
Nov 23 12:03:19 pve1 systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Nov 23 12:03:19 pve1 systemd[1]: Started Cleanup of Temporary Directories.
Nov 23 12:03:24 pve1 pvedaemon[1531]: <root@pam> successful auth for user 'root@pam'
Nov 23 12:04:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:04:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:04:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:05:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:05:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:05:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:06:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:06:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:06:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:07:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:07:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:07:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:08:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:08:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:08:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:09:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:09:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:09:01 pve1 systemd[1]: Started Proxmox VE replication runner.

is it normal this loop of replication runner and the message afterwards that the pvesr.service: succeded ??

PS In case of any interest I am using proxmox upon the zfs file system on a raid 1 array


New Update after a restart back to basics

Code: 
Nov 23 12:38:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:38:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:38:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 12:38:16 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 12:38:16 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 12:38:16 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 12:38:16 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 12:38:16 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 12:38:16 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 12:38:26 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 12:38:26 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 12:38:26 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 12:38:26 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 12:38:26 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 12:38:26 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 12:38:36 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 12:38:36 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 12:38:36 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 12:38:36 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 12:38:36 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 12:38:36 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 12:38:46 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 12:38:46 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 12:38:46 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 12:38:46 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 12:38:46 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
Nov 23 12:38:46 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 12:38:56 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 12:38:56 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
Nov 23 12:38:56 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1
Nov 23 12:38:56 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Nov 23 12:38:56 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local-zfs: -1
Nov 23 12:38:56 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve1/local: -1
 
Last edited:
ieronymous said:
Thank you for your quick reply. Well after issuing the commands this is what i get afterwards

Code: 
Nov 23 11:59:24 pve1 systemd[1]: Starting LSB: start or stop rrdcached...
Nov 23 11:59:24 pve1 rrdcached[13250]: rrdcached started.
Nov 23 11:59:24 pve1 systemd[1]: Started LSB: start or stop rrdcached.
Nov 23 11:59:39 pve1 systemd[1]: Stopping The Proxmox VE cluster filesystem...
Nov 23 11:59:39 pve1 pmxcfs[1375]: [main] notice: teardown filesystem
Nov 23 11:59:39 pve1 systemd[6633]: etc-pve.mount: Succeeded.
Nov 23 11:59:39 pve1 systemd[1]: etc-pve.mount: Succeeded.
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: State 'stop-sigterm' timed out. Killing.
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: Killing process 1375 (pmxcfs) with signal SIGKILL.
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: Main process exited, code=killed, status=9/KILL
Nov 23 11:59:49 pve1 systemd[1]: pve-cluster.service: Failed with result 'timeout'.
Nov 23 11:59:49 pve1 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Nov 23 11:59:49 pve1 systemd[1]: Starting The Proxmox VE cluster filesystem...
Nov 23 11:59:51 pve1 pve-ha-lrm[1692]: updating service status from manager failed: Connection refused
Nov 23 11:59:51 pve1 pve-firewall[1503]: status update error: Connection refused
Nov 23 11:59:51 pve1 pve-firewall[1503]: firewall update time (10.014 seconds)
Nov 23 11:59:51 pve1 pve-firewall[1503]: status update error: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[1] failed: Connection refused
Nov 23 11:59:51 pve1 pvestatd[1506]: ipcc_send_rec[2] failed: Connection refused
.....
Nov 23 12:08:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:08:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:09:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:09:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:09:01 pve1 systemd[1]: Started Proxmox VE replication runner.

is it normal this loop of replication runner and the message afterwards that the pvesr.service: succeded ??

PS In case of any interest I am using proxmox upon the zfs file system on a raid 1 array


New Update after a restart back to basics

Code: 
Nov 23 12:38:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 12:38:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 12:38:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve1: -1
Nov 23 12:38:06 pve1 pmxcfs[1356]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/100: -1
....
[/QUOTE]

[/QUOTE]
Yeah, i do remember a lot of noise as well.

I think what you are now seeing is exclusively due to filesystem permission issues or non existent file.
Try something like: touch /var/lib/rrdcached/db/pve2-vm/100
and see what happens
Click to expand...
Yeah, i do remember a lot of noise as well.

I think what you are now seeing is exclusively due to filesystem permission issues or non existent file.
Try something like: touch /var/lib/rrdcached/db/pve2-vm/100
and see what happens[/CODE]
 
Joris L. said:
Yeah, i do remember a lot of noise as well.

I think what you are now seeing is exclusively due to filesystem permission issues or non existent file.
Try something like: touch /var/lib/rrdcached/db/pve2-vm/100
and see what happens[/CODE]
Click to expand...
For several minutes now (by the way i run the touch line for all vm and lxcs) i have the below trilogy all over again. Is it a normal procedure? What the hell is it trying to replicate?

Code: 
Nov 23 16:05:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 16:06:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 16:06:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 16:06:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 16:07:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 16:07:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 16:07:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 16:08:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 16:08:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 16:08:01 pve1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 16:09:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 16:09:01 pve1 systemd[1]: pvesr.service: Succeeded.
Nov 23 16:09:01 pve1 systemd[1]: Started Proxmox VE replication runner.

By the way the one VM and two lxcs I have are shut down
 
Last edited:
It happened in my server (6.2-4) right now. Today is Dec, 29, and the server was stopped at Dec, 26 at 09:35...

The only solution was a server reboot.

Weird.
 
Had the same errors and just resetting the rrdcached dbs helped if you don´t want to wait ;-)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back