notice: RRDC/RRD update error

J

jic5760

Active Member
Nov 10, 2020
41
8
28
26
Not all methods on the forum work.

1. NTP Sync (Now, all nodes synced < 100ms)
2. systemctl stop rrdcached; rm -rf /var/lib/rrdcached/db; systemctl start rrdcached; systemctl restart pve-cluster
3. The rtc clock is also synchronized.

What's strange is that nodes that are itselves are marked with different times.

Even if you wait for a long time, it does not resolve itself and fills the DISK.

Code: 
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2670419]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2728807]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2670419]: status update time (5.517 seconds)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[3332871]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[655887]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2728807]: status update time (5.429 seconds)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: -1

# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph: 16.2.7
ceph-fuse: 16.2.7
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
 
Last edited:
Code: 
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2670419]: status update time (5.517 seconds)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[3332871]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[655887]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2728807]: status update time (5.429 seconds)

das sieht so aus als würden mehrere pvestatd prozesse gleichzeitig laufen (die sich dann natürlich in die quere kommen)

was sagt denn
Code: 
ps aux | grep '[p]vestatd'
?
 
As you said, there were several pvestatd.

Code: 
# killall pvestatd
# systemctl stop rrdcache
# rm -rf /var/lib/rrdcache/db
# systemctl start rrdcache
# systemctl start pvestatd

The noisy log has quieted down, but I'm still getting errors.

Code: 
Jan 21 22:22:31 pve-node-stor-08101 pvestatd[1449724]: rados_connect failed - Invalid argument
Jan 21 22:22:31 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08112: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-os: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local-lvm: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-jhomeuser: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08111: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-jhomeuser: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local-lvm: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-os: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local: -1
Jan 21 22:22:41 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08112: -1
Jan 21 22:22:42 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08111: -1
Jan 21 22:22:46 pve-node-stor-08101 pvestatd[1449724]: got timeout
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-jhomeuser: -1
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local: -1
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local-lvm: -1
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-os: -1
Jan 21 22:22:47 pve-node-stor-08101 pvestatd[1449724]: status update time (5.398 seconds)
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-jhomeuser: -1
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local-lvm: -1
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-os: -1
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local: -1
Jan 21 22:22:51 pve-node-stor-08101 pvestatd[1449724]: rados_connect failed - Invalid argument
Jan 21 22:22:51 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08112: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local-lvm: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-os: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-jhomeuser: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08111: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-jhomeuser: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local-lvm: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-os: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local: -1
 
This error message keeps filling up the disk.
Killing the pvestatd on node-01 stops the error, but it is not a solution.

Code: 
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/local: /var/lib/rrdcached/db/pve2-storage/node-01/local: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/node-01/local-lvm: -1
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/local-lvm: /var/lib/rrdcached/db/pve2-storage/node-01/local-lvm: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-jhomeuser: -1
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-jhomeuser: /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-jhomeuser: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-os: -1
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-os: /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-os: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)
 
are there still multiple pvestatds running? if yes, i'd investigate why and how they are started exactly.. pve should only ever have one instance of that running...
 
Huh... I shut down pvestatd on the node that says in the error for a few hours and the error went away.
However, this has happened before, and I think there will be problems again sometime.

Note that all servers are synchronized with NTP, usually with an error of fewer than 0.0001 seconds.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back