notice: RRDC/RRD update error

[jic5760]

Not all methods on the forum work.

1. NTP Sync (Now, all nodes synced < 100ms)
2. systemctl stop rrdcached; rm -rf /var/lib/rrdcached/db; systemctl start rrdcached; systemctl restart pve-cluster
3. The rtc clock is also synchronized.

What's strange is that nodes that are itselves are marked with different times.

Even if you wait for a long time, it does not resolve itself and fills the DISK.

Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2670419]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2728807]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: illegal attempt to update using time 1642754655 when last update time is 1642754655 (minimum one second step)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2670419]: status update time (5.517 seconds)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[3332871]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[655887]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2728807]: status update time (5.429 seconds)
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-os: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/local-lvm: -1
Jan 21 17:44:21 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-node-stor-08101/vm-pool-jhomeuser: -1

# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph: 16.2.7
ceph-fuse: 16.2.7
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

 

[dcsapak]

Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2670419]: status update time (5.517 seconds)
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[3332871]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[655887]: got timeout
Jan 21 17:44:21 pve-node-stor-08101 pvestatd[2728807]: status update time (5.429 seconds)

das sieht so aus als würden mehrere pvestatd prozesse gleichzeitig laufen (die sich dann natürlich in die quere kommen)

was sagt denn

ps aux | grep '[p]vestatd'

?

 

[jic5760]

As you said, there were several pvestatd.

# killall pvestatd
# systemctl stop rrdcache
# rm -rf /var/lib/rrdcache/db
# systemctl start rrdcache
# systemctl start pvestatd

The noisy log has quieted down, but I'm still getting errors.

Jan 21 22:22:31 pve-node-stor-08101 pvestatd[1449724]: rados_connect failed - Invalid argument
Jan 21 22:22:31 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08112: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-os: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local-lvm: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-jhomeuser: -1
Jan 21 22:22:32 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08111: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-jhomeuser: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local-lvm: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-os: -1
Jan 21 22:22:33 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local: -1
Jan 21 22:22:41 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08112: -1
Jan 21 22:22:42 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08111: -1
Jan 21 22:22:46 pve-node-stor-08101 pvestatd[1449724]: got timeout
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-jhomeuser: -1
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local: -1
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local-lvm: -1
Jan 21 22:22:46 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-os: -1
Jan 21 22:22:47 pve-node-stor-08101 pvestatd[1449724]: status update time (5.398 seconds)
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-jhomeuser: -1
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local-lvm: -1
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-os: -1
Jan 21 22:22:48 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local: -1
Jan 21 22:22:51 pve-node-stor-08101 pvestatd[1449724]: rados_connect failed - Invalid argument
Jan 21 22:22:51 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08112: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/local-lvm: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-os: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08112/vm-pool-jhomeuser: -1
Jan 21 22:22:52 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-stor-node-08111: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-jhomeuser: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local-lvm: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/vm-pool-os: -1
Jan 21 22:22:53 pve-node-stor-08101 pmxcfs[3488753]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-stor-node-08111/local: -1

 

[jic5760]

This error message keeps filling up the disk.
Killing the pvestatd on node-01 stops the error, but it is not a solution.

Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/local: /var/lib/rrdcached/db/pve2-storage/node-01/local: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/node-01/local-lvm: -1
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/local-lvm: /var/lib/rrdcached/db/pve2-storage/node-01/local-lvm: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-jhomeuser: -1
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-jhomeuser: /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-jhomeuser: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-os: -1
Feb  9 13:30:38 pve-node-02112 pmxcfs[1788890]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-os: /var/lib/rrdcached/db/pve2-storage/node-01/vm-pool-os: illegal attempt to update using time 1644381037 when last update time is 1644381037 (minimum one second step)

 

[dcsapak]

are there still multiple pvestatds running? if yes, i'd investigate why and how they are started exactly.. pve should only ever have one instance of that running...

 

[jic5760]

There is only one pvestatd running on every node.

 

[jic5760]

Huh... I shut down pvestatd on the node that says in the error for a few hours and the error went away.
However, this has happened before, and I think there will be problems again sometime.

Note that all servers are synchronized with NTP, usually with an error of fewer than 0.0001 seconds.