[SOLVED] Proxmox ZFS install error on RRDC update

[RedBlood9]

Hi to all,
I've recently installed proxmox 6 with ZFS, the installation went right, but after a while I received from the syslog the following errors:

Oct  2 11:21:41 bs202.local.mobytgroup.com pmxcfs[3728]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/bs202: -1
Oct  2 11:21:41 bs202.local.mobytgroup.com pmxcfs[3728]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/bs202/local-zfs: -1
Oct  2 11:21:41 bs202.local.mobytgroup.com pmxcfs[3728]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/bs202/local: -1
Oct  2 11:21:51 bs202.local.mobytgroup.com pmxcfs[3728]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/bs202: -1

I've already installed the latest updates, and rebooted the server, but it didn't resolve the problem.
The ntp service starts but ends everytime with this error:

Oct 02 11:14:41 bs202 ntpd[38295]: Listening on routing socket on fd #20 for interface updates
Oct 02 11:14:41 bs202 ntpd[38295]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Oct 02 11:14:41 bs202 ntpd[38295]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Oct 02 11:23:38 bs202 ntpd[38295]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

Here's all the details of my software version:

pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-2-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-5.0.21-2-pve: 5.0.21-6
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-5
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2

Correlated to this error, after a bit I logged in the web gui, i get logged out with the message "Invalid PVE tickets"

Any idea on how to resovle the problem?

 

[Alwin]

Do you see any errors (disk/memory) related in the journal/syslog? On what hardware does the PVE run?

 

[RedBlood9]

This promox runs on a PowerEdge R440, cpu 2 socket and a H330 Mini raid controller in HBA mode, all disks are passed as JBOD
I've just seen an error like this one:

Oct 2 11:55] systemd-journald[2691]: Failed to set ACL on /var/log/journal/73f6355fcc644c2096e10869c7123be6/user-9001.journal, ignoring: Operation not supported
[  +0.248372] systemd-journald[2691]: Failed to set ACL on /var/log/journal/73f6355fcc644c2096e10869c7123be6/user-9001.journal, ignoring: Operation not supported

The dimm check exited with test passed

 

[Stoiko Ivanov]

All of the symptoms speak for a problem with the system's clock - check the settings in bios, typing date in the commandline and make sure to get NTP running (via timesyncd or some other way)

Oct 2 11:55] systemd-journald[2691]: Failed to set ACL on /var/log/journal/73f6355fcc644c2096e10869c7123be6/user-9001.journal, ignoring: Operation not supported

This just says that the filesystem your /var/log/journal resides on does not support ACLs, thus user 9001 probably won't be able to read their journal with journalctl unless they are also in the systemd-journal group

I hope this helps!

 

[RedBlood9]

Hi Stoiko,
I've updated the ntp server on the drac and the error regarding the RRDC disappeared.
But checking the ntp status I continue to get this error:

systemctl status ntp
● ntp.service - Network Time Service
   Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-10-04 08:21:33 CEST; 19min ago
     Docs: man:ntpd(8)
  Process: 3225 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
 Main PID: 3243 (ntpd)
    Tasks: 2 (limit: 11059)
   Memory: 2.9M
   CGroup: /system.slice/ntp.service
           └─3243 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 111:119

Oct 04 08:21:33 bs202 ntpd[3243]: proto: precision = 0.044 usec (-24)
Oct 04 08:21:33 bs202 ntpd[3243]: Listen and drop on 0 v6wildcard [::]:123
Oct 04 08:21:33 bs202 ntpd[3243]: Listen and drop on 1 v4wildcard 0.0.0.0:123
Oct 04 08:21:33 bs202 ntpd[3243]: Listen normally on 2 lo 127.0.0.1:123
Oct 04 08:21:33 bs202 ntpd[3243]: Listening on routing socket on fd #19 for interface updates
Oct 04 08:21:33 bs202 ntpd[3243]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Oct 04 08:21:33 bs202 ntpd[3243]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Oct 04 08:21:33 bs202 systemd[1]: Started Network Time Service.
Oct 04 08:21:37 bs202 ntpd[3243]: Listen normally on 3 vmbr0 192.168.0.2:123
Oct 04 08:21:37 bs202 ntpd[3243]: new interface(s) found: waking up resolver

In the drac/bios settings I don't see anything regarding the system's clock configuration or permission

 

[RedBlood9]

I know it stupid, it was a hardware problem on our Router, who has a problem on the RTC module.
The only strange thing, was that ntp daemon on pve side reported a kernel error, rather than a server error.

Anyway, now is resolved