issue adding node, time jump 1 month into future

[ilia987]

i have just added another node (to an existing 10 node cluster)
an i have the following error:

Jan 15 15:07:14 pve-srv5 pmxcfs[2914]: [quorum] crit: quorum_initialize failed: 2
Jan 15 15:07:14 pve-srv5 pmxcfs[2914]: [confdb] crit: cmap_initialize failed: 2
Jan 15 15:07:14 pve-srv5 pmxcfs[2914]: [dcdb] crit: cpg_initialize failed: 2
Jan 15 15:07:14 pve-srv5 pmxcfs[2914]: [status] crit: cpg_initialize failed: 2
Jan 15 15:07:14 pve-srv5 pmxcfs[2914]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve-srv5: -1
Jan 15 15:07:14 pve-srv5 pmxcfs[2914]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve-srv5: /var/lib/rrdcached/db/pve2-node/pve-srv5: illegal attempt to update using time 1579093634 when last update time is 1581773139 (minimum one second step)
Jan 15 15:07:15 pve-srv5 pmxcfs[2914]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve-srv5/local: -1
Jan 15 15:07:20 pve-srv5 pmxcfs[2914]: [quorum] crit: quorum_initialize failed: 2
Jan 15 15:07:20 pve-srv5 pmxcfs[2914]: [confdb] crit: cmap_initialize failed: 2
Jan 15 15:07:20 pve-srv5 pmxcfs[2914]: [dcdb] crit: cpg_initialize failed: 2
Jan 15 15:07:20 pve-srv5 pmxcfs[2914]: [status] crit: cpg_initialize failed: 2

in had a file /var/lib/rrdcached/db/pve2-storage/pve-srv5/local-zfs that was in the futre (1 month in the future) i set to the same date other files in the folder approx an hour ago (install time)

anyone got an idea?

 

[Chris]

Hi,
please check the status of systemd-timesyncd.service on all nodes and make sure that they use the same time servers.
It might be useful to set up a local timeserver and use that one to keep the nodes time synced in some cases.
See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_time_synchronization
Also check the status of pvecm status, it seems your node is not part of the quorum.

 

[ilia987]

it was easier to remove the node and make full install again.
now the node is working and join successfully to the cluster

 

[idleman]

Hi,
please check the status of systemd-timesyncd.service on all nodes and make sure that they use the same time servers.
It might be useful to set up a local timeserver and use that one to keep the nodes time synced in some cases.
See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_time_synchronization
Also check the status of pvecm status, it seems your node is not part of the quorum.

After replacing the CMOS battery on the motherboard and adjusting the BIOS time to the correct time upon booting, I encounter time jumping issues when logging into the system and checking logs. Additionally, abnormal behavior is observed in the monitoring graphs in ProxmoxVE WebUI Console/Server View. Is there any other quick solution to this problem, or should I allow the system to synchronize time with the cluster automatically?
Note: ProxmoxVE originally uses the systemd-timesync service for time synchronization.

 

[Chris]

After replacing the CMOS battery on the motherboard and adjusting the BIOS time to the correct time upon booting, I encounter time jumping issues when logging into the system and checking logs. Additionally, abnormal behavior is observed in the monitoring graphs in ProxmoxVE WebUI Console/Server View. Is there any other quick solution to this problem, or should I allow the system to synchronize time with the cluster automatically?
Note: ProxmoxVE originally uses the systemd-timesync service for time synchronization.

Hi,
please check that systemctl status chrony.service is up and running (Proxmox VE uses it as default since version 7) and try to adjust your time offset by running a chronyc makestep. Note that all the nodes in the cluster require their time synced, especially when running Ceph.

 

[idleman]

Hi,
please check that systemctl status chrony.service is up and running (Proxmox VE uses it as default since version 7) and try to adjust your time offset by running a chronyc makestep. Note that all the nodes in the cluster require their time synced, especially when running Ceph.

Apologies for not providing clear environment parameters. Here are the details of my system:
Proxmox Virtual Environment (PVE): 6.2-6 (Debian 10).
My system does not have the chrony service installed (output message: "Unit chrony.service could not be found."). Additionally, my Ceph storage is external and not part of a hyper-converged architecture.
I have previously explored some relevant forum posts and adopted their methods (allowing the system time to gradually catch up with the current time after replacing the battery). I'm wondering if there is any method available to quickly correct the time jumping issue upon logging into the ProxmoxVE system after replacing the motherboard battery. This way, we wouldn't have to wait for the time jumping issue to resolve before migrating VMs from other healthy nodes to the main host.

 

[Chris]

Proxmox Virtual Environment (PVE): 6.2-6 (Debian 10).

That version is end of life since a long time, I urge you to upgrade to the latest version available following [0] and [1].

I'm wondering if there is any method available to quickly correct the time jumping issue upon logging into the ProxmoxVE system after replacing the motherboard battery. This way, we wouldn't have to wait for the time jumping issue to resolve before migrating VMs from other healthy nodes to the main host.

You can of course set the system time manually, e.g. via timedatectl set-time TIME, after stopping the systemd-timesyncd.service, then start it again. Or you could stop and disable the systemd-timesyncd.service altogether, and directly switch over to chrony by installing it via apt install chrony.

[0] https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0
[1] https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

Edit: Fixed typo