[SOLVED] ceph time skew issue after reboot of cluster [it was DNS]

scyto

Active Member
Aug 8, 2023
359
64
28
I shut all 3 nodes for maintenance
When they came back up I have the following ceph eerors

Code:
    health: HEALTH_WARN
            clock skew detected on mon.pve1-IPv6

On this node and this node only, choronyc shows no sources (simple restart of chrony service did not fix this)

Code:
root@pve1:~# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample              
===============================================================================
root@pve1:~#

i also note on this node, and this node only that the system clock is not synchronized and the others are

Code:
root@pve1:~#  timedatectl status
               Local time: Sun 2023-10-08 14:54:41 PDT
           Universal time: Sun 2023-10-08 21:54:41 UTC
                 RTC time: Sun 2023-10-08 21:54:41
                Time zone: America/Los_Angeles (PDT, -0700)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

chronyc makestep did not resolve this issue

what should i try next?
 
You can look at "systemctl status chronyd"
and check the output of "cat /etc/chrony/chrony.conf |egrep -v "^#|^$""

Compare the outputs to working nodes. If you manually configured additional stuff that may be "sourcedir" or "configdif" - check those too.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
@bbgeek17 sorry to waste your time, thanks for the help

(it of course would have helped if chrony had actually said in any of its chronyc output or chronyd logs it couldn't resolve the Debian pool name - it didn't... that seems like a huge miss, unless i am missing something else...)

e1251252-c311-4200-8d9c-d5db7fe4511a-Ex7BbOXUYAcurA3.jpg

this one node had slightly different DNS to the other two - it was looking for a container i had not yet started....
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!