Ceph - Monitor clock skew

yena

Renowned Member
Nov 18, 2011
373
4
83
Hello, on my test Ceph cluster (4 nodes ) this morning i have this Warning
"Monitor clock skew detected"

root@n1:~# ceph health detail
HEALTH_WARN clock skew detected on mon.1, mon.2; Monitor clock skew detected
mon.1 addr 10.10.10.2:6789/0 clock skew 0.085488s > max 0.05s (latency 0.00120293s)
mon.2 addr 10.10.10.3:6789/0 clock skew 0.0755085s > max 0.05s (latency 0.00137496s)

I have installed ntp ntpdate but still have this error..
Any suggestion ?

Thanks
 
Hi,

have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?
 
apt-get install ntp ntpdate and reboot fix the problem.

Thanks
 
Hi,

have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?

Hello Wolfgang,
we also have ceph clock skew.

currently we use for all nodes:
1- timedatectl set-ntp true
2- same servers in timesyncd.conf

I was reading up on systemd ntp and instead want to run ntp the old way.

so I figure these are the steps:

disable systemd timedatectl
Code:
timedatectl set-ntp false

install and configure real ntp
at all pve nodes
Code:
apt-get install ntpdate
configure all by using the same /etc/default/ntpdate , 1ST entry to local ntp server

set up ntp server . we'll use pfsense to start.

am I missing something in that set up?
 
Hi Rob

use ntpd.

ntpdate set the time and this makes problems.

What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.
 
Hi Rob

use ntpd.

ntpdate set the time and this makes problems.

What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.

OK - How do I use ntpd ? Is that just the systemd set up using :
Code:
timedatectl set-ntp true
 
First you have to install ntpd
then edit the /etc/ntp.conf

set systemd-timesyncd on ntpd as you wrote
restart ntpd
 
OK then this is what we'll do to try to fix clock skew warning in ceph.
Code:
apt-get install ntp
Code:
timedatectl set-ntp true
that should just work to deal with time settings on a node and not server time to network

adjust /etc/ntp.conf if you want to - change/add servers or be a ntp server for the local network
-
 
  • Like
Reactions: KORG
We are still getting 'Monitor clock skew detected' after ntp changes.

quick question -

we have mon on same network as osd .
our osd systems are not mons

is it a good practice to use the same network for osd and mon?
 
Yes the for small setups it is ok.
The main propose of separating this 2 networks is the network load.
So if you network has no capacity problems it is ok.
 
Meet time sync problem with PVE 5.4 strange, that ntp time sync not enabled by default.
 
No-matter what, but something should be enable by default, to sync time :)
 
No-matter what, but something should be enable by default, to sync time :)
systemd-timesyncd.service is enable by default.

but it's not precise like a true ntp server. (it's basicaly something like a ntpdate running at some interval, but ceph really need low drift)
 
  • Like
Reactions: KORG
Hello Spirit ,

after install chrony ,

what adjustments if any do you make to /etc/chrony/chrony.conf ?

so far we just adjust the 'server' line to attempt to use something local to our timezone.
 
Last edited:
  • Like
Reactions: Rigel Valladares
Hello Spirit ,

after install chrony ,

what adjustments if any do you make to /etc/chrony/chrony.conf ?

so far we just adjust the 'server' line to attempt to use something local to our timezone.

I keep default settings and only change server to our internal ntp server.
 
  • Like
Reactions: RobFantini

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!