Ceph - Monitor clock skew

yena

Member
Nov 18, 2011
293
1
18
Hello, on my test Ceph cluster (4 nodes ) this morning i have this Warning
"Monitor clock skew detected"

root@n1:~# ceph health detail
HEALTH_WARN clock skew detected on mon.1, mon.2; Monitor clock skew detected
mon.1 addr 10.10.10.2:6789/0 clock skew 0.085488s > max 0.05s (latency 0.00120293s)
mon.2 addr 10.10.10.3:6789/0 clock skew 0.0755085s > max 0.05s (latency 0.00137496s)

I have installed ntp ntpdate but still have this error..
Any suggestion ?

Thanks
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
4,763
316
83
Hi,

have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?
 

yena

Member
Nov 18, 2011
293
1
18
apt-get install ntp ntpdate and reboot fix the problem.

Thanks
 

RobFantini

Well-Known Member
May 24, 2012
1,597
26
48
Boston,Mass
Hi,

have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?
Hello Wolfgang,
we also have ceph clock skew.

currently we use for all nodes:
1- timedatectl set-ntp true
2- same servers in timesyncd.conf

I was reading up on systemd ntp and instead want to run ntp the old way.

so I figure these are the steps:

disable systemd timedatectl
Code:
timedatectl set-ntp false
install and configure real ntp
at all pve nodes
Code:
apt-get install ntpdate
configure all by using the same /etc/default/ntpdate , 1ST entry to local ntp server

set up ntp server . we'll use pfsense to start.

am I missing something in that set up?
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
4,763
316
83
Hi Rob

use ntpd.

ntpdate set the time and this makes problems.

What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.
 

RobFantini

Well-Known Member
May 24, 2012
1,597
26
48
Boston,Mass
Hi Rob

use ntpd.

ntpdate set the time and this makes problems.

What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.
OK - How do I use ntpd ? Is that just the systemd set up using :
Code:
timedatectl set-ntp true
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
4,763
316
83
First you have to install ntpd
then edit the /etc/ntp.conf

set systemd-timesyncd on ntpd as you wrote
restart ntpd
 

RobFantini

Well-Known Member
May 24, 2012
1,597
26
48
Boston,Mass
OK then this is what we'll do to try to fix clock skew warning in ceph.
Code:
apt-get install ntp
Code:
timedatectl set-ntp true
that should just work to deal with time settings on a node and not server time to network

adjust /etc/ntp.conf if you want to - change/add servers or be a ntp server for the local network
-
 
  • Like
Reactions: dkorzhevin

RobFantini

Well-Known Member
May 24, 2012
1,597
26
48
Boston,Mass
We are still getting 'Monitor clock skew detected' after ntp changes.

quick question -

we have mon on same network as osd .
our osd systems are not mons

is it a good practice to use the same network for osd and mon?
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
4,763
316
83
Yes the for small setups it is ok.
The main propose of separating this 2 networks is the network load.
So if you network has no capacity problems it is ok.
 

RobFantini

Well-Known Member
May 24, 2012
1,597
26
48
Boston,Mass
Hello Spirit ,

after install chrony ,

what adjustments if any do you make to /etc/chrony/chrony.conf ?

so far we just adjust the 'server' line to attempt to use something local to our timezone.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!