Ceph - Monitor clock skew

[yena]

Hello, on my test Ceph cluster (4 nodes ) this morning i have this Warning
"Monitor clock skew detected"

root@n1:~# ceph health detail
HEALTH_WARN clock skew detected on mon.1, mon.2; Monitor clock skew detected
mon.1 addr 10.10.10.2:6789/0 clock skew 0.085488s > max 0.05s (latency 0.00120293s)
mon.2 addr 10.10.10.3:6789/0 clock skew 0.0755085s > max 0.05s (latency 0.00137496s)

I have installed ntp ntpdate but still have this error..
Any suggestion ?

Thanks

 

[wolfgang]

Hi,

have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?

 

[yena]

apt-get install ntp ntpdate and reboot fix the problem.

Thanks

 

[RobFantini]

Hi,

have you also disable systemd-timesyncd.service
and do you use the same server at the ntpd?

Hello Wolfgang,
we also have ceph clock skew.

currently we use for all nodes:
1- timedatectl set-ntp true
2- same servers in timesyncd.conf

I was reading up on systemd ntp and instead want to run ntp the old way.

so I figure these are the steps:

disable systemd timedatectl

timedatectl set-ntp false

install and configure real ntp
at all pve nodes

apt-get install ntpdate

configure all by using the same /etc/default/ntpdate , 1ST entry to local ntp server

set up ntp server . we'll use pfsense to start.

am I missing something in that set up?

 

[wolfgang]

Hi Rob

use ntpd.

ntpdate set the time and this makes problems.

What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.

 

[RobFantini]

Hi Rob

use ntpd.

ntpdate set the time and this makes problems.

What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.

OK - How do I use ntpd ? Is that just the systemd set up using :

timedatectl set-ntp true

 

[wolfgang]

First you have to install ntpd
then edit the /etc/ntp.conf

set systemd-timesyncd on ntpd as you wrote
restart ntpd

 

[RobFantini]

OK then this is what we'll do to try to fix clock skew warning in ceph.

apt-get install ntp

timedatectl set-ntp true

that should just work to deal with time settings on a node and not server time to network

adjust /etc/ntp.conf if you want to - change/add servers or be a ntp server for the local network
-

 

[wosp]

Also see https://forum.proxmox.com/threads/pve-4-1-systemd-timesyncd-and-ceph-clock-skew.27043/

 

[RobFantini]

We are still getting 'Monitor clock skew detected' after ntp changes.

quick question -

we have mon on same network as osd .
our osd systems are not mons

is it a good practice to use the same network for osd and mon?

 

[wolfgang]

Yes the for small setups it is ok.
The main propose of separating this 2 networks is the network load.
So if you network has no capacity problems it is ok.

 

[spirit]

in ntp.conf,
you can also reduce pooling interval

server xxx iburst minpoll 4 maxpoll 7

https://access.redhat.com/solutions/39194

 

[KORG]

Meet time sync problem with PVE 5.4 strange, that ntp time sync not enabled by default.

 

[mbaldini]

Every time I find problems with time sync, I use chrony

 

[spirit]

Every time I find problems with time sync, I use chrony

+1 for chrony.
(I think it could be great to have it by default for proxmox6 instead timesyncd)

 

[KORG]

No-matter what, but something should be enable by default, to sync time

 

[spirit]

No-matter what, but something should be enable by default, to sync time

systemd-timesyncd.service is enable by default.

but it's not precise like a true ntp server. (it's basicaly something like a ntpdate running at some interval, but ceph really need low drift)

 

[RobFantini]

Hello Spirit ,

after install chrony ,

what adjustments if any do you make to /etc/chrony/chrony.conf ?

so far we just adjust the 'server' line to attempt to use something local to our timezone.

 

[spirit]

Hello Spirit ,

after install chrony ,

what adjustments if any do you make to /etc/chrony/chrony.conf ?

so far we just adjust the 'server' line to attempt to use something local to our timezone.

I keep default settings and only change server to our internal ntp server.