Ceph - Monitor clock skew

Discussion in 'Proxmox VE: Installation and configuration' started by yena, Apr 3, 2017.

  1. yena

    yena Member

    Joined:
    Nov 18, 2011
    Messages:
    280
    Likes Received:
    1
    Hello, on my test Ceph cluster (4 nodes ) this morning i have this Warning
    "Monitor clock skew detected"

    root@n1:~# ceph health detail
    HEALTH_WARN clock skew detected on mon.1, mon.2; Monitor clock skew detected
    mon.1 addr 10.10.10.2:6789/0 clock skew 0.085488s > max 0.05s (latency 0.00120293s)
    mon.2 addr 10.10.10.3:6789/0 clock skew 0.0755085s > max 0.05s (latency 0.00137496s)

    I have installed ntp ntpdate but still have this error..
    Any suggestion ?

    Thanks
     
  2. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,598
    Likes Received:
    306
    Hi,

    have you also disable systemd-timesyncd.service
    and do you use the same server at the ntpd?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. yena

    yena Member

    Joined:
    Nov 18, 2011
    Messages:
    280
    Likes Received:
    1
    apt-get install ntp ntpdate and reboot fix the problem.

    Thanks
     
  4. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,488
    Likes Received:
    21
    Hello Wolfgang,
    we also have ceph clock skew.

    currently we use for all nodes:
    1- timedatectl set-ntp true
    2- same servers in timesyncd.conf

    I was reading up on systemd ntp and instead want to run ntp the old way.

    so I figure these are the steps:

    disable systemd timedatectl
    Code:
    timedatectl set-ntp false
    
    install and configure real ntp
    at all pve nodes
    Code:
    apt-get install ntpdate
    
    configure all by using the same /etc/default/ntpdate , 1ST entry to local ntp server

    set up ntp server . we'll use pfsense to start.

    am I missing something in that set up?
     
  5. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,598
    Likes Received:
    306
    Hi Rob

    use ntpd.

    ntpdate set the time and this makes problems.

    What I mean is ceph do not like time jumps. ntpd is a daemon and do not set the time hard, it makes correction with accelerate the time or decelerate the time.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,488
    Likes Received:
    21
    OK - How do I use ntpd ? Is that just the systemd set up using :
    Code:
    timedatectl set-ntp true
    
     
  7. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,598
    Likes Received:
    306
    First you have to install ntpd
    then edit the /etc/ntp.conf

    set systemd-timesyncd on ntpd as you wrote
    restart ntpd
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,488
    Likes Received:
    21
    OK then this is what we'll do to try to fix clock skew warning in ceph.
    Code:
    apt-get install ntp
    
    Code:
    timedatectl set-ntp true
    
    that should just work to deal with time settings on a node and not server time to network

    adjust /etc/ntp.conf if you want to - change/add servers or be a ntp server for the local network
    -
     
    dkorzhevin likes this.
  9. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,488
    Likes Received:
    21
    We are still getting 'Monitor clock skew detected' after ntp changes.

    quick question -

    we have mon on same network as osd .
    our osd systems are not mons

    is it a good practice to use the same network for osd and mon?
     
  10. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,598
    Likes Received:
    306
    Yes the for small setups it is ok.
    The main propose of separating this 2 networks is the network load.
    So if you network has no capacity problems it is ok.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,302
    Likes Received:
    131
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. dkorzhevin

    dkorzhevin New Member

    Joined:
    Dec 25, 2012
    Messages:
    16
    Likes Received:
    0
    Meet time sync problem with PVE 5.4 strange, that ntp time sync not enabled by default.
     
  13. mbaldini

    mbaldini Member

    Joined:
    Nov 7, 2015
    Messages:
    166
    Likes Received:
    19
    Every time I find problems with time sync, I use chrony
     
    dkorzhevin likes this.
  14. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,302
    Likes Received:
    131
    +1 for chrony.
    (I think it could be great to have it by default for proxmox6 instead timesyncd)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    dkorzhevin and mbaldini like this.
  15. dkorzhevin

    dkorzhevin New Member

    Joined:
    Dec 25, 2012
    Messages:
    16
    Likes Received:
    0
    No-matter what, but something should be enable by default, to sync time :)
     
  16. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,302
    Likes Received:
    131
    systemd-timesyncd.service is enable by default.

    but it's not precise like a true ntp server. (it's basicaly something like a ntpdate running at some interval, but ceph really need low drift)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,488
    Likes Received:
    21
    Hello Spirit ,

    after install chrony ,

    what adjustments if any do you make to /etc/chrony/chrony.conf ?

    so far we just adjust the 'server' line to attempt to use something local to our timezone.
     
    #18 RobFantini, Apr 21, 2019
    Last edited: Apr 21, 2019
  18. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,302
    Likes Received:
    131
    I keep default settings and only change server to our internal ntp server.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    RobFantini likes this.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice