ntp service on pve host

[RobFantini]

I've an issue with ntp and ceph health. a few times I've seen this:

clock skew detected

So instead of runing ntp server I'm considering to switch to just using the client ntpdate and running that from from every 5 minutes.

My question - is ntp server on pve host needed for kvms or anything else?

 

[cesarpk]

is ntp server on pve host needed for kvms or anything else?

The ntp service in the PVE nodes are necessary for that the cluster communication works well, and get the correct timestamp of your log files (very important when you need the debug).

Also you can have a ntp service as client and Server in some PVE node, of this manner, the rest of nodes can query to your ntp server

 

[RobFantini]

On a test ceph cluster I am getting more accurate time using ntpdate client instead of server. At least while systems are busy rebuilding some replaced disks...

Also boot is a lot faster. Sometimes nodes hang when ntp server starts up... I think it is a networking issue to be solved.

So I'll assume it is OK to run ntp client on the hosts. from a crontab i run ' ntpdate <IP of pfsense>' every 5 minutes.

 

[cesarpk]

On a test ceph cluster I am getting more accurate time using ntpdate client instead of server. At least while systems are busy rebuilding some replaced disks...

Also boot is a lot faster. Sometimes nodes hang when ntp server starts up... I think it is a networking issue to be solved.

So I'll assume it is OK to run ntp client on the hosts. from a crontab i run ' ntpdate <IP of pfsense>' every 5 minutes.

If you have problems of networking, ntpdate or NTP server don't have that work.

I believe that your problems is in the firewall, that it don't have the port required open for allow to do the query

But if you have as firewall a pfsence, for me the best configuration will be to have as NTP server and client to pfsense, according to this diagram:


PVE nodes and other PCs-client ------ LAN -----> pfsense Client and Server---- Internet -----> some NTP server


In this setup, pfsense is a Client/Server of NTP services

Where the PVE nodes and other PCs-client in your LAN have the NTP service as client enabled (typically a clean instalation of PVE have the NTP service as client enabled and configured, but with the difference that for make the queries, it are pointing to debian.org)

And i prefer this configuration due to that the service NTP have several advantages front to ntpdate, between the more important are:

1- Do several queries and statistics of queries for determine when is convenient do again next query (more precise that ntpdate)

2- Compare the Harware clock and the Clock of the OS for register the time difference, of this manner, when you start the machine again, the NTP service know the time difference and adjust the time of the OS automatically

3- Are many more diffrences, only try it.

Best regards
Cesar

 

[RobFantini]

Cesar - thank you for the explanation, I'll work on ntp again at end of the week so there will be delay in response.

 

[e100]

The startup issues can be caused by a setting in ntpd that forces it to sync time on startup.
On a CEPH node you want that option enabled.
I've had issue with that where ntpd starts before the network is fully up.

In my proxmox clusters I setup two servers that sync to external time servers and each other. All other nodes sync time to those two nodes.
The only time I have had issues with CEPH and time is on reboot. I need to get the sync time on startup option working reliably and share the solution here.

I would like to get a couple GPS hooked up to the servers to provide a better time source than Internet servers.
It is not uncommon to see three or for second differences between various third party time servers.

 

[josen]

Hello,

try openntpd instead. Works like a charm and should solve your problem. ntpdate makes your local time jump, which is not desireable.

See: https://lists.debian.org/debian-user/2002/12/msg04091.html

Greets

 

[RobFantini]

e100 - I know what you mean about ntp starting before network is ready. have you also seen the clock issue reported in ' ceph health' ?

josen - thanks for the info. I'll check out openntpd . AFAIK pfsense uses openntpd . For a years I've seen ntp slow system startup and changing when ntp starts solves it , or just using ntpdate from rc.local AND cron
I think the issue is with any ntp program starting before the network is ready , and other programs [ in /etc/init.,d ] waiting to start after ntp is running. the start up delay is long.
Would openntpd deal with that differently?

Having one system - pfsense - running ntp at each location makes it so we are using ntp and not just ntpdate. . Then running ntp clients against pfsense makes the ceph nodes all have the same time quickly. Of course there may be a better way to accomplish the time sync.

 

[e100]

Yes, I have had time warning in chef health output but only right after rebooting.
I think I found the cause of this, the NTP servers set to use on my CEPH nodes was not quite right.

To fix my startup issues, where the network is not fully up when ntpd starts I added this to /etc/default/ntp

sleep 3

 

[RobFantini]

Later I discovered that NTP MUST run on host and kvm:

see this article:
https://access.redhat.com/documenta...lation_Guide-KVM_guest_timing_management.html

per searches on internet, slow ntp startup issue is not uncommon. it is usually related to a network issue. so e100's suggestion of adding a sleep statement to /etc/default/ntp is what we'll use to attempt to eliminae the annoying ntp timeout at boot.