[SOLVED] Best practice for timeserver setup in 3 node cluster ?

[swe]

Hi,

we are using a 3 node Proxmox Cluster where the hosts do not have access to the internet, nor to the internal network.
All hosts are managed via a dedicated VPN network. Accessing the hosts from the internal network is not possible.
VMs, which provide any services to the internal network, use bridged network devices.

Some time ago, we had problems with the server times on the hosts, resulting in "clock skews" and failures of ceph.
To fix it, we decided to use the VPN Box as the timeserver for the cluster, beacause all hosts can reach it.

This setup worked well for some years.
Last month, we had a major crash and detected as reason wronggoing times on the hosts.
VPN box also provides time, which is 12 minutes ahead of real time.

We have discussed different timeserver setups.

What is the recommended setup for timeservers, PVE, CEPH and how would i do the switch from the vpn box as timeserver to the new one.
I am a bit afraid of what CEPH will do, when time jumps these 12 minutes and the hosts change it not at the same time.

Thank you Stefan

 

[LnxBil]

What is the recommended setup for timeservers, PVE, CEPH and how would i do the switch from the vpn box as timeserver to the new one.
I am a bit afraid of what CEPH will do, when time jumps these 12 minutes and the hosts change it not at the same time.

Therefore, a mechanism called slewing was introduced in NTP, which is able to adjust the time over a long period of time (days/weeks) in order to now f**k with anything. You need to set this in order to use it.

Best to have always two real high-stratum NTP servers available via network (just enable Port 123/udp to some pool NTP servers in the firewall) or build your own stratum 1 network server with e.g. DCF77.

 

[leesteken]

If the nodes can only see each other, make one the time server for the others. At least they will agree on the time, which may or may not match the rest of the world. Chrony can easily be configured as a server (in contrast to systemd-timesyncd) and as a client with one specific server.

 

[Falk R.]

I like mouseclock for timesync in some situations.

 

[swe]

If the nodes can only see each other, make one the time server for the others. At least they will agree on the time, which may or may not match the rest of the world. Chrony can easily be configured as a server (in contrast to systemd-timesyncd) and as a client with one specific server.

Thanks for your replies.
This is interesting.
Always thought i have to use an stand-alone NTP server.

NTP VM, proxying to internal ntp or global ntp seems no good idea at all.

 

[leesteken]

Thanks for your replies.
This is interesting.
Always thought i have to use an stand-alone NTP server.

NTP VM, proxying to internal ntp or global ntp seems no good idea at all.

The problems is of course when the time server node goes down, the other two will once again get out of sync. Maybe every node can be a time server for all others, but I'm not sure if such a dependency cycle is a good idea, or maybe it can handle it fine? Chrony allows systems to be peers instead of a client-server relation, but maybe that still requires a external time source? I assume you can connect to the nodes from the outside to manage them, then you can forcefully set the time regularly via a script over ssh (or use a reverse ssh tunnel?).
EDIT: You can maybe use the option prefer to make sure cycles don't become problematic (and if you lose two nodes then the cluster won't work anyway).

 

[swe]

Thanks for your help.
I think i will use one hosts as ntp server as you recommend.

 

[LnxBil]

I think i will use one hosts as ntp server as you recommend.

I really don't get why you're not using some upstream server (e.g. Internet or own stratum 0). Timekeeping is VERY important for most things like Logfiles and also TOTP.

 

[swe]

I really don't get why you're not using some upstream server (e.g. Internet or own stratum 0).

The proxmox hosts are neighter connected to the internet, nor do they have an uplink to the internal network.
They are just accessible via an on-demand vpn connection. Always on vpn tunnel is not allowed.

 

[Falk R.]

The proxmox hosts are neighter connected to the internet, nor do they have an uplink to the internal network.
They are just accessible via an on-demand vpn connection. Always on vpn tunnel is not allowed.

For isolated networks I use a MouseClock. This is a radio clock receiver connected via USB or serial.
Either connect it to a host and it provides the time server or buy a more expensive version with a network connection.