Nodes on Cluster NTP sync out - invalid PVE ticket (401) [SOLVED]

[CousinLarry]

I have a two node ProxMox 6.1-3 cluster, when selecting node 2 from 1 UI I get "invalid PVE ticket (401)" and the UI session time seems very short (20 secs) and logs you out.

I have read that system clocks being out of sync can cause this and it appears that this is the case.

timedatectl status (ran within 2-3 second of each other)
Node 1 (incorrect)

root@pve:~# timedatectl status
               Local time: Wed 2020-07-29 15:43:58 AEST
           Universal time: Wed 2020-07-29 05:43:58 UTC
                 RTC time: Wed 2020-07-29 05:45:09
                Time zone: Australia/Melbourne (AEST, +1000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
root@pve:~#

Node 2 (correct)

root@centurion:/# timedatectl status
               Local time: Wed 2020-07-29 05:46:08 AEST
           Universal time: Tue 2020-07-28 19:46:08 UTC
                 RTC time: Tue 2020-07-28 19:46:08
                Time zone: Australia/Melbourne (AEST, +1000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
root@centurion:/#

running systemctl status systemd-timesyncd on Node 1

root@pve:/etc/default# systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Wed 2020-07-29 16:56:26 AEST; 5min ago
     Docs: man:systemd-timesyncd.service(8)
Main PID: 2666 (systemd-timesyn)
   Status: "Idle."
    Tasks: 2 (limit: 4915)
   Memory: 1.5M
   CGroup: /system.slice/systemd-timesyncd.service
           └─2666 /lib/systemd/systemd-timesyncd

Jul 29 16:56:26 pve systemd[1]: Starting Network Time Synchronization...
Jul 29 16:56:26 pve systemd[1]: Started Network Time Synchronization.
root@pve:/etc/default#

But it never syncs.

Node 1: journalctl -u systemd-timesyncd

root@pve:/etc/default# journalctl -u systemd-timesyncd
-- Logs begin at Wed 2020-07-29 16:47:50 AEST, end at Wed 2020-07-29 17:04:01 AEST. --
Jul 29 16:47:52 pve systemd[1]: Starting Network Time Synchronization...
Jul 29 16:47:52 pve systemd[1]: Started Network Time Synchronization.
Jul 29 16:52:27 pve systemd[1]: Stopping Network Time Synchronization...
Jul 29 16:52:28 pve systemd[1]: systemd-timesyncd.service: Succeeded.
Jul 29 16:52:28 pve systemd[1]: Stopped Network Time Synchronization.
Jul 29 16:52:28 pve systemd[1]: Starting Network Time Synchronization...
Jul 29 16:52:28 pve systemd[1]: Started Network Time Synchronization.
Jul 29 16:55:50 pve systemd[1]: Stopping Network Time Synchronization...
Jul 29 16:55:53 pve systemd[1]: systemd-timesyncd.service: Succeeded.
Jul 29 16:55:53 pve systemd[1]: Stopped Network Time Synchronization.
Jul 29 16:56:26 pve systemd[1]: Starting Network Time Synchronization...
Jul 29 16:56:26 pve systemd[1]: Started Network Time Synchronization.
root@pve:/etc/default#

and Node 2 is very different (which is working correctly)

root@centurion:~# journalctl -u systemd-timesyncd
-- Logs begin at Mon 2020-07-27 15:37:49 AEST, end at Wed 2020-07-29 07:06:00 AEST. --
Jul 27 15:37:51 centurion systemd[1]: Starting Network Time Synchronization...
Jul 27 15:37:51 centurion systemd[1]: Started Network Time Synchronization.
Jul 27 15:38:21 centurion systemd-timesyncd[609]: Synchronized to time server for the first time 193.190.147.153:1
Jul 28 19:56:38 centurion systemd-timesyncd[609]: Timed out waiting for reply from 193.190.147.153:123 (2.debian.p
Jul 28 19:56:48 centurion systemd-timesyncd[609]: Timed out waiting for reply from 94.247.111.10:123 (2.debian.poo
Jul 28 19:56:48 centurion systemd-timesyncd[609]: Synchronized to time server for the first time 5.34.248.224:123
Jul 29 05:11:25 centurion systemd[1]: Stopping Network Time Synchronization...
Jul 29 05:11:25 centurion systemd[1]: systemd-timesyncd.service: Succeeded.
Jul 29 05:11:25 centurion systemd[1]: Stopped Network Time Synchronization.
Jul 29 05:11:25 centurion systemd[1]: Starting Network Time Synchronization...
Jul 29 05:11:25 centurion systemd[1]: Started Network Time Synchronization.
Jul 29 05:11:26 centurion systemd-timesyncd[32593]: Synchronized to time server for the first time 103.239.8.21:12
Jul 29 05:11:28 centurion systemd[1]: Stopping Network Time Synchronization...
Jul 29 05:11:28 centurion systemd[1]: systemd-timesyncd.service: Succeeded.
Jul 29 05:11:28 centurion systemd[1]: Stopped Network Time Synchronization.
Jul 29 05:11:28 centurion systemd[1]: Starting Network Time Synchronization...
Jul 29 05:11:28 centurion systemd[1]: Started Network Time Synchronization.
Jul 29 05:11:29 centurion systemd-timesyncd[32621]: Synchronized to time server for the first time 195.113.20.2:12

Any thoughts as to how I can troubleshoot this?

I have tried:
- restarting the NTP service on Node 1
- timedatectl set-ntp true...
- I have tried all of the steps on https://forum.proxmox.com/threads/wrong-time-wrong-time-synchronisation.55208/page-2, including the GRUB commands but still System clock synchronized: no is in the status.

 

[CousinLarry]

Solved - My DNS was pointing to some other server which means no domains were being resolved, including NTP.