a new added node shows grey question mark when access from a specific node

[riaanp]

I have never seen this before so no idea what could be causing it.



I have a 7 node cluster.

From web GUI node07 -> node1 - works no grey question mark on VM's
From web GUInode06 -> node1 - works no grey question mark on VM's
From web GUInode05 -> node1 - works no grey question mark on VM's
From web GUI node04 -> node1 - Shows node1 with the grey question marks.
From web GUI node03 -> node1 - works no grey question mark on VM's
From web GUI node02 -> node1 - works no grey question mark on VM's
node1 - works directly logged into server web UI

So my logic tell me the problem is with node04. Looking through the forum and google i found that most people say this would fix the problem:

systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd

But it had no effect.

Any suggestions as to why this is happening?

 

[Moayad]

Hi,

Did you see anything interesting message in the Syslog?

 

[riaanp]

No errors being reported. Earlier i had a issue with authkey.pub being rotated the whole time. I deleted it and it was re-created (found that from another post). Earlier in the week I had a NTP issue. But that has since been corrected. Does not explain why this is happening now. And the weird thing is, it was working 100% yesterday.

 

[Moayad]

Proxmox VE 7.x uses Chrony [0]. Can you try to install the Chrony instead and see if this fix the issue?


[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_time_synchronization

 

[riaanp]

I am currently using Proxmox VE 7.2-3 and chrony is running on all nodes already.

 

[riaanp]

I am just trying to understand what went broken. Why is this one node not seeing the new node I added properly but every other node is working 100%

 

[Moayad]

I am just trying to understand what went broken. Why is this one node not seeing the new node I added properly but every other node is working 100%

Hard to say without syslog! Can you provide us with the syslog since added the new node?

I would also, try to update the certificates using:

pvecm updatecerts --force

 

[riaanp]

After that command i see this in the log now (was not there before)

Mar 3 16:02:49 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log
Mar 3 16:02:51 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log
Mar 3 16:02:51 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log
Mar 3 16:02:51 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log
Mar 3 16:02:58 zacptprspmkv04 pvedaemon[737622]: <root@pam> successful auth for user 'root@pam'
Mar 3 16:02:58 zacptprspmkv04 pveproxy[716140]: Could not verify remote node certificate '15:B1:AE:27:28:BD:8A:6E:CE:39:4E:3F:5F:46:BE:C1:78:42:B8:19:BA:ED:72:A0:24:17:4A:AF:32:BF:E5:70' with list of pinned certificates, refreshing cache
Mar 3 16:03:03 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log
Mar 3 16:03:05 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log
Mar 3 16:03:05 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log
Mar 3 16:03:05 zacptprspmkv04 pmxcfs[687969]: [status] notice: received log




But should forcing the cert updates not fix this?

 

[Moayad]

Can you please provide us with the Corosync file (cat /etc/pve/corosync.conf)?

 

[riaanp]

out of desperation, I did something i dont like to do. I upgrade that proxmox host with the pve-no-subcription channel. after the update everything worked again like it used to. Will never now that the issue was. But going to assume it was one of the many proxmox services that stopped working.