[SOLVED] Web GUI "Communication Failure" for all components

[raulnor516]

Having a tough time with my 4-node Proxmox cluster, and trying to avoid reinstalling everything. I've had these 4 nodes set up for about 6 months now with no issues in my homelab, but starting around a month ago I can't get anything done anymore.

All of my hosts show up as online on the webgui and "pvecm status", with full quorum. I can start and shutdown VMs without errors, and open up shells or consoles to anything I need. Unfortunately, I can't actually manage any of my nodes without having a separate webGUI open for each one.

To be more clear: I have 4 nodes (B, O, P and T). When I log into the B webGUI, everything LOOKS fine, until I click the summary for O, or one of O's VMs. Attempting to manage or get information from any non-local host or VM leads to a error saying "Communication Error (0)" or "Connection error 596: Connection timed out". But if I open up a new webpage for O's webgui, I can manage all of O's things just fine...but nothing else.

 

[Alwin]

pvecm updatecerts -force

Try to generate a new set of certificates, command needs to run only on one node.

 

[raulnor516]

pvecm updatecerts -force

Try to generate a new set of certificates, command needs to run only on one node.

No change. I'm doing my testing by clicking on "Package Versions" on the Summary screen. It always works on the locally logged in host, and never on remote hosts, even after running the updatecerts -force command.By using the Package Version, I always receive the same error "Connection error - Timeout.". I can still get to the shells for each of the remote hosts though, but not any of the VM consoles.

 

[raulnor516]

Just to be safe, I just checked and make sure all of the nodes are synced to the same NTP server using timedatectl/timesyncd

 

[raulnor516]

Figured it out. I had set a port-mirroring session for traffic to one of the NICs on that Proxmox box. Apparently that caused the cluster traffic to get all confused. Removing the node from the proxmox cluster solved the problem.