novnc suddenly failing with certificate error

[jmpfas]

4 node cluster running proxmox 5.1-36
Suddenly I cannot connect to VMs or node shell via novnc. It fails to connect to service.
js console shows:
app.js:4696 WebSocket connection to 'wss://{redacted host name}:8006/api2/json/nodes/node01/qemu/1101/vncwebsocket?port=5901&vncticket=PVEVNC%3A5D81A159%3A%3ApoAXdKYZe2a3Ut3Y1vWfDeuA8NiI%2Bea9R08rv2zEtiSUt%2BW030R6zSsHpksxUgjJ8%2FRlr67Grd%2BlOlkBn0MZbsJ%2FIbEP1wd9bVMFWQSj1urpWWlI3b%2F9mJBtKxnDS8byqUanKaDugjQVYa%2BUASBUdYXiahzrZCyy9xEwnQgFPQ2THKDBh80eZbAlxJ74Ldo%2BFePOzX%2FLNPdZ9CGv5xYNGLXmF89JDQg%2FStjtLbJTD2GsmovyklSdj0j62MyGNWwLN%2BwhLmky%2FILvHibd%2F45yA7Flo88QQiE5tY9MJKSwB%2FyIhzIdxEut%2FioOm0WeLfm9oz1wr2J2zdwGg3OpkEDewg%3D%3D' failed: Error in connection establishment: net::ERR_CERT_AUTHORITY_INVALID
I tried using pvecm updatecert and have restarted all daemons.
Any thoughts?

 

[Stoiko Ivanov]

Which browser are you using - try with a different one without any plugins installed and with cleared cache

else - check the journal and /var/log/pveproxy/access.log for hints

I hope this helps!

proxmox 5.1-36

please consider upgrading since 5.1 is quite outdated

 

[jmpfas]

Which browser are you using - try with a different one without any plugins installed and with cleared cache

else - check the journal and /var/log/pveproxy/access.log for hints

I hope this helps!


please consider upgrading since 5.1 is quite outdated

I tried chrome, ie, edge, even safari
I have two clusters - one still on 5.1 the other on 5.4 (latest 5.x) both fail the same way

 

[Stoiko Ivanov]

Any Antivirus solution installed on your client, which could intervene?

 

[jmpfas]

I tried disabling everything. I also found today that if I connect with firefox in private mode I can connect to console on maybe 25% of the VMs.
It totally fails in chrome, ie, edge and safari.
so...I am certain it is on the server end. But since some of them have not been updated in months (I let some expire by accident) it is something old that perhaps has been triggered by updates in browsers.

 

[Stoiko Ivanov]

hmm - on a 4 node cluster having a success-rate of 25% might indicate that there is indeed a problem with your intra-cluster communication.
* did it maybe work for those VMs which are running on the node you were connected through with your browser?

* maybe try to run `pvecm updatecerts --force` on all nodes and restart pveproxy, pvedaemon afterwards

 

[jmpfas]

I had already tried updatecerts --force. Just tried again.
the 25% success rate is random. does not matter which node I am connecting on vs node running vm.
i.e. on gui for node04, I can connect to console on one vm out of ten right now, all running on that node

 

[jmpfas]

I have the solution (based on an answer on another thread). It was being blocked by ESET security. Even if you try "pause firewall/allow all network traffic" that does nOT pause the TLS filtering in ESET. You have to turn that off here