So did you executed pvecem delnode <oldnode> after you removed the node?
If not gain temporary quorum with
pvecm expected 1
pvecm delnode old-node-name
If you have done this and it does not work please post output from pvecm status
If you just added the Agent option in the VM, you must fully shutdown the VM and start it again.
Agent option creates a hardware device on the VM. This device is not hot-pluggable.
Blockbridge : Ultra low latency all-NVME shared storage for...
Thanks for hint. In my situation this was the problem. Not having a reverse proxy but using <IPv4>:8006 directly.
After disabling ipv6 console access works again properly.
I'm using a single dns entry (as PVE is a consul service in my network), to point to all the nodes in my proxmox cluster. All of them have their own unique SSL certificate, but all of them also have a common name (let's call it pve.my.domain)...