If you just added the Agent option in the VM, you must fully shutdown the VM and start it again.
Agent option creates a hardware device on the VM. This device is not hot-pluggable.
Blockbridge : Ultra low latency all-NVME shared storage for...
Thanks for hint. In my situation this was the problem. Not having a reverse proxy but using <IPv4>:8006 directly.
After disabling ipv6 console access works again properly.
I'm using a single dns entry (as PVE is a consul service in my network), to point to all the nodes in my proxmox cluster. All of them have their own unique SSL certificate, but all of them also have a common name (let's call it pve.my.domain)...