Thanks for hint. In my situation this was the problem. Not having a reverse proxy but using <IPv4>:8006 directly.
After disabling ipv6 console access works again properly.
I'm using a single dns entry (as PVE is a consul service in my network), to point to all the nodes in my proxmox cluster. All of them have their own unique SSL certificate, but all of them also have a common name (let's call it pve.my.domain)...