Thanks Haider, however the differences between XCP-NG and Proxmox got me thinking a lot, and I decided to compare both versions to see why it worked there and it didn't in my setup.
Lo and behold, for future readers,
my solution was to use OVS (openvswitch) instead of Linux Bridge. Holy cow, how complicated Proxmox can be sometimes.
There, it's working fine and dandy:
View attachment 9915
Network config:
View attachment 9916
Caveat: Since I'm using pfSense as a firewall VM in DHCP (107.151 instead of 107.161), starting it up will
disconnect the management interface (or maybe block it). I think I have to open the port 8006 in pfSense to allow it to work - which is good, anyways, extra security and works as a failover: If the pfSense VM doesn't start up the default port will be open anyways, and through pfSense I can apply port forward to use other a different port than 8006.
Thanks a lot for your help and insights, it was really helpful as well and I learned a lot!