I do not know much about bridges, but either do TFA on PVE host for better password security, or get a new network switch firewall capabilities to block traffic from all ips, except XYZ, get the vm's on a different network switch, , or tweak around with Proxmox bridges and firewall.