I was having the same issue until I found this thread: https://forum.proxmox.com/threads/communication-issue-between-sriov-vm-vf-and-ct-on-pf-bridge.68638/
Using the hookscript at https://github.com/jdlayman/pve-hookscript-sriov for my VMs and CTs fixed the problem.
