I've been troubleshooting this for a few days. It's been happening for the past several months.
If your proxmox server is connected to more than one subnet, and has an IP configured on them, I'd look at that.
For me, I had an issue a while back and configured a spare ethernet port to connect to the subnet my computer was on, so I could connect to a local IP to perform network changes. I never removed it... Fast forward, because I normally connect to the proxmox IP that's on another subnet, I guess the network path was being split and causing problems (return packets would come out a different interface because it was local). As soon as I removed that IP from proxmox on my local subnet, the issue went away.
FYI, I was able to reliably reproduce the freeze by watching a video in the proxmox VM console and keep moving the mouse around. It would consistently freeze after about 50 seconds. No more!