I've noticed that the default net.netfilter.nf_conntrack_max is too low and I was hitting connection problems too often. Dropped connections, connection failures, nasty stuff.
I had to manually increase net.netfilter.nf_conntrack_max to 1000000000 in all my proxmox nodes.
Perhaps, the staff...