What exactly did you check: VM(Container) to host, host to external or VM(container) to external? What is the result when you compare the figures of these 3 scenarios?
Host has 3 nics, one is used only for proxmox, the other 2 are used for testing things like OPNSense/OpenWRT (WAN/LAN).
External host (Smokeping on Synology NAS) to Proxmox:
0.540 ms avg
External host (Smokeping on Synology NAS) to OpenWRT LXC:
0.562 ms avg
External (using Ubuntu via gigabit Eth) host to Proxmox: rtt min/avg/max/mdev = 0.443/
0.787/1.038/0.145 ms
External (using Ubuntu via gigabit Eth) host to OpenWRT LXC: rtt min/avg/max/mdev = 0.579/
0.931/1.289/0.179 ms
Proxmox to OpenWRT LXC container: rtt min/avg/max/mdev = 0.188/
0.292/0.659/0.041 ms
Proxmox to Synology NAS (what runs Smokeping in a container): rtt min/avg/max/mdev = 0.188/
0.220/0.254/0.015 ms
OpenWRT container to Synology NAS: round-trip min/avg/max = 0.218/
0.306/0.333 ms
OpenWRT container to Proxmox: round-trip min/avg/max = 0.190/
0.327/0.355 ms
Conclusion is that outbound from Proxmox is more stable than inbound. If there's a setting or something I can try, I will give it a shot.