High internal latency.

[yctn]

i come in despair.

machines has 56 cores 250 gb ram and 2 x 10 gb link.

i have 5 machines using proxmox they are all affected by the same problem.
when i ping a vm from another vm on the same hypervisor latency is Horrible.

64 bytes from 172.20.0.131: icmp_seq=3867 ttl=64 time=26.3 ms
64 bytes from 172.20.0.131: icmp_seq=3868 ttl=64 time=0.647 ms
64 bytes from 172.20.0.131: icmp_seq=3869 ttl=64 time=1.26 ms
64 bytes from 172.20.0.131: icmp_seq=3870 ttl=64 time=15.6 ms
64 bytes from 172.20.0.131: icmp_seq=3871 ttl=64 time=0.799 ms
64 bytes from 172.20.0.131: icmp_seq=3872 ttl=64 time=0.539 ms
64 bytes from 172.20.0.131: icmp_seq=3873 ttl=64 time=15.6 ms
64 bytes from 172.20.0.131: icmp_seq=3876 ttl=64 time=1.42 ms
64 bytes from 172.20.0.131: icmp_seq=3877 ttl=64 time=3.35 ms
64 bytes from 172.20.0.131: icmp_seq=3878 ttl=64 time=1.00 ms
64 bytes from 172.20.0.200: icmp_seq=10 ttl=64 time=1.30 ms
64 bytes from 172.20.0.200: icmp_seq=14 ttl=64 time=1.19 ms
64 bytes from 172.20.0.200: icmp_seq=25 ttl=64 time=1.96 ms

i already checked that the interfaces are not fully saturated.

does anyone have any suggestions on what i could do to investigate this further?