Network lag

[jazzl0ver]

Hi,

I'm running Proxmox 2.3 with the latest updates installed (though, I noticed the same issue on the previous 2.3 builds).

When I ping (from another server) an openvz container which has no network load at the moment, the 1st ping has high latency:

# ping vm-centos6-2
PING vm-centos6-2.domain.tld (172.26.200.250) 56(84) bytes of data.
64 bytes from vm-centos6-2.domain.tld (172.26.200.250): icmp_seq=1 ttl=64 time=227 ms
64 bytes from vm-centos6-2.domain.tld (172.26.200.250): icmp_seq=2 ttl=64 time=0.096 ms
64 bytes from vm-centos6-2.domain.tld (172.26.200.250): icmp_seq=3 ttl=64 time=0.088 ms

Digging it deeper, it appears that the delay occurs before arp reply packet:

17:32:24.289426 ARP, Request who-has 172.26.200.250 tell 172.26.1.14, length 46
17:32:25.040691 ARP, Reply 172.26.200.250 is-at 00:30:48:79:20:d6, length 28
17:32:25.040763 IP 172.26.1.14 > 172.26.200.250: ICMP echo request, id 36915, seq 1, length 64
17:32:25.040812 IP 172.26.200.250 > 172.26.1.14: ICMP echo reply, id 36915, seq 1, length 64

# cat /etc/network/interfaces
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
        address  172.26.1.36
        netmask  255.255.0.0
        gateway  172.26.1.1
        bridge_ports eth0.1
        bridge_stp off
        bridge_fd 0

auto vmbr12
iface vmbr12 inet static
        address  172.27.1.36
        netmask  255.255.0.0
        bridge_ports eth0.12
        bridge_stp off
        bridge_fd 0

Any ideas why that happens? Is there anything that could be tweaked to fix this?

 

[jazzl0ver]

anybody?

 

[sgtfoo]

I have noticed the first pings always have more latency. I haven't yet had a chance to examine the packets with wireshark yet, but I intend to look into it.

Is there something in the virtio abstraction of the NICs that is adding the latency?

 

[jazzl0ver]

Is there something in the virtio abstraction of the NICs that is adding the latency?

If you're asking me, I've no idea

 

[snowman66]

It could be ARP lookup.

 

[jazzl0ver]

snowman66, I do not think so. I see this issue only with openvz containers. All other servers are ARP-replying pretty fast. Also, there are only two D-Link switches that all servers are plug into and there is one meter between them.

Any more ideas?

 

[snowman66]

The cable length has nothing to do with this..

.. similar problem -> http://forum.openvz.org/index.php?t=msg&goto=32294&

 

[jazzl0ver]

Yeah, I've already found that thread (thank you for the string to google). I'm reading it forward and back and back and forward and still not sure I understand how they solved the issue.. Do they mean I have to use veth interface instead of venet?

 

[spirit]

Yeah, I've already found that thread (thank you for the string to google). I'm reading it forward and back and back and forward and still not sure I understand how they solved the issue.. Do they mean I have to use veth interface instead of venet?

I think bridged mode (veth) should be faster than venet, which is a userland virtual network.

(Same thing for kvm with bridged vs nat)

 

[dietmar]

I think bridged mode (veth) should be faster than venet, which is a userland virtual network.

I don't think venet involves any userland code.

 

[tom]

I think bridged mode (veth) should be faster than venet, which is a userland virtual network.

(Same thing for kvm with bridged vs nat)

Recently I did not run any intensive benchmarks but see this page:

http://openvz.org/Differences_between_venet_and_veth

 

[jazzl0ver]

This issue (along with another one - see below) has been resolved after changing CT settings from venet to veth. Thanks to all of you!

NB In parallel, I struggled with another strange issue. I have two containers on two Proxmox servers. Both are CentOS 6. The CTs are in different VLANs. VLANs are routed by pfSense host. I needed to import mysql dump from a file on CT12 (vlan12, veth interface) to CT1 (vlan1, venet interface, the one I had arp reply issue). For some reason, the import stopped after some time and tcpdump showed that some ACK packets do not come out of the Proxmox host which hosted CT1, while I saw them coming out of CT1 (running tcpdump inside CT1). After changing interface type from venet to veth, the issue disappeared. Perhaps, it would help somebody.