I noticed this too.
Updates were taking hours, wget favorite-disto-from-nearby-I2.iso was getting
~7kB/s in an ubuntu OpenVZ container, but the host was pulling 7MB/s
for the same iso image. I retrieved a newer container template from the OpenVZ site with no change in download speed. I created a centos container node, same problem.
Then I migrated it to another host node and download speeds were identical with the
new host node (i.e. faster). Migrate it back, now it is 3 orders of magnitude slower again.
I have 9 nodes, all running ProxMox 1.6 in a clustered configuration,
1 Sunfire X2100m2, (master, Opteron)
2 Sunfire X2270's, (Xeon based)
2 Supermicro Twin (Xeon based)
4 Supermicro Twin^2 (Opteron based)
All are updated and running:
x2270a:~# pveversion -v
pve-manager: 1.6-5 (pve-manager/1.6/5261)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.6-24
pve-kernel-2.6.32-4-pve: 2.6.32-24
pve-kernel-2.6.24-5-pve: 2.6.24-6
qemu-server: 1.1-22
pve-firmware: 1.0-9
libpve-storage-perl: 1.0-14
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-8
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.12.5-2
ksm-control-daemon: 1.0-4
I began migrating the containers (centos and ubuntu) around and found:
Host eth0 Chipset CPU Host speed OpenVZ speed
------------------------------------------------------------------------------------------
Sun X2100m2 Bcm95713 Opteron 7+MB/s 7+MB/s
Sun X2270 (x2) Intel 82575EB Xeon 5520 7+MB/s slow 7KiloB/s
Supermicro Twin2 Intel 82576 Opteron 6100 7+MB/s slow 7KiloB/s
SuperMicro Twin2 Intel 82574L Xeon 5600 7+MB/s 7+MB/s
lspci:
Sun X2100m2 (Opteron) (fast - driver tg3?)
06:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3)
Sun X2270 (Xeon) (slow - driver igb)
04:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
Supermicro (Xeon) (fast - driver e1000e)
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Supermicro (Opteron) (slow - driver igb)
02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
Some selected results of: dmesg | grep eth0 on the four nodes:
X2100: eth0: Tigon3 [partno(BCM95715) rev 9003] (PCIX:133MHz:64-bit) MAC
X2270: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
SM(Xeon): e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
SM(Opteron): igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Both "slow" nodes seem to be loading the igb drive for the intel chipset.
Both "slow" nodes appear to have RX flow control set.
Running the wget command on another (netbsd) machine across campus with
tcpdump files read with wireshark revealed the sending machine not getting packets acknowledged quite right and the sending node retransmitting many many many packets, multiple per packet. (hundreds, maybe thousands of retransmits)
Other observations:
- KVM based vm's had the same speed as the PVE host on all hardware. (fast)
- Oddly, wget from the *slow* containers to a MacPro (2xquad core Xeon macintosh)
were very fast. (same subnet in local network, but still 96MB/s!)
Then I stumbled upon the OpenVZ bug #1619
http://bugzilla.openvz.org/show_bug.cgi?id=1619
It describes a similar effect, with a comment (#2) describing a workaround:
i.e: ethtool -K eth0 rx off
So I tried it and it works! Download speed matches the hosts speed.
you can turn flow control back on and it still keeps the speed up.
I havn't done a tcpdump to check, but I suspect the retransmission of packets
has stopped. I don't know where to take it from here, but I speculate the
intel igb driver is not getting correctly initalized for the container (if thats how it works) and the ethtool command appears to correct it.
hope this helps.
