networking painfully slow after upgrade to 3.4

M

mo_

Renowned Member
Oct 27, 2011
401
8
83
Germany
Hi,

recently upgraded my old 2.3 server to 3.4 and am now finding that the networking is just painfully slow (Im talking 200kb/s here!).

First things first:

pveversion -v said:
proxmox-ve-2.6.32: 3.4-156 (running kernel: 2.6.32-39-pve)
pve-manager: 3.4-6 (running version: 3.4-6/102d4547)
pve-kernel-2.6.32-39-pve: 2.6.32-156
pve-kernel-2.6.32-17-pve: 2.6.32-83
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-17
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
Click to expand...

Code: 
 lspci| grep -i real
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)

Code: 
modinfo r8169
filename:       /lib/modules/2.6.32-39-pve/kernel/drivers/net/r8169.ko
firmware:       rtl_nic/rtl8168g-1.fw
firmware:       rtl_nic/rtl8106e-1.fw
firmware:       rtl_nic/rtl8411-1.fw
firmware:       rtl_nic/rtl8402-1.fw
firmware:       rtl_nic/rtl8168f-2.fw
firmware:       rtl_nic/rtl8168f-1.fw
firmware:       rtl_nic/rtl8105e-1.fw
firmware:       rtl_nic/rtl8168e-3.fw
firmware:       rtl_nic/rtl8168e-2.fw
firmware:       rtl_nic/rtl8168e-1.fw
firmware:       rtl_nic/rtl8168d-2.fw
firmware:       rtl_nic/rtl8168d-1.fw
version:        2.3LK-NAPI
license:        GPL
description:    RealTek RTL-8169 Gigabit Ethernet driver
author:         Realtek and the Linux r8169 crew <netdev@vger.kernel.org>
srcversion:     4C34A7693E03D5EF3239253
alias:          pci:v00000001d00008168sv*sd00002410bc*sc*i*
alias:          pci:v00001737d00001032sv*sd00000024bc*sc*i*
alias:          pci:v000016ECd00000116sv*sd*bc*sc*i*
alias:          pci:v00001259d0000C107sv*sd*bc*sc*i*
alias:          pci:v00001186d00004302sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias:          pci:v000010ECd00008169sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008168sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008167sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008136sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008129sv*sd*bc*sc*i*
depends:        mii
vermagic:       2.6.32-39-pve SMP mod_unload modversions
parm:           use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
parm:           debug:Debug verbosity level (0=none, ..., 16=all) (int)

Amongst the apparent lack of speed, the problem is this:

Code: 
ethtool -s eth0 speed 1000 duplex full autoneg off
Cannot set new settings: Invalid argument
  not setting speed
  not setting duplex
  not setting autoneg
Note that I can disable autoneg (which more often than not doesnt work) if I run that command without speed. I can however not set the mode to 1000baseT. Prior to the upgrade I very much could and had.

I tried installing firmware-realtek, but...

Code: 
The following packages have unmet dependencies:
 pve-firmware : Conflicts: firmware-realtek but 0.36+wheezy.1 is to be installed.

Do I have to download and compile realtek drivers myself now? Any other solutions?
 
dpkg -s pve-firmware
Package: pve-firmware
Status: install ok installed
Priority: optional
Section: misc
Maintainer: Proxmox Support Team <support@proxmox.com>
Architecture: all
Version: 1.1-4
Replaces: firmware-linux-free, firmware-linux-nonfree, firmware-atheros, firmware-bnx2, firmware-bnx2x, firmware-intelwimax, firmware-ipw2x00, firmware-iwlwifi, firmware-netxen, firmware-qlogic, firmware-ralink, firmware-realtek, firmware-libertas, firmware-myricom
Suggests: linux-image
Conflicts: firmware-linux-free, firmware-linux-nonfree, firmware-atheros, firmware-bnx2, firmware-bnx2x, firmware-intelwimax, firmware-ipw2x00, firmware-iwlwifi, firmware-netxen, firmware-qlogic, firmware-ralink, firmware-realtek, firmware-libertas, firmware-myricom
Description: Binary firmware code for the pve-kernel
This package contains the binary firmware for various modules used in
the pve-kernel.

And

dpkg -L pve-firmware |grep realtek returns no hit.

Since pve-firmware replaces firmware-realtek the firmware should be part of the package but it isn't so I guess this is a packaging error.

As a work a round you could download the package and extract the firmware files and then install them under:
/lib/firmware/rtl_nic/
 
hm.. this is very weird. I get like 2mb/s if I download something from a Hetzner-internal server to mine.

I can download files from my server just fine with like 10mb/s

but file transfers TO my server from outside hetzners network are very slow.

This leads me to conclude that its not actually a pve/driver issue, it just coincided with my upgrade. Sorry!


Still confused whats going on though... my best working theory might be a ddos on a neighboring server or network segment?
 
Could it be that the nic does not fulfill the below requirements?
alias: pci:v00000001d00008168sv*sd00002410bc*sc*i*
alias: pci:v00001737d00001032sv*sd00000024bc*sc*i*
alias: pci:v000016ECd00000116sv*sd*bc*sc*i*
alias: pci:v00001259d0000C107sv*sd*bc*sc*i*
alias: pci:v00001186d00004302sv*sd*bc*sc*i*
alias: pci:v00001186d00004300sv*sd*bc*sc*i*
alias: pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias: pci:v000010ECd00008169sv*sd*bc*sc*i*
alias: pci:v000010ECd00008168sv*sd*bc*sc*i*
alias: pci:v000010ECd00008167sv*sd*bc*sc*i*
alias: pci:v000010ECd00008136sv*sd*bc*sc*i*
alias: pci:v000010ECd00008129sv*sd*bc*sc*i*

What does lspci -vv | grep -i real show?
 
the command you gave doesnt work like that ;)

however:

Code: 
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
        Subsystem: ASUSTeK Computer Inc. P8P67 and other motherboards
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 33
        Region 0: I/O ports at e000 [size=256]
        Region 2: Memory at d0004000 (64-bit, prefetchable) [size=4K]
        Region 4: Memory at d0000000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000feeff00c  Data: 41b9
        Capabilities: [70] Express (v2) Endpoint, MSI 01
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
                Unknown small resource type 00, will not decode more.
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
        Capabilities: [160 v1] Device Serial Number 0c-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169

maybe we can figure out why it wont let me set it to gbit?
 
Do you see something similar like this:

dmesg |grep rtl
[ 15.386531] r8169 0000:03:00.0: firmware: direct-loading firmware rtl_nic/rtl8168e-3.fw
 
sortof...

dmesg|grep rtl
r8169 0000:04:00.0: firmware: requesting rtl_nic/rtl8168e-2.fw
 
So firmware is found and loaded.

Just tried the same here:
sudo ethtool -s eth2 speed 1000 duplex full autoneg off
Cannot set new settings: Invalid argument
not setting speed
not setting duplex
not setting autoneg

sudo ethtool -i eth2
driver: r8169
version: 2.3LK-NAPI
firmware-version: rtl8168e-3_0.0.4 03/27/12
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

I think the reason is this: supports-eeprom-access: no

Above works on my Intel nics

sudo ethtool -i eth0
driver: e1000e
version: 2.3.2-k
firmware-version: 1.8-0
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
 
I Propose to compile the Realtek driver from source. i had the Same issues Month ago. With a fresh Compiled driver you will reach better Performance.

On the otherside i Need to say that Realtek nic's are Not recommended to use with Proxmox. if possible go and get an Intel NIC.
 
just an FYI, the weird situation continues. I suppose it actually is a firmware/driver issue and will now try to compile drivers. another short night for me...
 
If you want I can send you the source for realtek's source (r8168) layout so in can be compiled and installed through dkms. It works here both against 2.6.32.x and 3.10.x kernels.

You can see this driver has all the options:
modinfo r8168
filename: /lib/modules/3.10.0-10-pve/updates/dkms/r8168.ko
version: 8.037.00-NAPI
license: GPL
description: RealTek RTL-8168 Gigabit Ethernet driver
author: Realtek and the Linux r8168 crew <netdev@vger.kernel.org>
rhelversion: 7.1
srcversion: 4B9CB513BB22516F2449F45
alias: pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias: pci:v000010ECd00008168sv*sd*bc*sc*i*
depends:
vermagic: 3.10.0-8-pve SMP mod_unload modversions
parm: eee_enable:int
parm: speed:force phy operation. Deprecated by ethtool (8). (ushort)
parm: duplex:force phy operation. Deprecated by ethtool (8). (int)
parm: autoneg:force phy operation. Deprecated by ethtool (8). (int)
parm: aspm:Enable ASPM. (int)
parm: s5wol:Enable Shutdown Wake On Lan. (int)
parm: rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int)
parm: use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
parm: debug:Debug verbosity level (0=none, ..., 16=all) (int)
 
Last edited:
thanks for the offer but since I have never used dkms, I guess I'll stick to known methods (compile, blacklist old one, mkinitrd, reboot).

I read somebody else recommending doing rmmod [old] + modprobe [new] in a screen session (since the ssh session will be terminated) but I have my doubts that the networking would remain configured and 'service networking restart' tends to not do anything
 
aight compiled the drivers. attempted the no-reboot approach (where you run the autorun.sh the driver comes with in a screen session) but the server did not come back. I had to soft-reboot it, but it started up just fine and I am now seing the transfer rates I was expecting. I had started to worry my iptables rules were causing some loops or whatever but... phew, its all back to normal now (well almost, but thats my fault for not saving a backup of the iptables rules... debian happily overrode the ones I had stored in /etc/iptables/rules)


PS: I still had to disable autoneg and manually set the nic to 1000baseT but I expected that... autoneg just almost never works. But at least with the newer drivers I can do that
 
Last edited:
mir said:
If you want I can send you the source for realtek's source (r8168) layout so in can be compiled and installed through dkms. It works here both against 2.6.32.x and 3.10.x kernels.

You can see this driver has all the options:
modinfo r8168
filename: /lib/modules/3.10.0-10-pve/updates/dkms/r8168.ko
version: 8.037.00-NAPI
license: GPL
description: RealTek RTL-8168 Gigabit Ethernet driver
author: Realtek and the Linux r8168 crew <netdev@vger.kernel.org>
rhelversion: 7.1
srcversion: 4B9CB513BB22516F2449F45
alias: pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias: pci:v000010ECd00008168sv*sd*bc*sc*i*
depends:
vermagic: 3.10.0-8-pve SMP mod_unload modversions
parm: eee_enable:int
parm: speed:force phy operation. Deprecated by ethtool (8). (ushort)
parm: duplex:force phy operation. Deprecated by ethtool (8). (int)
parm: autoneg:force phy operation. Deprecated by ethtool (8). (int)
parm: aspm:Enable ASPM. (int)
parm: s5wol:Enable Shutdown Wake On Lan. (int)
parm: rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int)
parm: use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
parm: debug:Debug verbosity level (0=none, ..., 16=all) (int)
Click to expand...

I'd like to get these and if you could possible point me to a howto for the dkms?

The upgrade has killed the network to the point of un-usability.
 
mir said:
Simply grab this deb package and install it: http://ftp.datanom.net/pve/r8168-dkms_8.037.00_all.deb

1) wget http://ftp.datanom.net/pve/r8168-dkms_8.037.00_all.deb -O /tmp/r8168-dkms_8.037.00_all.deb
2) sudo apt-get install dkms
3) sudo dpkg -i /tmp/r8168-dkms_8.037.00_all.deb
4) sudo modprobe r8168
Click to expand...

Thank you so much!
I've some intel nics being overnighted. :)
However, I'd like to try this. 2 more questions, if you don't mind.

1st Shall we assume I cannot do this remotely?
2nd What might I do if I need to "roll this back?"

Wouldn't ask, but we're live out in the sticks and I do not have another pci express nic on hand. (This machine has no pci slots.)
 
dragonslayr said:
1st Shall we assume I cannot do this remotely?
2nd What might I do if I need to "roll this back?"
Click to expand...
1) As always when dealing with the nic in the machine providing the remote access the safest why is to do it on-site.
2) To roll back you simply need: apt-get purge r8168-dkms
 
mir said:
1) As always when dealing with the nic in the machine providing the remote access the safest why is to do it on-site.
2) To roll back you simply need: apt-get purge r8168-dkms
Click to expand...

Thanks mir,

For completeness, here's what I did. I had to blacklist the original drivers.

apt-get install pve-headers-$(uname -r) build-essential dkms ethtool
wget wget http://ftp.datanom.net/pve/r8168-dkms_8.037.00_all.deb
dpkg -i r8168-dkms_8.037.00_all.deb
echo "blacklist r8169" | tee -a /etc/modprobe.d/blacklist-r8169.conf
modprobe -rfv r8169
modprobe -v r8168
reboot

#ethtool -i eth0
Should now say
version: 8.037.00-NAP
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back