Problem with RealTek RTL8111 / RTL8168 / RTL8169 On-Board Ethernet...

[oeginc]

See this post for usage details

Motherboard is a Gigabyte MA78GM-S2HP which claims to have an RTL8111C On-Board Gigabit Ethernet.

We are running 50 MiB to 100 MiB/sec on average over this interface and we are constantly getting kernel panics. I don't know the exact message because it doesn't seem to save it anywhere, but when I looked at the screen it seemed to be related to the R8169 driver it was using.

I did some research and it appears this is a "known" problem that under high load the RTL8169 kernel panics from years ago (2007'ish?), has this really not been fixed in recent kernels?

Another thing I don't understand is if this is a RTL8111C, why does linux think it is an RTL8169?

From dmesg:
=========
eth0: RTL8168c/8111c at 0xffffc2000081e000, 00:1f:d0:d1:8d:d7, XID 3c4000c0 IRQ 209

r8169: eth0: link up

I checked on Realtek's website, and there are in fact different drivers for the RTL8111C and the RTL8169, but before I started mucking around on a live system I figured I'd ask here.

 

[tom]

post your pveversion -v

 

[oeginc]

server01:~# pveversion -v
pve-manager: 1.6-2 (pve-manager/1.6/5087)
running kernel: 2.6.18-2-pve
proxmox-ve-2.6.18: 1.5-5
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-18
pve-firmware: 1.0-8
libpve-storage-perl: 1.0-14
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-7
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-6

 

[tom]

can you try the latest 2.6.18er? if this does not help, try the 2.6.32 kernel branch.

 

[oeginc]

Has this specific problem been addressed in those kernels? I can't afford to try it on a whim, these are production servers with a lot of traffic...

I'm running a mix of OpenVZ & KVM machines (90% / 10%) and using snapshot backups if that matters.

What is the recommended way of trying these from this point?

 

[tom]

Has this specific problem been addressed in those kernels? I can't afford to try it on a whim, these are production servers with a lot of traffic...

I'm running a mix of OpenVZ & KVM machines (90% / 10%) and using snapshot backups if that matters.

What is the recommended way of trying these from this point?

Just try it on a test machine.

If you want stable servers with good NIC´s you should use server hardware - why are you using cheap desktop main boards as servers?

Our 2.6.18 kernel supports more or less the same then the RHEL5 kernel does.

 

[sumsum]

As tom mentioned. go for a good NIC or compile yourself a better r8168 driver version. (yes r8168 works as well - and better)
It is known by the community, that Realtek does not has a good driver implementation in "old" kernels.
On my cheap Testing Server, which also has a onboard Realtek NIC, I could avoid high traffic problems by compiling a newer driver.

cd /tmp
wget [URL]http://download.hetzner.de/drivers/r8168-8.015.00.tar.bz2[/URL]
tar xjf r8168-8.015.00.tar.bz2

cd r8168-8.015.00
make all

depmod -a

Blackllist the r8169 driver

echo "blacklist r8169" >> /etc/modprobe.d/blacklist

force the implementation of the new driver into initrd.

  echo "r8168" >> /etc/initramfs-tools/modules

create new initrd

 update-initramfs -v -u -k `uname -r`

reboot the server and test under high traffic. the short interruption during high traffic is gone.

This is just a workaround ! go for better hardware if you use proxmox for productive environments.

 

[oeginc]

@sumsum - Thanks for the information, however when I try to do the above, I get:

update-initramfs: /boot/initrd.img-2.6.18-2-pve has been altered.
update-initramfs: Cannot update. Override with -t option.

I am not sure if I should force the update, or if there is some specific reason my proxmox initramfs has been modified outside of the normal channels...

@tom - I'm not going to get into the debate on whether I should be using server class hardware, that has nothing to do with this particular problem. If Realtek has a problem with their controller, then recommend a good alternative, don't just say "Use a real server". What do you recommend for an alternative gigabit ethernet card, Intel? The fact of the matter is that any hardware that doesn't have a good driver implementation will have problems.. I use Dell 1950's with dual quad core CPU's, 16GB RAM, dual gig-E's, and RAID controllers for a bunch of my servers - but I've still had problems with them too. Nothing is perfect, and yes server class hardware does have a tendency to survive harsh environments better, but in this particular case this hardware is what I have to deal with and replacing it is not an option. I could however get my client to opt for a new/better ethernet card if you had one you could suggest.

Getting back to the original question, what is the recommended way of trying an alternate kernel - and is there an easy way to recover from that should it not work?

 

[tom]

as already posted, realtek and the kernel community are not best friends, their driver policy is not "optimal". if you use server hardware, you got most time Intel or Broadcom NIC´s where we at Proxmox make sure that we always got the best and latest drivers in our kernels - same as other server targeted distributions.

you can choose whatever you want but a server distribution is best on server hardware and you have a good change for a reliable and future proof combination.

 

[oeginc]

As tom mentioned. go for a good NIC or compile yourself a better r8168 driver version. (yes r8168 works as well - and better)
It is known by the community, that Realtek does not has a good driver implementation in "old" kernels.
On my cheap Testing Server, which also has a onboard Realtek NIC, I could avoid high traffic problems by compiling a newer driver.

This is just a workaround ! go for better hardware if you use proxmox for productive environments.

Ok, I have a couple Intel ET NIC's on order, but in the meantime I downloaded the latest ethernet drivers from Realtek, compiled, installed, and forced the update to initramfs.

So far, everything is working great (not only did my kernel panics seem to go away, but our throughput is even higher now!).

Thank you very much for your help.