Network Card negotiates down from 1 Gig to 100 Mb

A

alitvak69

Renowned Member
Oct 2, 2015
105
3
83
I have Dell R720xd with BCM 5720 nic. I upgraded firmware replaced boards, changed the switch ports, switch and cable. I still get card going down and changing speed to 100 from 1gbit. I can get it up to 1000 with ethtool but it comes back down to 100 and interrupts connection multiple times. Any clues ?
 
What does the dmesg out show? And from my personal experience, I had to set & reset the link speed on the switch itself, to get past a similar event.
 
[Thu Oct 17 04:10:33 2019] bpfilter: Loaded bpfilter_umh pid 1337 [Thu Oct 17 04:10:37 2019] tg3 0000:18:00.0 eno1: Link is up at 1000 Mbps, full duplex [Thu Oct 17 04:10:37 2019] tg3 0000:18:00.0 eno1: Flow control is on for TX and on for RX [Thu Oct 17 04:10:37 2019] tg3 0000:18:00.0 eno1: EEE is disabled [Thu Oct 17 04:10:37 2019] vmbr0: port 1(eno1) entered blocking state [Thu Oct 17 04:10:37 2019] vmbr0: port 1(eno1) entered forwarding state [Thu Oct 17 04:10:37 2019] IPv6: ADDRCONF(NETDEV_CHANGE): vmbr0: link becomes ready [Thu Oct 17 04:10:39 2019] EXT4-fs (dm-7): mounted filesystem with ordered data mode. Opts: (null) [Thu Oct 17 04:10:39 2019] kauditd_printk_skb: 6 callbacks suppressed [Thu Oct 17 04:10:39 2019] audit: type=1400 audit(1571299840.208:18): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-101_</var/lib/lxc>" pid=6822 comm="apparmor_parser" [Thu Oct 17 04:10:39 2019] fwbr101i0: port 1(fwln101i0) entered blocking state [Thu Oct 17 04:10:39 2019] fwbr101i0: port 1(fwln101i0) entered disabled state [Thu Oct 17 04:10:39 2019] device fwln101i0 entered promiscuous mode [Thu Oct 17 04:10:39 2019] fwbr101i0: port 1(fwln101i0) entered blocking state [Thu Oct 17 04:10:39 2019] fwbr101i0: port 1(fwln101i0) entered forwarding state [Thu Oct 17 04:10:39 2019] vmbr0: port 2(fwpr101p0) entered blocking state [Thu Oct 17 04:10:39 2019] vmbr0: port 2(fwpr101p0) entered disabled state [Thu Oct 17 04:10:39 2019] device fwpr101p0 entered promiscuous mode [Thu Oct 17 04:10:39 2019] vmbr0: port 2(fwpr101p0) entered blocking state [Thu Oct 17 04:10:39 2019] vmbr0: port 2(fwpr101p0) entered forwarding state [Thu Oct 17 04:10:39 2019] fwbr101i0: port 2(veth101i0) entered blocking state [Thu Oct 17 04:10:39 2019] fwbr101i0: port 2(veth101i0) entered disabled state [Thu Oct 17 04:10:39 2019] device veth101i0 entered promiscuous mode [Thu Oct 17 04:10:39 2019] eth0: renamed from veth1UAO0V [Thu Oct 17 04:10:40 2019] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [Thu Oct 17 04:10:40 2019] fwbr101i0: port 2(veth101i0) entered blocking state [Thu Oct 17 04:10:40 2019] fwbr101i0: port 2(veth101i0) entered forwarding state [Thu Oct 17 04:10:48 2019] Process accounting resumed [Thu Oct 17 04:11:24 2019] tg3 0000:18:00.0 eno1: Management side-band traffic will be interrupted during phy settings change [Thu Oct 17 04:11:24 2019] tg3 0000:18:00.0 eno1: Link is down [Thu Oct 17 04:11:24 2019] vmbr0: port 1(eno1) entered disabled state [Thu Oct 17 04:11:24 2019] tg3 0000:19:00.1 eno4: Management side-band traffic will be interrupted during phy settings change [Thu Oct 17 04:11:24 2019] tg3 0000:18:00.1 eno2: Management side-band traffic will be interrupted during phy settings change [Thu Oct 17 04:11:24 2019] tg3 0000:19:00.0 eno3: Management side-band traffic will be interrupted during phy settings change [Thu Oct 17 04:11:27 2019] tg3 0000:18:00.0 eno1: Link is up at 100 Mbps, full duplex [Thu Oct 17 04:11:27 2019] tg3 0000:18:00.0 eno1: Flow control is on for TX and on for RX [Thu Oct 17 04:11:27 2019] tg3 0000:18:00.0 eno1: EEE is disabled [Thu Oct 17 04:11:27 2019] vmbr0: port 1(eno1) entered blocking state [Thu Oct 17 04:11:27 2019] vmbr0: port 1(eno1) entered forwarding state
 
I just booted live centos 7 cd. So I will wait and see if the card changes its ways. Right now it is 1G and up.
 
alitvak69 said:
[Thu Oct 17 04:11:24 2019] tg3 0000:18:00.0 eno1: Management side-band traffic will be interrupted during phy settings change
Click to expand...
Is the out-of-band management interface (ipmi) running on the same port?
 
Hmm,

No ipmi interface (iDRAC) is using its dedicated port. So far centos 7 live CD keeps interface up at 1000 Gbit. I am not doing anything with the box yet just watching.
 
Check with modinfo what module version the tg3 has, I suppose it will differ from the Proxmox VE one.
 
The version in CentOS is 3.137 and so was the version in Proxmox 6

[Thu Oct 17 04:10:26 2019] tg3.c:v3.137 (May 11, 2014)
 
So far with multiple hours of iperf3 centos7 with tg3 driver and its native kernel stays on 1000 without going down. Is there something I can do other than using different card vendor? This was a preproduction test for us and is not going very good for proxmox.
 
Try with different pve-kernels to see, if it is a regression.
 
I will do that. Meanwhile a completely different driver intel chipset e1000 exhibits the same issues. It boots to 1000 mbit and in 45 seconds port link goes down an comes up at 100 mbit. Ethtool also saying that card advertising 100 mg full duplex while being capable of doing 1 gigabit full.
 
I attempted multiple proxmox kernels and behavior is the same. I even installed a native Debian kernel which is 4.19 vs proxmox 5.0. It is exactly the same behavior with all kernels it boots to 1000 Mbit and then ~38 seconds later port goes into a disabled mode and comes back as 100 Mbit. At this point, it becomes unstable as it becomes nonresponsive for a few seconds from time to time.
 
Ubuntu 18.0.4. wit kernel 5.0.23 has no problems either. So doesn't seem to be a generic kernel/driver issue here.
 
Clean reinstall of PV 6 with non-subscription + Debian sources enabled. Two reboots no down negotiate so far, waiting and watching for any changes.
 
If anyone cares, I have no clue why I went through this ordeal. The only guess I have so far is the last time I had enabled both tests and no-subscription repositories. This time I didn't touch the test source at all. I updated to the latest no-sub and all Debian and rebooted several times. I also ran a number of iperf3 tests. So far the card stays on 1 Gbit. I will keep the box on until tomorrow morning to see if anything changes. @Alwin thank you for your very sane advice.
 
  • Like
Reactions: Alwin
I had this happen a few times on a Dell R410. Replaced the network cable, Cat 6, and connected it into a different switch but still once in a while it drops to 100 Mbit. I ran this server for a few years with Ubuntu and always had 1G connection. I switched to Proxmox a couple months ago.

So far I have not found anything happening around the time the link goes down to 100 Mbps.

Code: 
Dec 06 08:49:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 06 08:49:01 pve systemd[1]: pvesr.service: Succeeded.
Dec 06 08:49:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 06 08:49:22 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Down
Dec 06 08:49:22 pve kernel: vmbr0: port 1(eno1) entered disabled state
Dec 06 08:49:25 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Up, 1000 Mbps full duplex
Dec 06 08:49:25 pve kernel: vmbr0: port 1(eno1) entered blocking state
Dec 06 08:49:25 pve kernel: vmbr0: port 1(eno1) entered forwarding state
Dec 06 08:49:26 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Down
Dec 06 08:49:26 pve kernel: vmbr0: port 1(eno1) entered disabled state
Dec 06 08:49:28 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Up, 1000 Mbps full duplex
Dec 06 08:49:28 pve kernel: vmbr0: port 1(eno1) entered blocking state
Dec 06 08:49:28 pve kernel: vmbr0: port 1(eno1) entered forwarding state
Dec 06 08:49:29 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Down
Dec 06 08:49:29 pve kernel: vmbr0: port 1(eno1) entered disabled state
Dec 06 08:49:31 pve pvestatd[1510]: storage 'nasBackups' is not online
Dec 06 08:49:32 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Up, 1000 Mbps full duplex
Dec 06 08:49:32 pve kernel: vmbr0: port 1(eno1) entered blocking state
Dec 06 08:49:32 pve kernel: vmbr0: port 1(eno1) entered forwarding state
Dec 06 08:49:33 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Down
Dec 06 08:49:33 pve kernel: vmbr0: port 1(eno1) entered disabled state
Dec 06 08:49:35 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Up, 1000 Mbps full duplex
Dec 06 08:49:35 pve kernel: vmbr0: port 1(eno1) entered blocking state
Dec 06 08:49:35 pve kernel: vmbr0: port 1(eno1) entered forwarding state
Dec 06 08:49:36 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Down
Dec 06 08:49:36 pve kernel: vmbr0: port 1(eno1) entered disabled state
Dec 06 08:49:39 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Up, 1000 Mbps full duplex
Dec 06 08:49:39 pve kernel: vmbr0: port 1(eno1) entered blocking state
Dec 06 08:49:39 pve kernel: vmbr0: port 1(eno1) entered forwarding state
Dec 06 08:49:40 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Down
Dec 06 08:49:40 pve kernel: vmbr0: port 1(eno1) entered disabled state
Dec 06 08:49:41 pve pvestatd[1510]: storage 'nasBackups' is not online
Dec 06 08:49:41 pve kernel: bnx2 0000:01:00.0 eno1: NIC Copper Link is Up, 100 Mbps full duplex
Dec 06 08:49:41 pve kernel: vmbr0: port 1(eno1) entered blocking state
Dec 06 08:49:41 pve kernel: vmbr0: port 1(eno1) entered forwarding state
 
pclements said:
I had this happen a few times on a Dell R410. Replaced the network cable, Cat 6, and connected it into a different switch but still once in a while it drops to 100 Mbit.
Click to expand...
Did you try a different NIC port? And is all the firmware up-to-date?
 
I am not using tuned. Firmware is up to date. I changed the ethernet cable again and upgraded from proxmox 6.0-4 to 6.1-3. It has held steady at 1G for the last couple days.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back