Single port on NIC randomly disconnecting

S

Sakreton

New Member
Sep 21, 2024
6
0
1
Hey,

i am running a 3 node PVE + Ceph Cluster on 3x Minesforum MS-A2.
The nodes are running identical hardware.

On one of the 3 nodes

At supposedly random times one 2.5Gbit Port on a RTL8125 2.5GbE Controller (rev 05) NIC disconnects.
The affected port is used for the guest traffic thats probably the only reason why i even noticed it in the first place lol.
All other NICs (2x Ceph), even the other port (used for Corosync to a different switch) on this NIC are not affected.

This also never occured on any other node.

I have tried a different Switch-Port and entire switch, its the host nic not the switch side.
I have also obviously rebooted this node.

I am running PVE 9.2.2, the problem persists for some time already since troubleshooting is a bit difficult since i can't force the problem to appear.

Any ideas before i approach minisforum any help or ideas are greatly appreciated! Thanks in advance and for reading this! :>

Proxmox Log:
May 27 20:07:44 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Down
May 27 20:07:44 prox-ganymede kernel: vmbr0: port 1(nic0) entered disabled state
May 27 20:07:52 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Up - 2.5Gbps/Full - flow control off
May 27 20:07:52 prox-ganymede kernel: vmbr0: port 1(nic0) entered blocking state
May 27 20:07:52 prox-ganymede kernel: vmbr0: port 1(nic0) entered forwarding state
May 27 20:07:54 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Down
May 27 20:07:54 prox-ganymede kernel: vmbr0: port 1(nic0) entered disabled state
May 27 20:07:55 prox-ganymede pvestatd[1566]: PBS_VM: error fetching datastores - 500 Can't connect to PBS:8007 (Connection timed out)
May 27 20:07:55 prox-ganymede pvestatd[1566]: status update time (7.093 seconds)
May 27 20:07:56 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Up - 1Gbps/Full - flow control off
May 27 20:07:56 prox-ganymede kernel: vmbr0: port 1(nic0) entered blocking state
May 27 20:07:56 prox-ganymede kernel: vmbr0: port 1(nic0) entered forwarding state
May 27 20:08:00 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Down
May 27 20:08:00 prox-ganymede kernel: vmbr0: port 1(nic0) entered disabled state
May 27 20:08:04 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Up - 2.5Gbps/Full - flow control off
May 27 20:08:04 prox-ganymede kernel: vmbr0: port 1(nic0) entered blocking state
May 27 20:08:04 prox-ganymede kernel: vmbr0: port 1(nic0) entered forwarding state

Switch Log:
2026-05-27 20:07:41 interface,info ether3(TO_GANYMEDE) link down
2026-05-27 20:07:49 interface,info ether3(TO_GANYMEDE) link up (speed 2.5G, full duplex)
2026-05-27 20:07:51 interface,info ether3(TO_GANYMEDE) link down
2026-05-27 20:07:52 interface,info ether3(TO_GANYMEDE) link up (speed 2.5G, full duplex)
2026-05-27 20:07:55 interface,info ether3(TO_GANYMEDE) link down
2026-05-27 20:08:02 interface,info ether3(TO_GANYMEDE) link up (speed 2.5G, full duplex)
 
The r8169 network driver is notorious in Linux and Proxmox for causing exactly this type of flapping behavior under heavy load or bridging.

apt update
apt install -y pve-headers pve-kernel-helper pve-edk2-firmware dkms r8168-dkms

install differnt drivers
 
  • Like
Reactions: Sakreton
If it's just one out of three you might have a bad patch cable. Did you also try another patch cable?
Also, Minisforum boxes are known to thermal issues. Maybe this particular system gets hotter than the other two?
 
  • Like
Reactions: Sakreton
Thanks for the possible solutions!
I am trying them now, i will update this Post as soon as i know if something worked^^
 
So updates...
It wasn't the cable, the drivers are now as far as i can tell bundled in the newer kernels (running PVE 9.2.10 latest at time of writing) so those are more or less out of the question.


I found out yesterday, that the MS-A2, has two different 2.5Gbit NICs. One i226-V and one RTL 8125.
(dont even ask how i missed this pls)
The RTL8125 was uses for VM Traffic and Migration traffic, the i226-v for corosync.

For the sake of testing, i flipped this around.
The Problem still persists. Even on the other NIC, which makes less to no sense in my eyes...

I have disabled Ethernet auto negotiation on the switch and set it fixed to the speed i want on that port (2.5Gbit).

I have thought about thermals, but that that should not be the issue, since this happens temperature independent. On all nodes of the 3 Node Cluster.

After i switched the roles and therefore the connection of the NICs, i came to the conclusion that this has to be a software / config problem.
The former Unstable NIC (RTL8125) is now stable and the former stable NIC (i226-V) is now unstable.

I am completely out of ideas.

This is "/etc/network/interfaces" from one of the nodes, they are all configured exactly the same.

Code: 
auto lo
iface lo inet loopback

auto nic0 #RTL8125
iface nic0 inet static
        address 192.168.0.31/24
#PVE_CLUSTER

auto nic1 #I226-V
iface nic1 inet manual
#BRIDGED_PVE_GUESTS

auto nic2
iface nic2 inet manual
        mtu 9000
#MBR_CEPH_REPLICATION

auto nic3
iface nic3 inet manual
        mtu 9000
#MBR_CEPH_REPLICATION

iface nic4 inet manual

iface wlp6s0 inet manual

auto bond0
iface bond0 inet static
        address 192.168.1.31/24
        bond-slaves nic2 nic3
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9000
#CEPH_REPLICATION

auto vmbr0
iface vmbr0 inet manual
        bridge-ports nic1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#VM_GUESTS

auto vlan15
iface vlan15 inet static
        address 10.1.0.31/24
        gateway 10.1.0.254
        vlan-raw-device vmbr0
#PVE_MANAGEMENT
 
Maybe you can try to pin the speed for the flapping interface to 1G and use it for corosync.

1G should be sufficient, and maybe the problems are related to the 2.5G firmware/driver implemtations.

Firmwareupdate might also be a solution, but O guess you already did that in case there is one available.

BR, Lucas
 
May 27 20:07:52 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Up - 2.5Gbps/Full - flow control off
May 27 20:07:56 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Up - 1Gbps/Full - flow control off
May 27 20:08:04 prox-ganymede kernel: r8169 0000:03:00.0 nic0: Link is Up - 2.5Gbps/Full - flow control off
Click to expand...
why would it negotiate at 1g? it does look like a hardware issue. you're not seeing the driver crash out in the kernel log. yes, pin the switch port to 1g. not sure why you think 2.5 is more stable, but I can tell you like speed!
 
Thanks for the replies!

Maybe you can try to pin the speed for the flapping interface to 1G and use it for corosync.
Click to expand...
The problem is, its not one of the two interfaces flapping, its the interface that "holds" the vmbr that flaps, regardless of hardware "below".
The corosync connection goes to a different switch only supporting 1Gbit.

I just tried a different switch with a 10GBaseT multigig RJ45 SFP+ module, my only other way of testing 2.5Gbit.
->Still happens got lucky with the timing, since this still happens randomly and i have no way of reproducing it.

yes, pin the switch port to 1g. not sure why you think 2.5 is more stable, but I can tell you like speed!
Click to expand...
Never said that, i said afaik is the i226-v more stable than the 8125, both 2.5Gbit NICs. But ofc i would prefer 2.5Gbit over 1Gbit.

why would it negotiate at 1g? it does look like a hardware issue.
Click to expand...
Thats the part i also dont understand at all, i disabled auto negotiate on the switch and fixed it to 2.5Gbit, so thats not happening anymore but the random disconnects still happen...
This problem occurs on all 3 nodes switch independent so it kind of doesn't make sense that its a hardware issue, no?
I mean sure it still could be but i think thats highly unlikely.

I am tempted to just build another bond with both the 2.5Gbit Links and share the Link for both corosync and Guest traffic.

Code: 
Aug 16 17:02:07 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Aug 16 17:02:07 prox-io kernel: vmbr0: port 2(nic1) entered blocking state
Aug 16 17:02:07 prox-io kernel: vmbr0: port 2(nic1) entered forwarding state
Aug 16 17:02:09 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Down
Aug 16 17:02:09 prox-io kernel: vmbr0: port 2(nic1) entered disabled state
Aug 16 17:02:13 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
Aug 16 17:02:13 prox-io kernel: vmbr0: port 2(nic1) entered blocking state
Aug 16 17:02:13 prox-io kernel: vmbr0: port 2(nic1) entered forwarding state
Aug 16 17:02:19 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Down
Aug 16 17:02:19 prox-io kernel: vmbr0: port 2(nic1) entered disabled state
Aug 16 17:02:32 prox-io pvestatd[1623]: metrics send error 'InfluxDB': 500 Can't connect to ****** (Temporary failure in name resolution)
### COMMENT here i switched the connection to the above mentioned other switch.
Aug 16 17:02:37 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Aug 16 17:02:37 prox-io kernel: vmbr0: port 2(nic1) entered blocking state
Aug 16 17:02:37 prox-io kernel: vmbr0: port 2(nic1) entered forwarding state
Aug 16 17:02:42 prox-io pvestatd[1623]: status update time (42.012 seconds)
Aug 16 17:02:42 prox-io corosync[1492]:   [KNET  ] rx: host: 2 link: 1 is up
Aug 16 17:02:42 prox-io corosync[1492]:   [KNET  ] link: Resetting MTU for link 1 because host 2 joined
Aug 16 17:02:42 prox-io corosync[1492]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 16 17:02:42 prox-io corosync[1492]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Aug 16 17:02:43 prox-io corosync[1492]:   [KNET  ] rx: host: 3 link: 1 is up
Aug 16 17:02:43 prox-io corosync[1492]:   [KNET  ] link: Resetting MTU for link 1 because host 3 joined
Aug 16 17:02:43 prox-io corosync[1492]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 16 17:02:43 prox-io corosync[1492]:   [KNET  ] pmtud: Global data MTU changed to: 1397
### COMMENT do i btw read this correct, that corosync still switched active links? The corosync link never went down and should therefore not failover to one of the other links...
Aug 16 17:04:16 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Down
Aug 16 17:04:16 prox-io kernel: vmbr0: port 2(nic1) entered disabled state
Aug 16 17:04:17 prox-io corosync[1492]:   [KNET  ] link: host: 2 link: 1 is down
Aug 16 17:04:17 prox-io corosync[1492]:   [KNET  ] link: host: 3 link: 1 is down
Aug 16 17:04:17 prox-io corosync[1492]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 16 17:04:17 prox-io corosync[1492]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 16 17:04:20 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Aug 16 17:04:20 prox-io kernel: vmbr0: port 2(nic1) entered blocking state
Aug 16 17:04:20 prox-io kernel: vmbr0: port 2(nic1) entered forwarding state
Aug 16 17:04:26 prox-io corosync[1492]:   [KNET  ] rx: host: 2 link: 1 is up
Aug 16 17:04:26 prox-io corosync[1492]:   [KNET  ] link: Resetting MTU for link 1 because host 2 joined
Aug 16 17:04:26 prox-io corosync[1492]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 16 17:04:26 prox-io corosync[1492]:   [KNET  ] rx: host: 3 link: 1 is up
Aug 16 17:04:26 prox-io corosync[1492]:   [KNET  ] link: Resetting MTU for link 1 because host 3 joined
Aug 16 17:04:26 prox-io corosync[1492]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 16 17:04:26 prox-io corosync[1492]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Aug 16 17:04:27 prox-io pvestatd[1623]: status update time (5.338 seconds)
Aug 16 17:05:48 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Down
Aug 16 17:05:48 prox-io kernel: vmbr0: port 2(nic1) entered disabled state
Aug 16 17:05:49 prox-io corosync[1492]:   [KNET  ] link: host: 2 link: 1 is down
Aug 16 17:05:49 prox-io corosync[1492]:   [KNET  ] link: host: 3 link: 1 is down
Aug 16 17:05:49 prox-io corosync[1492]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 16 17:05:49 prox-io corosync[1492]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 16 17:05:52 prox-io kernel: igc 0000:04:00.0 nic1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX
Aug 16 17:05:52 prox-io kernel: vmbr0: port 2(nic1) entered blocking state
Aug 16 17:05:52 prox-io kernel: vmbr0: port 2(nic1) entered forwarding state
Aug 16 17:05:58 prox-io pvestatd[1623]: status update time (5.769 seconds)
Aug 16 17:05:58 prox-io corosync[1492]:   [KNET  ] rx: host: 2 link: 1 is up
Aug 16 17:05:58 prox-io corosync[1492]:   [KNET  ] link: Resetting MTU for link 1 because host 2 joined
Aug 16 17:05:58 prox-io corosync[1492]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 16 17:05:58 prox-io corosync[1492]:   [KNET  ] rx: host: 3 link: 1 is up
Aug 16 17:05:58 prox-io corosync[1492]:   [KNET  ] link: Resetting MTU for link 1 because host 3 joined
Aug 16 17:05:58 prox-io corosync[1492]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 16 17:05:58 prox-io corosync[1492]:   [KNET  ] pmtud: Global data MTU changed to: 1397
 
well, my intention was to test the firmware implementation of the interfaces. (The kernel might never see any logs of that)
Just mentioned it, because I have already seen from time to time networkcards, that where not capable to deliver all their advertised speeds. (e.g. 10G cards failing connected to a 1G Switch - which should be compatible, defined by the standard for 10G-BaseT)

so maybe you can give it a shot.
I am also not sure, what kind of hardware from which vendor you use.
Of course it would be nice to use 2.5G Speed, if it works reliable. :)
------

In the journal you added the mode of flow control changed: from ``` Flow Control: RX/TX ``` to ``` Flow Control: RX ``` ,
maybe that observation is helpful.

BR, Lucas
 
intel 2.5 nics have had many complaints (even sometimes on 1g link) for YEARS, and Realteks have been getting more robust/better. in the end you have to go with what works for your situation.

Gemini says
Step 1: Disable Energy Efficient Ethernet (EEE) (with ethtool)
Step 2: Disable PCIe Active State Power Management (ASPM) (in BIOS)
 
You must log in or register to reply here.
Top Bottom
Back