Weird bond issue since 8.1 - i225-V bug ?

P

phastier

New Member
Apr 6, 2023
23
1
3
Hi all,

I have a network issue that started before 8.1, for which I had a workaround, that won't work anymore in 8.1.

I have a 9 node PVE cluster (pve0..8), all using LAGs to a single Edgeswitch, using Jumbo Frames. Nodes 1 to 8 are Mac Minis from various generations, using the internal Ethernet port, and an external Apple Gb Thunderbolt adapter. Node pve0 is a far more recent industrial PC, with 6 Intel 225-V NICs. I use four of them in a bond.

And on top of that, is use Ceph. Icurrently have 6 VMs and 2 CTs running fine.

Everything was working pretty smooth before 8.1, with one exception, I had to set autoneg off on the i225-V manually at each reboot. In fact, the NICs would properly negotiate at 1 Gb/s, but I found logs in the switch that the NIC was permanently negotiating, and this would degrade performance. Ceph was unusable until I turned off autoneg (like ethtool -s enp5s0 speed 1000 duplex full autoneg off). By the way, I had tried to put the change in /etc/network/interfaces, but it never worked, I had to do it manually.

Now since 8.1, this fix does not work anymore at all, and the i225-V interfaces are kept on autoneg, seems the command is ignored.

I tried reducing the bond to only two links, I changed the hash policy, etc... nothing works.

Weird thing is that apart from Ceph and PBS, networking is kind of slow but usable. Ping looks normal. When I SSH to or from that node, there is a small unusal delay which I do not have with the other nodes. Oh, and between node 0 and 3, communication seems absolutely normal.

Any clue ? there are for me two issues :
- why did I have to turn off autoneg on i225-V ??
- why can't I do it anymore in 8.1 ?
 
Using nethogs, I just realized that Ceph is apparently using the network devices directly rather than the bond itself, is that expected ?
 
OK. After a lof of investigation, this is a Jumbo Frame issue.

The cluster member with the i225-V NICs can not use packets above exactly 1990 bytes going out and 1970 bytes from other hosts, though all members of the bond, the bond itself and vmbr are set to 9000.

This used to work with Proxmox 8.0.x, was broken again with Proxmox 8.1.3, none of my configuration has changed.

Please note that I always had to set autonegotiation of speed to off manually for it to work Pre-8.1.
Since 8.1, this does not have any effect, autonegotiation won't be turned off anymore, and hence jumbo frames issues are present again.

And yes, I do not understand why MTU relates to autonegotiation, but this is what I see (including the weird 1990 and 1970 values... by the way, before I get flamed, this is the size of packets in a ping command for testing).

Any help on debugging this ? I will open another thread to motivate people :)

Could it be some buffer size issues ?
 
OK... did use TCPDump. When going over that 1990 value, I have missing bytes. Could that be a buffer size issue ?
In fact, the replies seem to go through about properly.

ping -s 9000 192.168.0.5 will give :

18:11:33.162363 48:a9:8a:be:b5:ff (oui Unknown) > 88:06:5b:50:41:6c (oui Unknown), ethertype IPv4 (0x0800), length 8978: truncated-ip - 64 bytes missing! (tos 0x0, ttl 64, id 5953, offset 0, flags [none], proto ICMP (1), length 9028)
192.168.0.5 > pve0.local: ICMP echo reply, id 62632, seq 8, length 9008

With just 1991 bytes :

18:12:37.630854 48:a9:8a:be:b5:ff (oui Unknown) > 88:06:5b:50:41:6c (oui Unknown), ethertype IPv4 (0x0800), length 2017: truncated-ip - 16 bytes missing! (tos 0x0, ttl 64, id 9245, offset 0, flags [none], proto ICMP (1), length 2019)
192.168.0.5 > pve0.local: ICMP echo reply, id 26072, seq 1, length 1999

With 1990 bytes, no issue :

18:15:54.666121 48:a9:8a:be:b5:ff (oui Unknown) > 88:06:5b:50:41:6c (oui Unknown), ethertype IPv4 (0x0800), length 2032: (tos 0x0, ttl 64, id 23058, offset 0, flags [none], proto ICMP (1), length 2018)
192.168.0.5 > pve0.local: ICMP echo reply, id 42839, seq 2, length 1998
 
Hello all,

Since my firs thread, I moved my network to 2.5 Gb/s, and tried to get back to kernel 6.5. Jumbo frames just won't work in that scenario.
I tried kernel 6.2.16-20 with no luck either.

Rolling back to 6.2.16-19 and manually forcing the port with ethtool will resume a normal behavior.

Some bug was défintely introduced since 6.2.16-20 and above with i226-V chipset.
 
Hi all,

I'm currently in the process of enabling Jumbo Frames on my proxmox nodes, and some of them have an i225-V (rev. 3) NIC. Investigating a bit before proceeding (because I know i225-V are full of issues) I'm fortunately landing on this topic here.

I'm running proxmox 8.2.2, kernel version is 6.8.4-2. Did anybody try to see if the issue mentioned by @phastier is fixed on 6.8 kernels ?

Thanks.
 
fzoc said:
Hi all,

I'm currently in the process of enabling Jumbo Frames on my proxmox nodes, and some of them have an i225-V (rev. 3) NIC. Investigating a bit before proceeding (because I know i225-V are full of issues) I'm fortunately landing on this topic here.

I'm running proxmox 8.2.2, kernel version is 6.8.4-2. Did anybody try to see if the issue mentioned by @phastier is fixed on 6.8 kernels ?

Thanks.
Click to expand...
Hello,

I tried again today with kernel 6.8.4-3, and this is still a no-go.
Jumbo frames with the igc driver (i225 / i226) will just not work with any kernel above 6.2.16-19.

Pinning the kernel to that version using 'proxmox-boot-tool kernel pin 6.2.16-19-pve' will work, and Proxmox 8.2.2 seems to run fine above that kernel, but that seems akward.
 
Hi, is there a guide on how to run Proxmox with lower kernel version? I also having a lot of issues with my i225-v nics.
 
You must log in or register to reply here.
Top Bottom