[SOLVED] 802.3ad bond0 to CISCO + vmbr0 ARP issues...

[alexolivan]

Hi forum!

Ok, I know this one is old... but something has to be very wrong to be still happening:

Hi have a single Node... no cluster.
This node has 3 interfaces, one for Management, two for bonding.
Switch is a CISCO Catalyst 3560g
The setup is a typical, no-VLAN, bridging between bond0, vmbr0 (which has IP addressing) and the VMs.
Setup was initially done on GUI, but has been comented/modified in tests, but basically is the original
Bonding works... everything works 99% (as did some 18 months ago when I first tried this)

The problem comes with VMs not geting connection with other VLAN host on the switch.
(The bond0 is a CISCO port-channel interface on the switch side member of a VLAN).
Traffic goes untagged to the PROXMOX server... as the port-channel (and its interfaces) are in access mode.

There seems (to me) some kind of ARP issue.
I can ping VMs from the Proxmox vmbr0 interface, and vice-versa.
As soon as I ping a LInux VM that way, it gains connectivity, and keeps it almost forever... (so it basically works)
Windows VM can ping vmbr0 and be pinged from vmbr0... but it does not gain connectivity to the exterior once done.

...but more strange is that, if I do (from my PC connected to a switch port on the VLAN/broadcast domain) an ARP ping to the windows VM IP address (using arping)... it loses some early frames, then it works.... and then, for a while, the windows VM networking works (can ping, be pinged, and the VM starts to be able to comunicate outside PROXMOX)... but after a short while it fails again... until arpinged... so this leads me to think there's some ARP issue here.

There are similar issues found regarding other Linux Based Virtualization technologies, some very old, so I hope I'm doing something wrong or there is some trick known by PROXMOX experienced users... it is hard to me to believe I'm the only one having this issue if same thing has happened to me the two times I tried this.


Best regards!

 

[manu]

if you suspect ARP to be an issue, you should have a look if arp requests are send and received properly

on a PC located on a acces port on the same VLAN, when the problem occurs
* flush your arp cache ( here arp -ad on the FreeBSD system I am using)
* inspect with tcpdump if the whohas request for the VM IP whose NIC is attached to vmbr0 is sent and answered

here for instance I am inspecting the initial whohas request from host 192.168.16.24, trying to reach 192.168.16.5

tcpdump -nnn -i vtnet0 -e "arp  and (host 192.168.16.24 or host 192.168.16.5)"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes

13:50:56.629227 32:36:31:39:65:61 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.16.5 tell 192.168.16.24, length 28
13:50:56.629450 0c:c4:7a:31:d2:5c > 32:36:31:39:65:61, ethertype ARP (0x0806), length 60: Reply 192.168.16.5 is-at 0c:c4:7a:31:d2:5c, length 46

 

[manu]

also does setting the bonding to a "simpler" mode like active/passive fixes the issue ? this can help to pinpoint hwere the problem is taking place

 

[alexolivan]

Hi...

Actually I have stopped the lab, I will report in a pair of days when I'll have the oportunity to put my hands again on that hardware...

In the mean time (before stopping), what I have tried is the following (and results have been promising, although I would not mark this as solved before seing such an scenario up and running for at least 24 hours without losing connection):

- I have disabled via ethtool gso/tso on all bond interfaces (win guest didn't use virtio...but just in case).
- I added a second dual NIC to up to 4 interfaces in the bond... (Testing throughput... probably unrelated)
- I set the bond balancing from layer2+3 to default layer2
- I have set on the Cisco Catalyst, the port-channel load-balance mode from dst-ip (throughput test) back to default src-mac.
- Reset MTU to default levels (again, plying with throughput)

my TODO now:
- investigate tcpdumps for ARP activity... will be fun! good idea!
- investigate about active/pasive mode bonding against a Catalyst... to be sincere, I do not know how to do it at this moment... I'm barely CCNA skilled... etherchannel is very basically covered and LACP is encouraged, and well... it has worked to me so no experience beyond that!.
- try to revert the changes one by one and see if one of them is really the single difference maker.

Best regards!

 

[alexolivan]

Hi again.

I have done tcpdump tests as suggested for ARP analisis.
The problems persist, but as I'm starting to test Proxmox 5, I'm hoping this problem is solved somehow with those newer software/kernels, because It is very weird how 3.4 behaves.

The results of tcp dup are somehow weird:

17:01:09.283245 a0:48:1c:dd:38:28 > f2:ff:d3:9b:b7:a3, ethertype ARP (0x0806), length 42: Request who-has 192.168.100.7 tell 192.168.100.212, length 28
17:01:10.307238 a0:48:1c:dd:38:28 > f2:ff:d3:9b:b7:a3, ethertype ARP (0x0806), length 42: Request who-has 192.168.100.7 tell 192.168.100.212, length 28
17:01:11.331250 a0:48:1c:dd:38:28 > f2:ff:d3:9b:b7:a3, ethertype ARP (0x0806), length 42: Request who-has 192.168.100.7 tell 192.168.100.212, length 28
17:01:12.419308 a0:48:1c:dd:38:28 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.100.7 tell 192.168.100.212, length 28
17:01:12.419713 f2:ff:d3:9b:b7:a3 > a0:48:1c:dd:38:28, ethertype ARP (0x0806), length 60: Reply 192.168.100.7 is-at f2:ff:d3:9b:b7:a3, length 46
17:01:12.419771 f2:ff:d3:9b:b7:a3 > a0:48:1c:dd:38:28, ethertype ARP (0x0806), length 60: Reply 192.168.100.7 is-at f2:ff:d3:9b:b7:a3, length 46
17:01:12.419774 f2:ff:d3:9b:b7:a3 > a0:48:1c:dd:38:28, ethertype ARP (0x0806), length 60: Reply 192.168.100.7 is-at f2:ff:d3:9b:b7:a3, length 46
17:01:12.419776 f2:ff:d3:9b:b7:a3 > a0:48:1c:dd:38:28, ethertype ARP (0x0806), length 60: Reply 192.168.100.7 is-at f2:ff:d3:9b:b7:a3, length 46
17:01:17.329036 f2:ff:d3:9b:b7:a3 > a0:48:1c:dd:38:28, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.212 (a0:48:1c:dd:38:28) tell 192.168.100.7, length 46

here my PC (192.168.100.212) is asking 3 times L2 unicast for ARP update with no reply by the windows VM (192.168.100.7), then it goes L2 broadcast and it gets 4 replies! (I guess, bond0 is made up of 4 links, so 1 reply per bonded link) .... and then the windows host seems to ask four times back to my PC...

Linux hosts, have the same problem initially, but just a few moments, after early pings, connection is stablished, and it becomes solid, and, by letting tcpdump on, with time I see those excerpts:

17:01:31.499648 02:00:00:85:4a:53 > a0:48:1c:dd:38:28, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.212 tell 192.168.100.11, length 46
17:01:31.499704 a0:48:1c:dd:38:28 > 02:00:00:85:4a:53, ethertype ARP (0x0806), length 42: Reply 192.168.100.212 is-at a0:48:1c:dd:38:28, length 28
17:01:31.555247 a0:48:1c:dd:38:28 > 02:00:00:85:4a:53, ethertype ARP (0x0806), length 42: Request who-has 192.168.100.11 tell 192.168.100.212, length 28
17:01:31.555466 02:00:00:85:4a:53 > a0:48:1c:dd:38:28, ethertype ARP (0x0806), length 60: Reply 192.168.100.11 is-at 02:00:00:85:4a:53, length 46

Here, a Linux VM (192.168.100.11) seems to be 'refreshing' its ARP entries by querying my PC.
The handshake looks more simple, no quadruple activity.

By just examinig ARP activity for the Windows VM (filering tcpdum just with it's IP, and let it go to see what happens) I see activity that looks as if the Host is having trouble all teh time with excerps like this, as it tries to find the gateway (192.168.100.1):

17:25:07.087021 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:07.828053 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:08.828049 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:10.093694 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:10.827939 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:11.827939 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:16.093732 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:16.827928 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:17.827977 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:28.093693 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:28.827978 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
17:25:29.827980 f2:ff:d3:9b:b7:a3 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.100.1 tell 192.168.100.7, length 46
... and no reply here from the router...

So, my overall feeling is that something is not OK on the way bonds behave when bridged on de Linux/Debian/Kernel side (and it seems to be an old problem from what i have readed by googling around) although by the way Linux network stack seems to work, the problem passes unnoticed for Linux VMs, while it is a no-go for Win7 VMs.
Probably the combination of CISCO LACP + bridge + WinVM is not much popular.

Best regards.

 

[alexolivan]

Got it working.

It turned out to be the configuration for bond0/LACP on /etc/network/interfaces.
The config for ifenslave has changed a lot, so most of the documentation online is outdated, ant the mine was not perfect.
On the other hand (and that's what lead me to error) the configuration ended up with a bond0 interface up and running on the hypervisor, while the channel group went down on the CISCO shortly after boot up.
Curiously, then somehow connection between hypervisor and switch worked in a strange state, the 3560 treated each link as an independent one (the interfaces are all up while the channel group is down), while the PROXMOX used the bond0 interface as a bond.
Even more misleading was that Linux guests where quite robust while Windows ones barely worked.

So, once a recent configuration in /etc/network/interfaces is in place, the LACP handshake works, the channel goes up and keeps up on the switch, and it all works.

 

[Elfy]

Hi there! Reviving this thread because I'm having strange issues with ARP, LACP and my Cisco switch. The symptoms are Corosync fails occasionally on startup given certain routing conditions. Strangely enough, Corosync starts when I use the bonded network, but fails when I use the unbonded network:

Spoiler: Corosync config for the cluster

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: Host 1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.0.3
  }
  node {
    name: Host 2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.0.5
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: Cluster
  config_version: 3
  interface {
    bindnetaddr: 192.168.0.3
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

Note that this is for my home lab, I have a simple router, and I am not very knowledgeable with networking/subnetting. I'm going to post my network config for both nodes and for the switch, perhaps something will stick out to someone who has done this before, and they could give me some guidance.

Spoiler: Host #1 /etc/network/interfaces

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp30s0
iface enp30s0 inet manual
#MOBO

auto enp31s0f0
iface enp31s0f0 inet manual
#NIC1

auto enp31s0f1
iface enp31s0f1 inet manual
#NIC2

auto bond0
iface bond0 inet manual
    bond-slaves enp31s0f0 enp31s0f1
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer2+3
#lagg

auto vmbr0
iface vmbr0 inet static
    address  192.168.0.3
    netmask  255.255.255.0
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
#Storage Network

auto vmbr1
iface vmbr1 inet static
    address  192.168.0.2
    netmask  255.255.255.0
    gateway  192.168.0.1
    bridge-ports enp30s0
    bridge-stp off
    bridge-fd 0
#Internet

Spoiler: Host #2 /etc/network/interfaces

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet manual
#MOBO

auto enp1s0f0
iface enp1s0f0 inet manual
#NIC1

auto enp1s0f1
iface enp1s0f1 inet manual
#NIC2

auto bond0
iface bond0 inet manual
    bond-slaves enp1s0f0 enp1s0f1
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer2+3
#lagg

auto vmbr0
iface vmbr0 inet static
    address  192.168.0.5
    netmask  255.255.255.0
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
#Storage Network

auto vmbr1
iface vmbr1 inet static
    address  192.168.0.4
    netmask  255.255.255.0
    gateway  192.168.0.1
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
#Internet

Spoiler: LAGG config on the switch. This is the same for all hosts.

I have verified with omping that my switch is properly set up for multicast and IGMP Snooping has been turned on. Additionally, the switch has LACP assigned to the correct LAGGs and ports (I've triple-checked). So I'm pretty confident the issue is with some setting on the Cisco switch, or with my PVE network config. I'm not using VLAN's or anything. I'm grateful for anyone taking a look at this.

-Matt

 

[Stoiko Ivanov]

please open a new thread instead of replying to a solved one.
* have you tried removing `bond-xmit-hash-policy layer2+3` from the bond-stanza?
* do you see anything related to lacp/bond in the dmesg output of the hosts?

 

[Elfy]

I haven't tried removing the "bond-xmit-hash-policy layer2+3" stanza, will give that a try when I have access to the server tonight.
I am seeing on my NAS's syslog, I get lots of:

arp: 192.168.0.3 moved from 30:9c:23:03:ce:a7 to 00:26:55:e2:c0:e4 on lagg0
 arp: 192.168.0.5 moved from 4c:cc:6a:25:36:55 to 00:15:17:95:f4:b0 on lagg0
 arp: 192.168.0.3 moved from 30:9c:23:03:ce:a7 to 00:26:55:e2:c0:e4 on lagg0
 arp: 192.168.0.5 moved from 00:15:17:95:f4:b0 to 4c:cc:6a:25:36:55 on lagg0
 arp: 192.168.0.3 moved from 30:9c:23:03:ce:a7 to 00:26:55:e2:c0:e4 on lagg0
 arp: 192.168.0.5 moved from 4c:cc:6a:25:36:55 to 00:15:17:95:f4:b0 on lagg0
 arp: 192.168.0.3 moved from 30:9c:23:03:ce:a7 to 00:26:55:e2:c0:e4 on lagg0
 arp: 192.168.0.5 moved from 00:15:17:95:f4:b0 to 4c:cc:6a:25:36:55 on lagg0
 arp: 192.168.0.3 moved from 30:9c:23:03:ce:a7 to 00:26:55:e2:c0:e4 on lagg0
 arp: 192.168.0.5 moved from 4c:cc:6a:25:36:55 to 00:15:17:95:f4:b0 on lagg0
 ...

On the host when I search for "bond":

Oct 15 02:22:30 Orion kernel: [   10.788171] bond0: link status definitely up for interface enp31s0f0, 1000 Mbps full duplex
Oct 15 02:22:30 Orion kernel: [   10.788176] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
Oct 15 02:22:30 Orion kernel: [   10.788184] bond0: first active interface up!
Oct 15 02:22:30 Orion kernel: [   10.788196] vmbr0: port 1(bond0) entered blocking state
Oct 15 02:22:30 Orion kernel: [   10.788198] vmbr0: port 1(bond0) entered forwarding state
Oct 15 02:22:30 Orion kernel: [   10.788299] IPv6: ADDRCONF(NETDEV_CHANGE): vmbr0: link becomes ready
Oct 15 02:22:30 Orion kernel: [   10.796986] ip6_tables: (C) 2000-2006 Netfilter Core Team
Oct 15 02:22:30 Orion kernel: [   10.883082] ip_set: protocol 6
Oct 15 02:22:30 Orion kernel: [   10.988984] e1000e: enp31s0f1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Oct 15 02:22:30 Orion kernel: [   10.996138] bond0: link status definitely up for interface enp31s0f1, 1000 Mbps full duplex

And later...

Oct 15 02:35:26 Orion kernel: [    7.692797] IPv6: ADDRCONF(NETDEV_UP): enp31s0f0: link is not ready
Oct 15 02:35:26 Orion kernel: [    7.968802] IPv6: ADDRCONF(NETDEV_UP): enp31s0f1: link is not ready
Oct 15 02:35:26 Orion systemd-udevd[1173]: Could not generate persistent MAC address for bond0: No such file or directory
Oct 15 02:35:26 Orion kernel: [    7.989866] Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Oct 15 02:35:26 Orion kernel: [    8.173821] e1000e: enp31s0f0 NIC Link is Down
Oct 15 02:35:26 Orion systemd-udevd[1172]: Could not generate persistent MAC address for vmbr0: No such file or directory
Oct 15 02:35:27 Orion kernel: [    8.432698] bond0: Enslaving enp31s0f0 as a backup interface with a down link
Oct 15 02:35:27 Orion kernel: [    8.433041] vmbr0: port 1(bond0) entered blocking state
Oct 15 02:35:27 Orion kernel: [    8.433042] vmbr0: port 1(bond0) entered disabled state
Oct 15 02:35:27 Orion kernel: [    8.433124] device bond0 entered promiscuous mode
Oct 15 02:35:27 Orion kernel: [    8.433125] device enp31s0f0 entered promiscuous mode
Oct 15 02:35:27 Orion kernel: [    8.609708] e1000e: enp31s0f1 NIC Link is Down
Oct 15 02:35:27 Orion kernel: [    8.864380] device enp31s0f1 entered promiscuous mode
Oct 15 02:35:27 Orion kernel: [    8.864440] bond0: Enslaving enp31s0f1 as a backup interface with a down link
Oct 15 02:35:27 Orion kernel: [    8.865285] IPv6: ADDRCONF(NETDEV_UP): vmbr0: link is not ready
Oct 15 02:35:27 Orion ifup[941]: ifup: waiting for lock on /run/network/ifstate.vmbr0
Oct 15 02:35:27 Orion systemd-udevd[1172]: Could not generate persistent MAC address for vmbr1: No such file or directory
Oct 15 02:35:27 Orion kernel: [    8.992532] vmbr1: port 1(enp30s0) entered blocking state
Oct 15 02:35:27 Orion kernel: [    8.992533] vmbr1: port 1(enp30s0) entered disabled state
Oct 15 02:35:27 Orion kernel: [    8.992649] device enp30s0 entered promiscuous mode
Oct 15 02:35:27 Orion kernel: [    8.995818] IPv6: ADDRCONF(NETDEV_UP): vmbr1: link is not ready