ProxMox to Cisco Switch and Cisco FirePower 1010 Firewall Duplicate IP Address

[rajivbaxi]

Hello all,

I'm having an issue with ProxMox and Cisco gear. The ProxMox server has a four-port ethernet NIC in an Open vSwitch LACP bond (I previously tried the built-in Linux interface config as well) to a Cisco 3560 Switch which then uplinks to a Cisco FirePower 1010 firewall. I have 5 VMs configured with static IPs that work with no issue (VLANs 4, 8, 9, 11, and 15). I have tried to introduce one more VLAN, VLAN 10 and I keep getting a duplicate IP address issue on the guest VMs (Windows and Linux). When I perform a packet capture from one of the guest machines with the issue, I see the DHCP DORA process, the client gets an IP address, sends ARP requests for the gateway and the IP that it has, then the Cisco Firepower 1010 responds that it has the same IP address, and then the IP address is released. The whole process keeps cycling through all of the IP addresses in the scope. This seems to be an issue with Duplicate Address Detection (DAD) on the client-side and gratuitous ARP, only on this VLAN. If I switch the VLAN to one of the working VLANs I can get an IP address just fine. Also, If I connect a device to an access port on the switch, in the same VLAN (VLAN 10), I can get an IP address and can access the Internet with no problem.

Here's the Cisco Switch config for the trunk ports and the port-channel
I'm running IOS version 12.2(44) SE6

interface FastEthernet0/18
description Trunk to ProxMox1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2-6,8-19,99
switchport mode trunk
switchport nonegotiate
speed 100
duplex full
channel-group 1 mode active
end

interface FastEthernet0/19
description Trunk to ProxMox1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2-6,8-19,99
switchport mode trunk
switchport nonegotiate
speed 100
duplex full
channel-group 1 mode active
end

interface FastEthernet0/20
description Trunk to ProxMox1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2-6,8-19,99
switchport mode trunk
switchport nonegotiate
speed 100
duplex full
channel-group 1 mode active
end

interface FastEthernet0/21
description Trunk to ProxMox1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2-6,8-19,99
switchport mode trunk
switchport nonegotiate
speed 100
duplex full
channel-group 1 mode active
end

interface Port-channel1
description Etherchannel to ProxMox1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2-6,8-19,99
switchport mode trunk
switchport nonegotiate
end

Here is the pertinent information from /etc/network/interfaces from the Proxmox server:
(enp2s0 is the management NIC)
Proxmox version is 7.1-10
The kernel version is 5.13.19-6 PVE #1 SMP PVE 5.13.19-14

auto enp2s0
iface enp2s0 inet manual
#Management - Lab - VLAN 15

auto enp1s0f0
iface enp1s0f0 inet manual
#Port 18 Trunk

auto enp1s0f1
iface enp1s0f1 inet manual
#Port 19 Trunk

auto enp1s0f2
iface enp1s0f2 inet manual
#Port 20 Trunk

auto enp1s0f3
iface enp1s0f3 inet manual
#Port 21 Trunk

auto vlan2
iface vlan2 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=2

auto vlan3
iface vlan3 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=3

auto vlan4
iface vlan4 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=4

auto vlan5
iface vlan5 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=5

auto vlan6
iface vlan6 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=6

auto vlan7
iface vlan7 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=7

auto vlan8
iface vlan8 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=8

auto vlan9
iface vlan9 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=9

auto vlan10
iface vlan10 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=10

auto vlan11
iface vlan11 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=11

auto vlan12
iface vlan12 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=12

auto vlan13
iface vlan13 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=13

auto vlan14
iface vlan14 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=14

auto vlan15
iface vlan15 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=15

auto vlan16
iface vlan16 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=16

auto vlan17
iface vlan17 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=17

auto vlan18
iface vlan18 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=18

auto vlan19
iface vlan19 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=19

auto vlan99
iface vlan99 inet manual
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_options tag=99

auto bond1
iface bond1 inet manual
ovs_bonds enp1s0f0 enp1s0f1 enp1s0f2 enp1s0f3
ovs_type OVSBond
ovs_bridge vmbr2
ovs_options lacp=active bond_mode=balance-slb other_config:lacp-time=fast

auto vmbr0
iface vmbr0 inet static
address 10.15.0.251/24
gateway 10.15.0.1
bridge-ports enp2s0
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet manual
ovs_type OVSBridge
ovs_ports bond1 vlan2 vlan3 vlan4 vlan5 vlan6 vlan7 vlan8 vlan9 vlan10 vlan11 vlan12 vlan13 vlan14 vlan15 vlan16 vlan17 vlan18 vlan19 vlan99

The Cisco Firepower 1010 is running software 6.6.1-91
I have configured a trunk interface on one of the ports, have created the same matching VLANs as the switch, and have added the VLANs on the firewall to the interface connected to the switch.

Any help would be appreciated...

 

[rajivbaxi]

Here's an update... I disabled LACP on ProxMox and the switch. I configured balance-rr on the bond interface for ProxMox and I still have the same issue. I have reverted the changes back to LACP.

 

[rajivbaxi]

Here's another update. I implemented Open vSwitch and still have the same issue. I did some further digging though. The issue only occurs on a single VLAN (VLAN 10). Both Windows and Linux detect a Duplicate Address coming from the Cisco Firepower 1010. However, if I disable Duplicate Address Detection (DAD) on the Windows PC, it gets and keeps an IP address and I can browse the Internet without an issue.

The reg key is HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ArpRetryCount (DWORD) and set to 0 (a reboot is necessary for the changes to take effect).

I have attempted to run the configuration command "ip arp gratuitous none" on the switch but that didn't work either. In fact, the L3 switch ignores all gratuitous ARP requests (as the command is as designed).

 

[rajivbaxi]

When the registry entry for ArpRetryCount is not present, the follow message shows on the switch: "IP ARP: ignored gratuitous arp src 0.0.0.0 521f.d9a8.f842, dst 10.10.0.121 001c.5827.f9c8, interface Vlan10"

Here's a post from https://community.cisco.com/t5/switching/strange-arp-behavior-which-hits-the-cpu-much/td-p/2351237:
I have seen this before on our network where we had servers with teamed broadcom NICs sending out thousands of ARPs to the switch SVI and ramping the processor right up.

In the end, updating the NIC drivers resolved the issue.

We couldn't even control it as it would be random servers that would do it at random times of the day. Because the ARPs were unicast aswell we could not use any kind of rate limitation or apply any filtering.

Here's a snippet from lspci on the ProxMox server:
$ lspci -nnk
01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter [14e4:1904]
Kernel driver in use: tg3
Kernel modules: tg3
01:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter [14e4:1904]
Kernel driver in use: tg3
Kernel modules: tg3
01:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter [14e4:1904]
Kernel driver in use: tg3
Kernel modules: tg3
01:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
Subsystem: Broadcom Inc. and subsidiaries 4-port 1Gb Ethernet Adapter [14e4:1904]
Kernel driver in use: tg3
Kernel modules: tg3

Here's ethtool from the ProxMox server:
$ sudo ethtool -i enp1s0f0
driver: tg3
version: 5.13.19-6-pve
firmware-version: FFV21.40.9 bc 5719-v1.45
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

I tried to compile the latest Broadcom driver. However, I can't get it to compile as the installer couldn't find the right kernel source package to install. I then tried to update ProxMox and I believe it updated the firmware, manager, and possibly the kernel.

I have a quad-port Intel NIC that I can test with or a newer L3 switch (3750) next.

 

[rajivbaxi]

I had the same issue with the quad-port Intel NIC and when replacing the switch with a 3750 running IOS 15.2.4(E9).

I think I figured this out. It was the ArpProxy on the Firepower 1010. I disabled ArpProxy on the Firepower 1010 with the sysopt noproxy <interface name> command, removed the registry entry for ArpRetryCount, rebooted the Windows 10 machine, and the issue no longer occurs.