2 subnets 2 nics, but subnets are flapping to both nics (ARP moved from xxx to xxx)

bazzi · Nov 20, 2019

Ok I am doing something wrong but I don't see it.

The situation:
1x 10GBe uplink to internal network (same VLAN as the 1GBe internal network)
1x 1GBe uplink to internal network (same VLAN as the 10GBe internal network)
1x 1GBe uplink to external network (diffrent VLAN's)

What am i tring to do:
10GBe for dedicated storage/migration/cluster
1GBe uplink internal for management
1 GBe uplink for external network

I thought:
vmbr0 -> 10GBe 10.221.183.92 no gateway
vmbr1 -> 1GBe 10.221.184.92 gateway 10.221.184.1
vmbr9 -> external so no ip for this bridge

But I have now ARP flapping issues:
arp: 10.221.184.92 moved from 80:30:e0:0d:2c:c0 to 48:df:37:ae:78:a0 on lagg0
arp: 10.221.184.92 moved from 48:df:37:ae:78:a0 to 80:30:e0:0d:2c:c0 on lagg0

the MAC 80:30:e0:0d:2c:c0 is vmbr2
the MAC 48:df:37:ae:78:a0 is vmbr1

What am I doing wrong here?

Code:

auto lo
iface lo inet loopback

iface eno5 inet manual

iface ens1f0 inet manual

iface ens1f1 inet manual

iface eno6 inet manual

iface eno7 inet manual

iface eno8 inet manual

auto vmbr1
iface vmbr1 inet static
        address  10.221.184.92
        netmask  24
        gateway  10.221.184.1
        bridge-ports eno5
        bridge-stp off
        bridge-fd 0
#INTERN

auto vmbr0
iface vmbr0 inet static
        address  10.221.183.92
        netmask  24
        bridge-ports ens1f0
        bridge-stp off
        bridge-fd 0
#INTERN 10G

auto vmbr9
iface vmbr9 inet manual
        bridge-ports eno8
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#EXTERN

aaron · Dec 17, 2019

Please post the output of ip address if you are still having this issue.

bazzi · Dec 17, 2019

Hi Aaron,

Unfortinaly still.

Bash:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether 48:df:37:ae:76:c8 brd ff:ff:ff:ff:ff:ff
3: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 80:30:e0:0d:2a:38 brd ff:ff:ff:ff:ff:ff
4: eno6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 48:df:37:ae:76:c9 brd ff:ff:ff:ff:ff:ff
5: ens1f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 80:30:e0:0d:2a:39 brd ff:ff:ff:ff:ff:ff
6: eno7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 48:df:37:ae:76:ca brd ff:ff:ff:ff:ff:ff
7: eno8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr9 state UP group default qlen 1000
    link/ether 48:df:37:ae:76:cb brd ff:ff:ff:ff:ff:ff
8: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 48:df:37:ae:76:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.221.184.91/24 brd 10.221.184.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::4adf:37ff:feae:76c8/64 scope link
       valid_lft forever preferred_lft forever
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 80:30:e0:0d:2a:38 brd ff:ff:ff:ff:ff:ff
    inet 10.221.183.91/24 brd 10.221.183.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::8230:e0ff:fe0d:2a38/64 scope link
       valid_lft forever preferred_lft forever
10: vmbr9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 48:df:37:ae:76:cb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4adf:37ff:feae:76cb/64 scope link
       valid_lft forever preferred_lft forever
11: tap520i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr9 state UNKNOWN group default qlen 1000
    link/ether 6a:42:3f:11:b5:51 brd ff:ff:ff:ff:ff:ff
12: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr101i0 state UNKNOWN group default qlen 1000
    link/ether 06:2c:e4:25:6b:a6 brd ff:ff:ff:ff:ff:ff
13: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b6:a0:31:d0:39:34 brd ff:ff:ff:ff:ff:ff
14: fwpr101p0@fwln101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr9 state UP group default qlen 1000
    link/ether 82:88:61:70:2e:c8 brd ff:ff:ff:ff:ff:ff
15: fwln101i0@fwpr101p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP group default qlen 1000
    link/ether b6:a0:31:d0:39:34 brd ff:ff:ff:ff:ff:ff
16: tap105i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr105i0 state UNKNOWN group default qlen 1000
    link/ether 5a:2a:bd:0c:2c:e8 brd ff:ff:ff:ff:ff:ff
17: fwbr105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ea:cb:f6:29:61:19 brd ff:ff:ff:ff:ff:ff
18: fwpr105p0@fwln105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr9 state UP group default qlen 1000
    link/ether 26:b8:f8:f2:35:31 brd ff:ff:ff:ff:ff:ff
19: fwln105i0@fwpr105p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP group default qlen 1000
    link/ether ea:cb:f6:29:61:19 brd ff:ff:ff:ff:ff:ff
21: tap521i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr9 state UNKNOWN group default qlen 1000
    link/ether 8a:cc:eb:fc:fa:9f brd ff:ff:ff:ff:ff:ff
22: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr102i0 state UNKNOWN group default qlen 1000
    link/ether 6e:42:b1:61:8e:9b brd ff:ff:ff:ff:ff:ff
23: fwbr102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:e6:29:ef:65:a1 brd ff:ff:ff:ff:ff:ff
24: fwpr102p0@fwln102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether be:d6:c5:02:1f:57 brd ff:ff:ff:ff:ff:ff
25: fwln102i0@fwpr102p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UP group default qlen 1000
    link/ether 0e:e6:29:ef:65:a1 brd ff:ff:ff:ff:ff:ff

aaron · Dec 18, 2019

Are the flapping MAC addresses still the same? They are not present in the ip a output. :/

bazzi · Dec 18, 2019

ahhh shoot. I am sending you the wrong ip address. This is the correct one:

Code:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether 48:df:37:ae:78:a0 brd ff:ff:ff:ff:ff:ff
3: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 80:30:e0:0d:2c:c0 brd ff:ff:ff:ff:ff:ff
4: eno6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 48:df:37:ae:78:a1 brd ff:ff:ff:ff:ff:ff
5: ens1f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 80:30:e0:0d:2c:c1 brd ff:ff:ff:ff:ff:ff
6: eno7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 48:df:37:ae:78:a2 brd ff:ff:ff:ff:ff:ff
7: eno8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr9 state UP group default qlen 1000
    link/ether 48:df:37:ae:78:a3 brd ff:ff:ff:ff:ff:ff
8: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 48:df:37:ae:78:a0 brd ff:ff:ff:ff:ff:ff
    inet 10.221.184.90/24 brd 10.221.184.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::4adf:37ff:feae:78a0/64 scope link
       valid_lft forever preferred_lft forever
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 80:30:e0:0d:2c:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.221.183.90/24 brd 10.221.183.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::8230:e0ff:fe0d:2cc0/64 scope link
       valid_lft forever preferred_lft forever
10: vmbr9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 48:df:37:ae:78:a3 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4adf:37ff:feae:78a3/64 scope link
       valid_lft forever preferred_lft forever
14: tap510i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr9 state UNKNOWN group default qlen 1000
    link/ether 22:84:94:5b:6e:d5 brd ff:ff:ff:ff:ff:ff
15: tap510i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether fa:c2:04:c6:7b:f2 brd ff:ff:ff:ff:ff:ff
16: tap510i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether aa:1b:dd:f2:3e:9c brd ff:ff:ff:ff:ff:ff
17: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 06:08:ea:27:8c:96 brd ff:ff:ff:ff:ff:ff
18: tap104i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr9 state UNKNOWN group default qlen 1000
    link/ether 2a:c2:69:f4:d4:31 brd ff:ff:ff:ff:ff:ff
20: tap505i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 7e:30:36:1b:44:a3 brd ff:ff:ff:ff:ff:ff
21: tap502i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether d2:32:2f:b4:67:8b brd ff:ff:ff:ff:ff:ff

aaron · Dec 19, 2019

Hmm, that is peculiar.

Did you change IP addresses in the meantime?
With the old ones:

IP 10.1.2.92 is configured on vmbr1.
vmbr1 and its bridge port eno5 have the MAC 48:df:37:ae:78:a0
vmbr0 and its bridge port ens1f0 have the MAC 80:30:e0:0d:2c:c0

The 10.1.2.92 IP should only be available on vmbr1 :/

Where do you observe the flapping?

bazzi · Dec 19, 2019

on a backup storage server with both ip's 10.221.183.35 and 10.221.184.35 in that VLAN.

No I din't change any IP's. But for the first post a collegue gived me bogus internal ip's. We was afraid for information on the web, but it are only private ip's....

I fix the first post.

aaron · Dec 19, 2019

Okay, most likely the kernel is answering ARP requests on the wrong interface.
You can check if the sysctl option is set to 0 with sysctl net.ipv4.conf.all.arp_filter.

If it is, set it to 1 and see if that solves the problem of the flapping ARP tables.

You can make it permanent by writing the following into /etc/sysctl.d/fix_arp_flapping.conf

Code:

net.ipv4.conf.all.arp_filter = 1

After a reboot the parameter should be set to 1 automatically.

For some background see [0], search for arp_filter.

[0] https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

bazzi · Dec 19, 2019

Hi Aaron,

I have added it and activate it.I will keep you posted!

Thanks,

Tim

shabsta · Sep 8, 2021

bazzi said:
Hi Aaron,

I have added it and activate it.I will keep you posted!

Thanks,

Tim

Any updates Tim?

Search

Search

2 subnets 2 nics, but subnets are flapping to both nics (ARP moved from xxx to xxx)

bazzi

Active Member

aaron

Proxmox Staff Member

bazzi

Active Member

aaron

Proxmox Staff Member

bazzi

Active Member

aaron

Proxmox Staff Member

bazzi

Active Member

aaron

Proxmox Staff Member

bazzi

Active Member

shabsta

Member