ARP reply from multiple PVE Hosts

A

AWoelfel

Member
Feb 17, 2021
3
0
6
41
We currently have two PROXMOX clusters with the same behavior.

The first one is our production cluster and consists of 9 Hosts (each with PVE 7.2-4/ca9d43cc kernel: 5.15.35-2-pve)
The second one is our two Host dev environment with 2 Hosts (currently running PVE 7.4-3/9002ab8a kernel: 5.15.74-1-pve)

The information posted here is from the dev environment.
For simplicity purposes i replaced the MACs with ones easier to read.


Bash: 
#/etc/network/interfaces
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address 172.16.0.3/25
        gateway 172.16.0.1
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
#HYPERVISOR

auto eno2
iface eno2 inet static
        address 172.16.0.129/25
#COROSYNC

auto eno3
iface eno3 inet static
        address 172.16.1.129/25
#CEPH

iface eno4 inet manual

iface idrac inet manual

auto vmbr1
iface vmbr1 inet static
        address 172.16.1.1/25
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#CONTROLPLANE
auto vmbr2
iface vmbr2 inet static
        address 172.16.2.1/23
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#WORKER

Bash: 
# ip route
default via 172.16.0.1 dev eno1 proto kernel onlink
172.16.0.0/25 dev eno1 proto kernel scope link src 172.16.0.3
172.16.0.128/25 dev eno2 proto kernel scope link src 172.16.0.129
172.16.1.0/25 dev vmbr1 proto kernel scope link src 172.16.1.1
172.16.1.128/25 dev eno3 proto kernel scope link src 172.16.1.129
172.16.2.0/23 dev vmbr2 proto kernel scope link src 172.16.2.1

Bash: 
# ip neigh
172.16.2.13 dev vmbr2 lladdr 0a:49:cd:88:40:8f STALE
172.16.2.16 dev vmbr2  FAILED
172.16.1.130 dev eno3 lladdr 80:18:44:e1:37:f6 REACHABLE
172.16.2.21 dev vmbr2  FAILED
172.16.1.11 dev vmbr1 lladdr f2:39:3e:a3:fe:d0 STALE
172.16.2.24 dev vmbr2  FAILED
172.16.0.130 dev eno2 lladdr 80:18:44:e1:37:f5 REACHABLE
172.16.2.15 dev vmbr2  FAILED
172.16.2.23 dev vmbr2  FAILED
172.16.0.4 dev eno1 lladdr 80:18:44:e1:37:f4 STALE
172.16.2.26 dev vmbr2  FAILED
172.16.2.12 dev vmbr2 lladdr be:09:80:51:f8:89 STALE
172.16.0.1 dev eno1 lladdr 40:a8:f0:12:86:0a REACHABLE
172.16.2.14 dev vmbr2 lladdr be:95:f7:35:ed:a6 STALE
172.16.2.22 dev vmbr2  FAILED
172.16.1.12 dev vmbr1  FAILED
172.16.2.25 dev vmbr2  FAILED
172.16.2.11 dev vmbr2 lladdr 8e:07:37:a6:43:f1 STALE

VMS on freezer
  • 101 master-1 172.16.1.11/25 (CONTROLPLANE)
  • 111 worker-11 172.16.2.11/23 (WORKER)
  • 112 worker-12 172.16.2.12/23 (WORKER)
  • 113 worker-13 172.16.2.13/23 (WORKER)
  • 114 worker-14 172.16.2.14/23 (WORKER)



Bash: 
#/etc/network/interfaces
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address 172.16.0.4/25
        gateway 172.16.0.1
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
#HYPERVISOR

auto eno2
iface eno2 inet static
        address 172.16.0.130/25
#COROSYNC

auto eno3
iface eno3 inet static
        address 172.16.1.130/25
#CEPH

iface eno4 inet manual

auto vmbr1
iface vmbr1 inet static
        address 172.16.1.2/25
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#CONTROLPLANE

auto vmbr2
iface vmbr2 inet static
        address 172.16.2.2/23
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#WORKER

Bash: 
# ip route
default via 172.16.0.1 dev eno1 proto kernel onlink
172.16.0.0/25 dev eno1 proto kernel scope link src 172.16.0.4
172.16.0.128/25 dev eno2 proto kernel scope link src 172.16.0.130
172.16.1.0/25 dev vmbr1 proto kernel scope link src 172.16.1.2
172.16.1.128/25 dev eno3 proto kernel scope link src 172.16.1.130
172.16.2.0/23 dev vmbr2 proto kernel scope link src 172.16.2.2

Bash: 
# ip neigh
172.16.2.24 dev vmbr2 lladdr ee:a0:fd:24:2c:3c STALE
172.16.2.21 dev vmbr2 lladdr ee:5f:1b:0d:57:fb STALE
172.16.2.14 dev vmbr2  FAILED
172.16.1.11 dev vmbr1  FAILED
172.16.2.25 dev vmbr2  FAILED
172.16.2.15 dev vmbr2  FAILED
172.16.2.12 dev vmbr2  FAILED
172.16.2.16 dev vmbr2  FAILED
172.16.0.1 dev eno1 lladdr 40:a8:f0:12:86:0a DELAY
172.16.2.13 dev vmbr2  FAILED
172.16.0.3 dev eno1 lladdr 14:18:77:2e:59:9d STALE
172.16.2.11 dev vmbr2  FAILED
172.16.0.129 dev eno2 lladdr 14:18:77:2e:59:9e REACHABLE
172.16.1.129 dev eno3 lladdr 14:18:77:2e:59:9f REACHABLE
172.16.2.22 dev vmbr2 lladdr 16:b5:a2:58:85:f7 STALE
172.16.1.12 dev vmbr1 lladdr 9a:cd:0d:dd:65:1e STALE
172.16.2.26 dev vmbr2  FAILED
172.16.2.23 dev vmbr2 lladdr 4e:cd:a6:5b:c9:8f STALE

VMS on trunks
  • 102 master-2 172.16.1.12/25 (CONTROLPLANE)
  • 121 worker-21 172.16.2.21/23 (WORKER)
  • 122 worker-22 172.16.2.22/23 (WORKER)
  • 123 worker-23 172.16.2.23/23 (WORKER)
  • 124 worker-24 172.16.2.24/23 (WORKER)

Here is deal:

We printed the arp table of our "top of rack" switch, which is the first hop after the PVE Nodes.

Code: 
# show arp
IP ARP table

  IP Address       MAC Address       Type    Port
  ---------------  ----------------- ------- ----
  10.10.0.1        TOPOF-RACK_HP     dynamic 1/48
  172.16.0.3       FREEZER--ENO1     dynamic 2/13
  172.16.0.4       TRUNKS---ENO1     dynamic 2/13
  172.16.1.11      TRUNKS---ENO1     dynamic 2/13
  172.16.1.12      TRUNKS---ENO1     dynamic 2/13
  172.16.2.11      TRUNKS---ENO1     dynamic 2/13
  172.16.2.12      FREEZER--ENO1     dynamic 2/13
  172.16.2.13      FREEZER--ENO1     dynamic 2/13
  172.16.2.14      FREEZER--ENO1     dynamic 2/13
  172.16.2.21      TRUNKS---ENO1     dynamic 2/13
  172.16.2.22      TRUNKS---ENO1     dynamic 2/13
  172.16.2.23      TRUNKS---ENO1     dynamic 2/13
  172.16.2.24      FREEZER--ENO1     dynamic 2/13


This arp table shows 172.16.2.11 to be contacted via ENO1 of trunks but the VM worker-11 is hosted on freezer.

We already went down the rabit hole and found multiple arp replies from trunks and freezer to the arp request for 172.16.2.11.

From any host attached to the "top of rack" switch we are not able to ping worker-11 because trunks does not know where to send the traffic.
On the other hand we are able to ping worker-12 because its next hop after the "top of rack" switch is correctly send to freezer.

We found this two posts and think they are related to this problem:
- https://forum.proxmox.com/threads/wrong-mac-addresses-in-arp-table.91178/
- https://forum.proxmox.com/threads/invalid-arp-responses-cause-network-problems.118128/

and tried multiple versions (arp_ignore 1 & 2; arp_announce 1 & 2; etc...) of following configurations but with no effect
Code: 
#systemd
net.ipv4.conf.all.arp_ignore = 2
net.ipv4.conf.default.arp_ignore = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_notify = 1
 
Did you ever solve this?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back