Proxmox claiming MAC address

tmsg · Mar 19, 2019

I have a 3 node cluster setup that is claiming mac addresses used by other devices on the network. I have one central switch, E1 and several switches, A##, connected to that one. One proxmox host is connected to A10 and two are connected to A11. The ports connected to the proxmox servers (only one at a time) show mac address belonging to other devices on the network assigned to this port and E1 will start routing traffic to that switch. I have OVS setup on the proxmox boxes.

Each proxmox box has
- eth0 (vmbr1) connected to switch for local network only
- eth1 (vmbr2) connected to switch for ceph network (ceph runs on different servers)
- eth2 (vmbr0) connected to public A## switch for Internet traffic

There is nothing common about the the mac addresses that are being claimed. They may be connected to different switches running different OSs with different hardware and on different vlans. The switch tends to happen to lightly used devices, but not always. I can prevent this from happening by putting static mac address entries in E1 or by having a device run a continuous ping of E1. MAC addresses will be claimed randomly and it may get fixed on its own after some random period of time. The mac addresses are not duplicated on the proxmox host or vms.

I see that OVS is creating interfaces like fwln1115o0. It seems that this interface is getting the mac address from somewhere. How are these created and why?

Why would the Proxmox host be announcing a mac address that is not on the server?

Code:

# pveversion --verbose
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.4.134-1-pve: 4.4.134-112
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.6-1-pve: 4.4.6-48
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

On the network core switch:

Code:

E1#sho arp |  inc 0017.c5ac.97e5
Internet  X.X.35.40         154   0017.c5ac.97e5  ARPA   Vlan11 pv 1100
Internet  X.X.35.41          22   0017.c5ac.97e5  ARPA   Vlan11 pv 1100
Internet  X.X.35.42         129   0017.c5ac.97e5  ARPA   Vlan11 pv 1100

## When it is working correctly, I get this
E1#sho mac add add 0017.c5ac.97e5
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
11    0017.c5ac.97e5    DYNAMIC pv  Po22
1100    0017.c5ac.97e5    BLOCKED     Po22

## When it is failing, i get this
E1#sho mac add add 0017.c5ac.97e5
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
11    0017.c5ac.97e5    DYNAMIC pv  Po10


## Connection to A10
interface Port-channel10
 switchport mode trunk
 switchport nonegotiate
end

On the switch that this devices is on:

Code:

A22#sho mac add add 0017.c5ac.97e5
Unicast Entries
 vlan   mac address     type        protocols               port
-------+---------------+--------+---------------------+--------------------
11     0017.c5ac.97e5   dynamic ip                    GigabitEthernet1/34

On the switch that Proxmox is on:

Code:

A10#sho mac add int gi1/4
Unicast Entries
 vlan   mac address     type        protocols               port
-------+---------------+--------+---------------------+--------------------
...
11    0017.c5ac.97e5   dynamic ip                    GigabitEthernet1/4
...

Interface GigabitEthernet1/4
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 3,11,60,1100
 switchport mode trunk
 switchport nonegotiate
 spanning-tree cost 20
end

#connection to E1
interface Port-channel1
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport nonegotiate
end

On Proxmox host:

Code:

proxmox1:~# ovs-appctl fdb/show vmbr0 | grep e5
   13    11  00:17:c5:ac:97:e5  215

proxmox1:~# ovs-ofctl dump-ports-desc vmbr0
OFPST_PORT_DESC reply (xid=0x2):
 1(eth2): addr:6c:b3:11:31:b1:75
     config:     0
     state:      0
     current:    1GB-FD COPPER AUTO_NEG
     advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
     supported:  10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
     speed: 1000 Mbps now, 1000 Mbps max
 2(pxhost): addr:6c:b3:11:31:b1:75
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(fwln107o0): addr:56:d5:75:3a:b1:0e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(tap1002i0): addr:e6:f2:b4:4b:53:d5
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 5(tap1002i2): addr:56:b2:49:2c:c8:55
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 6(fwln1003o0): addr:76:08:42:13:1f:46
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 7(fwln1003o2): addr:5a:fa:10:d1:ba:64
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 8(fwln1113o1): addr:62:68:cc:b8:e7:73
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 9(tap1000i0): addr:76:fb:f1:c5:b7:1e
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 10(tap1000i2): addr:f6:f5:c6:e2:73:a1
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 11(fwln1007o0): addr:d6:df:0c:54:1a:7c
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 12(fwln1116o0): addr:4a:f9:a9:45:82:6d
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 13(fwln1115o0): addr:1e:ef:d2:63:41:e9
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(vmbr0): addr:6c:b3:11:31:b1:75
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max

Network config

Code:

# cat /etc/network/interfaces

auto lo
iface lo inet loopback

# Bridges
auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
  ovs_type OVSBridge
  ovs_ports eth2 pxhost

auto vmbr1
allow-ovs vmbr1
iface vmbr1 inet manual
  ovs_type OVSBridge
  ovs_ports eth0 llan llan2
  pre-up ( ifconfig eth0 mtu 8192)
  mtu 8192

auto vmbr2
allow-ovs vmbr2
iface vmbr2 inet manual
  ovs_type OVSBridge
  ovs_ports eth1 dlan
  pre-up ( ifconfig eth1 mtu 8192)
  mtu 8192

# Physical Ports
allow-vmbr1 eth0
iface eth0 inet manual
   ovs_type OVSPort
   ovs_bridge vmbr1
##   post-up echo 1> /proc/sys/net/ipv4/conf/eth2/proxy_arp

allow-vmbr2 eth1
iface eth1 inet manual
   ovs_type OVSPort
   ovs_bridge vmbr2

allow-vmbr0 eth2
iface eth2 inet manual
   ovs_type OVSPort
   ovs_bridge vmbr0


# Setup network for host
# Public
allow-vmbr0 pxhost
iface pxhost inet static
   ovs_type OVSIntPort
   ovs_bridge vmbr0
   ovs_options tag=11
   hwaddress ether 6c:b3:11:31:b1:75
   address  X.X.34.3
   netmask  255.255.248.0
   gateway  X.X.32.1

# LAN
allow-vmbr1 llan
iface llan inet static
   ovs_type OVSIntPort
   ovs_bridge vmbr1
   address  192.168.0.3
   netmask  255.255.255.0
   mtu 8192

allow-vmbr1 llan2
iface llan2 inet static
   ovs_type OVSIntPort
   ovs_bridge vmbr1
   ovs_options tag=15
   address 192.168.88.103
   netmask 255.255.248.0

# Ceph Network
allow-vmbr2 dlan
iface dlan inet static
   ovs_type OVSIntPort
   ovs_bridge vmbr2
   address  10.11.0.3
   netmask  255.255.254.0
   mtu 8192
##   post-up iptables -t nat -A POSTROUTING -s 10.11.0.0/23 -o vmbr0 -j MASQUERADE
##   post-down iptables -t nat -D POSTROUTING -s 10.11.0.0/23 -o vmbr0 -j MASQUERADE

tmsg · Mar 25, 2019

Any advice on how to troubleshoot this?

How does fwln1115o0 get created? Where? Why?

Stoiko Ivanov · Mar 26, 2019

the fwln - interfaces get created when you use the pve-firewall for a guest('s interface) - there the filtering happens.
if the firewall is not activated for a guest interface the tap interfaces are directly plugged into the bridge.

the described problem sounds like it could be related to proxy_arp being enabled on your node (check with `sysctl -a |grep proxy_arp`)

hope this helps!

Amin Vakil · Jul 22, 2020

I have this problem and Hetzner has sent me an email regarding this matter which they're going to block me in a week if I don't fix the issue

I have checked proxy_arp via sysctl -a | grep proxy_arp and they're disabled on all interfaces.

Code:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp6s0
iface enp6s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address i.p.a.d/27
    gateway g.a.t.e
    bridge-ports enp6s0
    bridge-stp off
    bridge-fd 0
    pointopoint g.a.t.e
#ipv6 subnet

iface vmbr0 inet6 static
    address i:p:a:d/64
    gateway fe80::1

auto vmbr1
iface vmbr1 inet static
    address 1.1.1.207/27
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#ipv4 subnet

auto vmbr2
iface vmbr2 inet static
    address 192.168.10.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '192.168.10.0/24' -o vmbr0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.10.0/24' -o vmbr0 -j MASQUERADE
    post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
#Local

Although I admit I have done something unusual on my network, I have been assigned a /28 subnet (1.1.1.192/28) from hetzner.

I should've choose main server ip address as gateway in vms, but as I wanted to use cloud-init and it can't have a gateway outside of its subnet, I defined 1.1.1.207/27 (broadcast of /28 subnet) on main server and now I set the vms ip like this:
ip: 1.1.1.206/27
gw: 1.1.1.207

This way there is no need to define a static route to main server ip.
The only downside is I can't access 1.1.1.1.208/28 which I assumed it's fine and that would be very bad luck if a website I want to work with is in that subnet

I don't know if this has to do anything regarding this matter.

Also tcpdump with one of unknown mac addresses output:

Code:

root@hostname ~ # tcpdump ether host 90:1b:0e:e1:82:c1 -n -vv
18:59:10.575113 IP (tos 0x0, ttl 248, id 64169, offset 0, flags [none], proto TCP (6), length 40)
    185.176.27.46.62000 > i.p.a.d.44964: Flags [S], cksum 0x7c32 (correct), seq 92084270, win 1024, length 0

Unknown mac address are not present in ip link.
All vms' firewall are enabled and mac filter option is enabled too.

Amin Vakil · Aug 5, 2020

Unfortunately Hetzner blocked my ip today, any idea how to resolve this issue?

tafkaz · Aug 10, 2020

Hi Amin,
we have pretty much the same issue i think, but we also don't have a solution.
We thought about using the macfiltering for the Proxmox Host too, as it works for the VMs allready, but until now, we haven't yet accomplished.
Have a look:
https://forum.proxmox.com/threads/how-to-disable-mac-address-change-in-vm.51022/

cheers
Sascha

Amin Vakil · Aug 10, 2020

I was waiting for see if the solution works or not, but for now I've disabled and stopped rpcbind and hetzner people told me that the issue is gone for now, and I haven't received an email from then since.

It seems that there are bugs in rpc bind which creates this issue (Hetzner support told me to check this, please share if you've find something relevant).

All these needs to be executed otherwise rpcbind service will gets started again.

Code:

systemctl disable --now rpcbind.socket
systemctl disable --now rpcbind.target
systemctl disable --now rpcbind.service

tafkaz · Aug 10, 2020

Hi,
well, what you describe would fix the port 111/Portmapper open messages from Hetzner, we know how to solve that allready.
But would it fix the unallowed MAC adresses warning we get from Hetzner also?
Don't believe so, would you?
Or am i missing something here?

cheers
Sascha

Amin Vakil · Aug 10, 2020

I think what they meant wasn't because of port 111 being opened from outside.

It seems that rpcbind service just being started would cause the issues.

Although I'm saying I couldn't find any article regarding this issue, but after stopping rpcbind they said they're not seeing any extra MAC address from my server.

tafkaz · Aug 10, 2020

ah, ok...interesting.
So

systemctl disable --now rpcbind.socket
systemctl disable --now rpcbind.target
systemctl disable --now rpcbind.service

on the Proxmox host solved it for you, you're saying?

Amin Vakil · Aug 10, 2020

Actually I'm not sure, As I've mentioned I was waiting to see if hetzner sends me their scary abuse emails again or not, it's been 5 days and they haven't yet.

Also they told me that the issue has got fixed after I've stopped the service, so that would be up to you whether to stop it or not.

Which gets me to another question, what is the use of rpcbind on a proxmox server?

tafkaz · Aug 10, 2020

this is really weird...the host i got hetzners mac adress message for has a firewall configured correctly, so the port is explicitly blocked:

So this would mean, as long as rpcbind is active, it does not matter, if the port is blocked for incoming traffic, it will still send outgoing packets with the wrong mac adress effect...

I will disable the service, and see how it goes.

Rpcbind is used for NFS (servers) if NFS is left to their defaults.
However it seems, that NFS can also work without...

Whatever...disable and see

Arvyr · Aug 10, 2020

Pls help me, where is the MAC 0017.c5ac.97e5 bound to?

I only see it multiple times on the Core-Switch with different IPs and only a single time on the switch that's connected to your node?

Looks more like Hetzner fucked up their Spanning-Tree. Not sure however.

tafkaz · Aug 11, 2020

@Amin Vakil
seems to resolve the issue...really...who would have thought....a solution, after all these years...

for everybody out there getting the same (forbidden MAC adresses) messages from Hetzner:

systemctl disable rpcbind.service rpcbind.socket rpcbind.target
systemctl stop rpcbind.service rpcbind.socket rpcbind.target

on the Proxmox Host.

cheers
Sascha

tafkaz · Nov 4, 2020

hey everyone.
after such a long time without any issues, i now get that mail from Hetzner again.
on the Proxmox Host however rpcbind is completely disabled already, all i can see is:

systemctl | grep -i rpc
run-rpc_pipefs.mount loaded active mounted RPC Pipe File System

disable that one too? What would you think?

Thanks
Sascha

Emil Makariev · Aug 13, 2021

Hello everyone! I had the same issue with Hetzner and I can confirm what @Stoiko Ivanov said:

the fwln - interfaces get created when you use the pve-firewall for a guest('s interface) - there the filtering happens.

When I use pve-firewall for my guest machines, I see an extra MAC in "bridge fdb show". Once you disable the firewall, the problem disappears and your IP is unblocked.

tafkaz · Aug 13, 2021

hi @Emil Makariev
just some update from me:
Back then i did disable the remaining services the command
systemctl | grep -i rpc
showed, and never had any issues with this again.
I make use of the integrated FW quite a lot.
HTH
Sascha

Emil Makariev · Sep 5, 2021

Hi @tafkaz, thanks for your advce! I tried disabling:

Bash:

systemctl disable rpcbind.service rpcbind.socket rpcbind.target run-rpc_pipefs.mount
systemctl stop rpcbind.service rpcbind.socket rpcbind.target run-rpc_pipefs.mount

I've contacted Hetzner support, right after disabling the rpcbind and they confirm there are no additional MACs's. It worked fine till yesterday, when I got the notification mail again

I haven't changed anything on my networking. Only server updates.

tafkaz · Sep 5, 2021

Hi,
are you sure you have disabled every single service systemctl | grep -i rpc shows you?
These messages are currently going out again with high frequency at Hetzner...
Our solution seems to work here still...

But actually i have been desperately trying to find the root cause over the last 2 or 3 years, but neither Proxmox nor Hetzner people have ever had any real explanation for this behavior.
So i was quite happy to see, that we never got any "Hetzner unauthorized MAC address use" messages again, since we found the solution i have posted.
However i am not completely sure, this will resolve the issue 100%.

cheers
Sascha

Emil Makariev · Sep 5, 2021

Here is the output:

Bash:

root@NodeH1:~# systemctl -all|grep -i rpc
  run-rpc_pipefs.mount                                                                         loaded    inactive   dead      RPC Pipe File System
  auth-rpcgss-module.service                                                                   loaded    inactive   dead      Kernel Module supporting RPCSEC_GSS
  rpc-gssd.service                                                                             loaded    inactive   dead      RPC security service for NFS client and server
  rpc-statd-notify.service                                                                     loaded    inactive   dead      Notify NFS peers of a restart
  rpc-svcgssd.service

However, I see also:

Bash:

root@NodeH1:~# ps -ax|grep -i rpc
   1600 ?        I<     0:00 [rpciod]

Proxmox claiming MAC address

Renowned Member

Renowned Member

Proxmox Staff Member

Active Member

Active Member

Renowned Member

Active Member

Renowned Member

Active Member

Renowned Member

Active Member

Renowned Member

Active Member

Renowned Member

Renowned Member

Member

Renowned Member

Member

Renowned Member

Member