Proxmox generate 2 mac address visibile on the switch not allowed by the data center

openaspace

Member
Sep 16, 2019
311
6
23
Italy
Hello.
After server reboot I received a MAC abuse notification from Hetzner.
Looking the networks device, now I have found this 2 mac in 2fwbr.
I have on vmbr0 opnesense with vmbr1 and vmbr2 as private LAN.
Others vps and xlc are directly connected on the vmbr0 with separate public ip's and dedicated MAC address relased from hetzner and someone at the same time on the vmbr2 private LAN

Why this happen?!?

> Unallowed MACs:
> 3a:3e:f1:e2:5f:2b - (code line 8 and 10)
> ea:9b:09:ea:b3:4f - (code line 12 and 14)


px3-Proxmox-Virtual-Environment.png

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000 link/ether d4:3d:7e:d8:bd:c5 brd ff:ff:ff:ff:ff:ff 3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d4:3d:7e:d8:bd:c5 brd ff:ff:ff:ff:ff:ff inet xxxx.xx.xx.35 peer xxx.xx.xx.33/32 brd xxx.xx.xxx.63 scope global vmbr0 valid_lft forever preferred_lft forever inet6 fe80::d63d:7eff:fed8:bdc5/64 scope link valid_lft forever preferred_lft forever 4: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether aa:20:df:e2:84:9c brd ff:ff:ff:ff:ff:ff inet 192.168.30.0/24 brd 192.168.30.255 scope global vmbr1 valid_lft forever preferred_lft forever inet6 fe80::78e5:9eff:fee4:5fa2/64 scope link valid_lft forever preferred_lft forever 5: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether b2:00:40:0b:c1:b6 brd ff:ff:ff:ff:ff:ff inet 192.168.40.0/24 brd 192.168.40.255 scope global vmbr2 valid_lft forever preferred_lft forever inet6 fe80::b000:40ff:fe0b:c1b6/64 scope link valid_lft forever preferred_lft forever 6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/none inet xxx.xx.xx.74/32 scope global tailscale0 valid_lft forever preferred_lft forever inet6 fd7a:115c:a1e0:ab12:4843:cd96:624c:194a/128 scope global valid_lft forever preferred_lft forever inet6 fe80::32c0:fb93:8c30:7cf9/64 scope link stable-privacy valid_lft forever preferred_lft forever 7: veth114i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr114i0 state UP group default qlen 1000 link/ether fe:cd:09:f0:2b:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 8: fwbr114i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether [B]3a:3e:f1:e2:5f:2b[/B] brd ff:ff:ff:ff:ff:ff 9: fwpr114p0@fwln114i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000 link/ether b6:8c:4d:12:53:8c brd ff:ff:ff:ff:ff:ff 10: fwln114i0@fwpr114p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr114i0 state UP group default qlen 1000 link/ether [B]3a:3e:f1:e2:5f:2b[/B] brd ff:ff:ff:ff:ff:ff 11: tap200i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master fwbr200i0 state UNKNOWN group default qlen 1000 link/ether c2:e8:08:25:fb:d1 brd ff:ff:ff:ff:ff:ff 12: fwbr200i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether [B]ea:9b:09:ea:b3:4f[/B] brd ff:ff:ff:ff:ff:ff 13: fwpr200p0@fwln200i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000 link/ether ea:68:35:62:f1:5f brd ff:ff:ff:ff:ff:ff 14: fwln200i0@fwpr200p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr200i0 state UP group default qlen 1000 link/ether [B]ea:9b:09:ea:b3:4f [/B]brd ff:ff:ff:ff:ff:ff 20: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master fwbr100i0 state UNKNOWN group default qlen 1000 link/ether aa:b0:74:ab:f4:e4 brd ff:ff:ff:ff:ff:ff 21: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether c2:e1:f1:3f:b4:ce brd ff:ff:ff:ff:ff:ff 22: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000 link/ether 76:38:69:53:7f:51 brd ff:ff:ff:ff:ff:ff 23: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000 link/ether c2:e1:f1:3f:b4:ce brd ff:ff:ff:ff:ff:ff 24: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UNKNOWN group default qlen 1000 link/ether aa:20:df:e2:84:9c brd ff:ff:ff:ff:ff:ff
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,042
459
103
www.odiso.com
Hi, see : https://forum.proxmox.com/threads/proxmox-claiming-mac-address.52601/

TLDR :

hertzner send crap to your box, with wrong destination mac/ip.
if you use proxmox firewall with "reject" rules, a RST packet is send with iptables with this wrong mac/ip as src.

The previous thread have some workaround with ebtables rules to block outgoing mac.

fast fix : use drop rules

if you use reject rules, I would like to test:

"sysctl -w net.netfilter.nf_conntrack_tcp_loose = 0"


another way, allow manually only all yours mac address with:

ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5,d4:3d:7e:da:f0:03 -j DROP

(where enp2s0 is your physical interface)
 

openaspace

Member
Sep 16, 2019
311
6
23
Italy
ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5,d4:3d:7e:da:f0:03 -j
Hi really thank you. I will try now.
Are this rules permanent among proxmox upgrade?
Could be nice to manage from the browser control panel like iptables
 

spirit

Famous Member
Apr 2, 2010
5,042
459
103
www.odiso.com
Hi really thank you. I will try now.
Are this rules permanent among proxmox upgrade?
Could be nice to manage from the browser control panel like iptables
I'm currently looking to add an option in pve-firewall to do it automaticaly.
but you can edit /etc/network/interfaces,

with something like

Code:
auto enp2s0
iface enp2s0 ...
     ....
     post-up ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5,d4:3d:7e:da:f0:03 -j
 

Undergrid

Member
Jul 15, 2018
13
0
6
46
I'm currently looking to add an option in pve-firewall to do it automaticaly.
but you can edit /etc/network/interfaces,

with something like

Code:
auto enp2s0
iface enp2s0 ...
     ....
     post-up ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5,d4:3d:7e:da:f0:03 -j

If I was bridging enp2s0 (or in my case enp0s31f6) with vmbr0, would the post-up rule need be done on the interface, bridge or for the host and each VM? And would just ommiting the "-p IPv4" make this work for dual IPv4/IPv6 network stacks too?

For reference, my network config is:

Code:
auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet static
        address  95.216.xx.yy
        netmask  255.255.255.192
        gateway  95.216.xx.zz
        bridge-ports enp0s31f6
        bridge-stp off
        bridge-fd 0

iface vmbr0 inet6 static
        address  2a01:4f9:xx:yyy::2
        netmask  64
        gateway  fe80::1

auto vmbr1
iface vmbr1 inet static
        address  10.1.1.1
        netmask  255.255.0.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge_maxwait 0
 

openaspace

Member
Sep 16, 2019
311
6
23
Italy
auto enp2s0 iface enp2s0 ... .... post-up ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5,d4:3d:7e:da:f0:03 -j
nothing...
I have rebooted the server... for test.. and new network mac address abuse received from hetzner....

tcpdump for these mac address don't report any connection in/out
I need to add the ebtables rules also for each virtual device?

ebtables -L Bridge table: filter Bridge chain: INPUT, entries: 0, policy: ACCEPT Bridge chain: FORWARD, entries: 1, policy: ACCEPT -j PVEFW-FORWARD Bridge chain: OUTPUT, entries: 0, policy: ACCEPT Bridge chain: PVEFW-FORWARD, entries: 3, policy: ACCEPT -p IPv4 -j ACCEPT -p IPv6 -j ACCEPT -o fwln+ -j PVEFW-FWBR-OUT Bridge chain: PVEFW-FWBR-OUT, entries: 4, policy: ACCEPT -i tap100i0 -j tap100i0-OUT -i tap200i0 -j tap200i0-OUT -i veth110i0 -j veth110i0-OUT -i veth114i0 -j veth114i0-OUT Bridge chain: tap100i0-OUT, entries: 2, policy: ACCEPT -s ! 0:50:56:0:84:12 -j DROP -j ACCEPT Bridge chain: tap200i0-OUT, entries: 2, policy: ACCEPT -s ! 0:50:56:0:b8:95 -j DROP -j ACCEPT Bridge chain: veth110i0-OUT, entries: 2, policy: ACCEPT -s ! 0:50:56:1:2:62 -j DROP -j ACCEPT Bridge chain: veth114i0-OUT, entries: 2, policy: ACCEPT -s ! 0:50:56:0:b8:8c -j DROP -j ACCEPT


auto lo iface lo inet loopback iface enp2s0 inet manual post-up ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! d4:3d:7e:d8:bd:c5,00:50:56:00:B1:05,00:50:56:00:84:12 -j
 
Last edited:

arhimede

New Member
Apr 2, 2021
11
1
3
61
I have the same situation today .

5 servers at Hetzner
all 5 running Proxmox 6.4 latest version
after today's Proxmox upgrade and reboot, I got the MAC abuse report from Hetner for all 5 machines.

Indeed , the result of
Code:
ifconfig -a
on one Proxmox server shows different MAC addresses then the original ones for all 3 virtual interfaces
fwln100i0
fwbr102i0
fwbr103i0

( i have 3 LXC containers on that machine )
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,042
459
103
www.odiso.com
I have the same situation today .

5 servers at Hetzner
all 5 running Proxmox 6.4 latest version
after today's Proxmox upgrade and reboot, I got the MAC abuse report from Hetner for all 5 machines.

Indeed , the result of
Code:
ifconfig -a
on one Proxmox server shows different MAC addresses then the original ones for all 3 virtual interfaces
fwln100i0
fwbr102i0
fwbr103i0

( i have 3 LXC containers on that machine )
fwlnX,fwbrX are internal firewall bridge, without any ip, so they shouldn't send traffic.
does the mac abuse report of hetner have the macs of theses interfaces ?
 

arhimede

New Member
Apr 2, 2021
11
1
3
61
fwlnX,fwbrX are internal firewall bridge, without any ip, so they shouldn't send traffic.
does the mac abuse report of hetner have the macs of theses interfaces ?
Yes, Hetzner abuse report have the "abusive" macs

and are the exact ones listed in fwbr102i0 and fwln102i0
 

arhimede

New Member
Apr 2, 2021
11
1
3
61
interesting, maybe it's another bug here. in your firewall rules (an default actions), do you use reject or drop ?
I have no rules in firewall, other then default ones

But in firewall logs i have something new, from this morning

0 5 - 13/Sep/2021:00:00:01 +0000 starting pvefw logger
0 5 - 13/Sep/2021:09:20:52 +0000 received terminate request (signal)
0 5 - 13/Sep/2021:09:20:52 +0000 stopping pvefw logger
0 5 - 13/Sep/2021:09:21:38 +0000 starting pvefw logger
 

openaspace

Member
Sep 16, 2019
311
6
23
Italy
Hetzner ?
Proxmox 6.4-13 ?
Virtual Environment 6.4-13 - Linux px3 5.4.140-1-pve #1 SMP PVE 5.4.140-1
Yes, Hetzner.

After the latest proxmox upgrade, one hour ago the support answered me that they see now only the allowed mac address.
I will try to restart the server newly now.. to verify if some mac address can communicate for some seconds at the server boot time first of the ebtables rules are applyed.
 

openaspace

Member
Sep 16, 2019
311
6
23
Italy
"NO LUCK".. to became crazy...!

Proxmox Stuff, please find a solutions for this big problem that is common to all user's running proxmox on Hetzner datacenter in bridged mode.

After a server restart new abuse message:

> Unallowed MACs:

> 02:41:a5:d2:f0:05
> 1a:be:49:df:3d:c5
> 82:6d:e7:0e:c1:03
> c2:0b:f3:8e:55:8b

From TCPDUMP no in/out traffic for the unallowed MACs after the server completely running.
-----------------------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
link/ether d4:3d:7e:d8:bd:c5 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d4:3d:7e:d8:bd:c5 brd ff:ff:ff:ff:ff:ff
inet xxx.xxx.xxx.xxx peer xx.xxx.xxx.xxx/32 brd xxx.xxx.xxx.xxx scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::d63d:7eff:fed8:bdc5/64 scope link
valid_lft forever preferred_lft forever
4: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 46:30:c3:9a:bc:ff brd ff:ff:ff:ff:ff:ff
inet 192.168.30.0/24 brd 192.168.30.255 scope global vmbr1
valid_lft forever preferred_lft forever
inet6 fe80::a808:c1ff:fe3b:ea4d/64 scope link
valid_lft forever preferred_lft forever
5: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 36:1f:84:67:92:9e brd ff:ff:ff:ff:ff:ff
inet 192.168.40.0/24 brd 192.168.40.255 scope global vmbr2
valid_lft forever preferred_lft forever
inet6 fe80::1c01:b9ff:fefa:2ace/64 scope link
valid_lft forever preferred_lft forever
6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 100.76.25.74/32 scope global tailscale0
valid_lft forever preferred_lft forever
inet6 fd7a:115c:a1e0:ab12:4843:cd96:624c:194a/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::cd90:7f8c:dce0:763/64 scope link stable-privacy
valid_lft forever preferred_lft forever
7: tap117i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UNKNOWN group default qlen 1000
link/ether 46:30:c3:9a:bc:ff brd ff:ff:ff:ff:ff:ff
8: tap117i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
link/ether 36:1f:84:67:92:9e brd ff:ff:ff:ff:ff:ff
9: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master fwbr100i0 state UNKNOWN group default qlen 1000
link/ether 2e:35:5d:bd:fc:51 brd ff:ff:ff:ff:ff:ff
10: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 82:6d:e7:0e:c1:03 brd ff:ff:ff:ff:ff:ff
11: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether d6:f7:07:49:4f:ce brd ff:ff:ff:ff:ff:ff
12: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
link/ether 82:6d:e7:0e:c1:03 brd ff:ff:ff:ff:ff:ff
13: veth110i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr110i0 state UP group default qlen 1000
link/ether fe:6c:4e:4a:6d:23 brd ff:ff:ff:ff:ff:ff link-netnsid 0
14: fwbr110i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether c2:0b:f3:8e:55:8b brd ff:ff:ff:ff:ff:ff
15: fwpr110p0@fwln110i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether 96:51:af:17:c4:c9 brd ff:ff:ff:ff:ff:ff
16: fwln110i0@fwpr110p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr110i0 state UP group default qlen 1000
link/ether c2:0b:f3:8e:55:8b brd ff:ff:ff:ff:ff:ff
17: veth110i1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr2 state UP group default qlen 1000
link/ether fe:df:62:4a:a2:5a brd ff:ff:ff:ff:ff:ff link-netnsid 0
18: veth114i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr114i0 state UP group default qlen 1000
link/ether fe:d6:8f:d7:31:25 brd ff:ff:ff:ff:ff:ff link-netnsid 1
19: fwbr114i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 1a:be:49:df:3d:c5 brd ff:ff:ff:ff:ff:ff
20: fwpr114p0@fwln114i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether b2:6e:35:51:e3:b6 brd ff:ff:ff:ff:ff:ff
21: fwln114i0@fwpr114p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr114i0 state UP group default qlen 1000
link/ether 1a:be:49:df:3d:c5 brd ff:ff:ff:ff:ff:ff
22: tap200i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master fwbr200i0 state UNKNOWN group default qlen 1000
link/ether 4a:15:08:d2:b2:e8 brd ff:ff:ff:ff:ff:ff
23: fwbr200i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:41:a5:d2:f0:05 brd ff:ff:ff:ff:ff:ff
24: fwpr200p0@fwln200i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
link/ether d6:cf:e3:8f:dd:bb brd ff:ff:ff:ff:ff:ff
25: fwln200i0@fwpr200p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr200i0 state UP group default qlen 1000
link/ether 02:41:a5:d2:f0:05 brd ff:ff:ff:ff:ff:ff
 
Last edited:
  • Like
Reactions: arhimede

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!