Proxmox claiming MAC address

hm...same here:
ps ax | grep rpc
417 ? I< 0:00 [rpciod]
can't seem to be killed nor does rebooting prevent this from reappearing.
So i think, this can be ignored, seems to be some workqueue in the kernel.
Nevertheless, i am still confident, disabling and stopping all rpc services should help with the MAC issue at hetzner.
This is the first time I've heard that it's not enough for someone.
To this day, we really do have some peace and quiet from these Hetzner e-mails on all our systems.
So if this is now not working anymore, either you have missed something or the issue has mutated somehow.
In the latter case, we should soon receive the same warnings.
cheers
Sascha
 
update:
Today we got this message on one of our systems again.
there's definitely no rpc service running on it anymore...
So here we are, stuck again... after so many months thinking we finally have found the cause...
Any ideas anyone?
Thanks
Sascha
 
@tafkaz

I faced this issue a couple of months before again and it was because I was using multiple ipv6 addresses using the same bridge as main ipv6 has been assigned to.

After assigning every ipv6 to a specific bridge, issue got resolved.
 
@Amin Vakil
this actually sounds quite promising.
However does this mean, i would have to create a separate vmbr-device for every guest and the host?
Maybe it would be enough to create a separate vmbr device for the host only?

Or do we even need separate hardware eth devices on the server maybe?

thanks
Sascha
 
I've pasted my ipv4 configuration of a server in this thread already, you can find details there.

But for IPv6:

I have put my main server IPv4 and IPv6 in vmbr0

But for every other IPv6 which I wanted to assign to virtual machines, I've created another vmbr and assign an IP to it, then put virtual machine ipv6 in its subnet.
 
Offtopic here, but I have not seen any official linux manual (not some random tutorial) ever which has advised to disable ipv6 entirely if you don't use it.

If you don't use multiple IPv6(s) on the same bridge, I don't think that's the problem.
 
Gents,

Got the same problem as you. The dreaded "Unauth MAC" from Hetzner. I'm running two IPs/MACs on my proxmox and I actually managed to catch to offending packets yesterday by running:

tcpdump -Q out -n -w /tmp/ether2.pcap -e not ether host 00:50:56:15:14:E5 and not ether host D4:3D:7E:DA:F0:03 &

So basically I'm looking for packets (outbound) where my two allowed addresses (00:50:56:15:14:E5 and D4:3D:7E:DA:F0:03) are not present. After running it for 4+ hours I got hit for the following packets:

No. Time Source Destination Protocol Length Info
1 0.000000 78.46.47.218 106.75.74.225 TCP 54 43 → 58914 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Frame 1: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: ASUSTekC_88:94:18 (ac:22:0b:88:94:18), Dst: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb)
Internet Protocol Version 4, Src: 78.46.47.218, Dst: 106.75.74.225
Transmission Control Protocol, Src Port: 43, Dst Port: 58914, Seq: 1, Ack: 1, Len: 0

No. Time Source Destination Protocol Length Info
2 1670.006896 78.46.47.219 106.75.74.225 TCP 54 43 → 58914 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Frame 2: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: ASUSTekC_54:c4:be (c8:60:00:54:c4:be), Dst: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb)
Internet Protocol Version 4, Src: 78.46.47.219, Dst: 106.75.74.225
Transmission Control Protocol, Src Port: 43, Dst Port: 58914, Seq: 1, Ack: 1, Len: 0

No. Time Source Destination Protocol Length Info
3 13932.020792 78.46.47.220 222.175.199.226 TCP 54 43 → 22789 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Frame 3: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: ASUSTekC_54:b7:3d (c8:60:00:54:b7:3d), Dst: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb)
Internet Protocol Version 4, Src: 78.46.47.220, Dst: 222.175.199.226
Transmission Control Protocol, Src Port: 43, Dst Port: 22789, Seq: 1, Ack: 1, Len: 0

No. Time Source Destination Protocol Length Info
4 20714.477417 78.46.47.201 222.175.199.226 TCP 54 43 → 22789 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Frame 4: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: ASUSTekC_af:f0:a8 (10:7b:44:af:f0:a8), Dst: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb)
Internet Protocol Version 4, Src: 78.46.47.201, Dst: 222.175.199.226
Transmission Control Protocol, Src Port: 43, Dst Port: 22789, Seq: 1, Ack: 1, Len: 0

What is interesting is that these packets are indeed coming from illegal MAC addresses (none of the IPs or MAC are mine). It seems to be some kind of spoof on whois (port 43) and I'm still not sure how I get these (I'm working on that) as I started with only looking at packets I sent out (thus offending Heztner).

Would be interesting though if you guys could see if you are showing the same thing. As said I had to let it run for 4+ hours in order to catch this so it happens very seldom.
 
another thread related I think:
https://forum.proxmox.com/threads/mac-address-abuse-report.95656/


So, hertzner is flooding crap traffic, seem that they have some kind of problem with their physical network. (maybe it's BUM traffic when they don't have the mac in their switches, but I think they was filtering mac by port? (or maybe only ingress traffic, and not egress ? )
 
spirit said:
another thread related I think:
https://forum.proxmox.com/threads/mac-address-abuse-report.95656/


So, hertzner is flooding crap traffic, seem that they have some kind of problem with their physical network. (maybe it's BUM traffic when they don't have the mac in their switches, but I think they was filtering mac by port? (or maybe only ingress traffic, and not egress ? )
Click to expand...
I totally agree that this looks more and more like a flooding problem. And since we have a software switch (aka bridge) when entering into our Proxmox servers, it (to me) looks like some solution to not have our software switch flood unknown unicast (received from the Heztner network) everywhere.
 
alxgarder said:
I totally agree that this looks more and more like a flooding problem. And since we have a software switch (aka bridge) when entering into our Proxmox servers, it (to me) looks like some solution to not have our software switch flood unknown unicast (received from the Heztner network) everywhere.
Click to expand...
yes, the problem is that if you receive an unknown mac, the vmbrX will try to flood it again to all ports, but I'm not sure it'll send back to physical interfaces from where it have receive it.

you can still try to enable rp_filter

sysctl -w net.ipv4.conf.vmbr0.rp_filter=1
sysctl -w net.ipv4.conf.ethX.rp_filter=1

, but I'm not sure it's work with layer2.
 
  • Like
Reactions: openaspace
spirit said:
yes, the problem is that if you receive an unknown mac, the vmbrX will try to flood it again to all ports, but I'm not sure it'll send back to physical interfaces from where it have receive it.

you can still try to enable rp_filter

sysctl -w net.ipv4.conf.vmbr0.rp_filter=1
sysctl -w net.ipv4.conf.ethX.rp_filter=1

, but I'm not sure it's work with layer2.
Click to expand...
To my knowledge rp_filter is IP reverse path check (layer3 checking). Since this is switch-to-switch we are looking for some sort of layer2 (aka MAC) filtering.
 
alxgarder said:
To my knowledge rp_filter is IP reverse path check (layer3 checking). Since this is switch-to-switch we are looking for some sort of layer2 (aka MAC) filtering.
Click to expand...
https://hechao.li/2018/09/11/Linux-Bridge-Port-Flags/

BR_HAIRPIN_MODE​

This flag controls whether traffic may be sent back out of the port on which it was received. By default, this flag is turned off and the bridge will not forward traffic back out of the receiving port. [2]



So, by default, il should not forward back a wrong frame coming from herztner.
 
cat /sys/class/net/eno1/brport/hairpin_mode = 0 -> seem to be ok.

but maybe :

cat /sys/class/net/eno1/brport/hairpin_mode/unicast_flood = 1 (not sure if it's break network if changed to 0)


with ifupdown2, it can be changed with

Code: 
auto eno1
iface eno1 inet manual
    bridge-unicast-flood off
    bridge-multicast-flood off

auto vmbr0
iface vmbr0 inet manual
    bridge-ports eno1
    ....
 
Last edited:
spirit said:
cat /sys/class/net/eno1/brport/hairpin_mode = 0 -> seem to be ok.

but maybe :

cat /sys/class/net/eno1/brport/hairpin_mode/unicast_flood = 1 (not sure if it's break network if changed to 0)


with ifupdown2, it can be changed with

Code: 
auto eno1
iface eno1 inet manual
    bridge-unicast-flood off
    bridge-multicast-flood off

auto vmbr0
iface vmbr0 inet manual
    bridge-ports eno1
    ....
Click to expand...

So I looked a lot at all these filters and was really unsure what they did (and more worried how they actually would behave) so I took a little different approach in order to troubleshoot.
ebtables does layer2 rules, so I started with just seeing if I could log every time the server was sending out offending frames. I ended with the following:

ebtables -I OUTPUT -o enp2s0 -p IPv4 --among-src ! d4:3d:7e:da:f0:03 --log-level info --log-prefix "MAC-FLOOD-O" --log-ip -j CONTINUE
ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j CONTINUE

As explained I have two MAC addresses. d4:3d:7e:da:f0:03 which is the BIA of the main NIC and 00:50:56:15:14:e5 which is paired with my secondary IP.
This behaves the same as the tcpdump I did earlier - basically saying: log if the source mac is not any of my mac addresses. That caught the next wave of frames (5 to be exact). All of them came on the FORWARD chain.

Therefore, I have now changed the FORWARD chain to drop:

ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j DROP

.. and I'm running the tcpdump also to see if this stops the packets.

Until tomorrow..
 
alxgarder said:
So I looked a lot at all these filters and was really unsure what they did (and more worried how they actually would behave) so I took a little different approach in order to troubleshoot.
ebtables does layer2 rules, so I started with just seeing if I could log every time the server was sending out offending frames. I ended with the following:

ebtables -I OUTPUT -o enp2s0 -p IPv4 --among-src ! d4:3d:7e:da:f0:03 --log-level info --log-prefix "MAC-FLOOD-O" --log-ip -j CONTINUE
ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j CONTINUE

As explained I have two MAC addresses. d4:3d:7e:da:f0:03 which is the BIA of the main NIC and 00:50:56:15:14:e5 which is paired with my secondary IP.
This behaves the same as the tcpdump I did earlier - basically saying: log if the source mac is not any of my mac addresses. That caught the next wave of frames (5 to be exact). All of them came on the FORWARD chain.

Therefore, I have now changed the FORWARD chain to drop:

ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j DROP

.. and I'm running the tcpdump also to see if this stops the packets.

Until tomorrow..
Click to expand...

So, I did indeed get some packets during the night and they were successfully dropped by the filter above in ebtables. I could verify this as I ran tcpdump at the same time and didnt record a single packet going out (to wrong mac addresses that is).

Last question I have: How do I incorporate this ebtables filter line into my Proxmox installation so it is always first line in the FORWARD chain, doesn't interfere with the updates Proxmox is doing to ebtables and is consistence over reloads?
 
alxgarder said:
So, I did indeed get some packets during the night and they were successfully dropped by the filter above in ebtables. I could verify this as I ran tcpdump at the same time and didnt record a single packet going out (to wrong mac addresses that is).
Click to expand...
I'm curious to knwn, is it a packet coming from outside, and proxmox reforward it again to outside ?

alxgarder said:
Last question I have: How do I incorporate this ebtables filter line into my Proxmox installation so it is always first line in the FORWARD chain, doesn't interfere with the updates Proxmox is doing to ebtables and is consistence over reloads?
Click to expand...
you could add them in /etc/network/interfaces , with a "post-up ebtables ..." on enp2s0 iface.

proxmox firewall don't drop custom iptables/ebtables rules.
 
spirit said:
I'm curious to knwn, is it a packet coming from outside, and proxmox reforward it again to outside ?


you could add them in /etc/network/interfaces , with a "post-up ebtables ..." on enp2s0 iface.

proxmox firewall don't drop custom iptables/ebtables rules.
Click to expand...

Thanks spirit. I did exactly that :)
Are you good with the solution or do I need to explain it better?
 
spirit said:
i just don't understand from where theses packets (going out your server) are coming from
Click to expand...

Okay. I will try to explain .. :)

A switch works under a couple of main rules:
  • Whenever a packet comes in:
    • Rule1: Record the source mac and the interface it came from in a table
    • Rule2: Lookup the destination mac in the table created in rule 1:
      • Rule2A: If the destination mac is found in table, send the packet out that interface
      • Rule2B: if the destination mac is NOT found in table, flood the packet to all interfaces but the interface the packet came from

What I am seeing on my server is that get ALOT of packets not destined for me like:

No. Time Source Destination Protocol Length Info
1 0.000000 162.62.9.237 78.47.101.73 TCP 60 47741 → 43 [SYN] Seq=0 Win=65535 Len=0

Frame 1: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Ethernet II, Src: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb), Dst: IntelCor_0c:5f:c3 (00:1b:21:0c:5f:c3)
Internet Protocol Version 4, Src: 162.62.9.237, Dst: 78.47.101.73
Transmission Control Protocol, Src Port: 47741, Dst Port: 43, Seq: 0, Len: 0

This packet is from 162.62.9.237 (not me) to 78.47.101.73 (not me either). As I don't have access to the switch I'm connected to I can only assume why I'm getting this packet which is Rule2B - flood the packet. 78.47.101.73 has properly been silent for a while (or not online) meaning the switch doesn't know out of which interface its mac address is located.
In a normal server setup this packet would have been dropped coming into my servers network card as it is neither my mac address (as destination) and its not multicast/broadcast.
The problem here is that my server is also a switch (linux bridge) which means I will allow all packets coming in and filter them through my firewalls (ebtables/iptables). In iptables I have a clear Proxmox rule saying: If the packet destination port is TCP/43 (WHOIS) then reject it.
REJECT is an active reply (contrary to DROP) meaning I would send the following packets:

No. Time Source Destination Protocol Length Info
2 0.000039 78.47.101.73 162.62.9.237 TCP 54 43 → 47741 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Frame 2: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: IntelCor_0c:5f:c3 (00:1b:21:0c:5f:c3), Dst: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb)
Internet Protocol Version 4, Src: 78.47.101.73, Dst: 162.62.9.237
Transmission Control Protocol, Src Port: 43, Dst Port: 47741, Seq: 1, Ack: 1, Len: 0

This is were it all breaks. I'm now sending out a packet that I didn't "own". This conflicts with Heztner because of switch Rule1. Their switch will now store that the source mac (IntelCor_0c:5f:c3 (00:1b:21:0c:5f:c3)) that I sent out is now located down the interface towards my server, thereby blackholing traffic to the "real" server connected somewhere else.

So, there are ways to combat this, I choose the dropping of outbound packets not from me. You could possibly also locate all rules in iptables and make sure they either a) drops silently (not sending an active reply) and/or b) only reacts if the packet includes your servers IP addresses inbound.

Hope this makes sense. :)
 
Last edited:
  • Like
Reactions: Kyle, EuroDomenii and doymer
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back