Proxmox claiming MAC address

[doymer]

You, guys, rock. You have saved my life and, what is more important, my hair.

I was just about trying the packet capture stage when I found this thread. Now it all drills down to Hetzner being the cuplrit!! And blaming their customers for a fault in their switches and banning them.

They should pay for your time because you have actually done what should have been their job.

Thanks for your time and for sharing with the community your findings. I'll be grateful to you for the centuries to come.

 

[alxgarder]

You, guys, rock. You have saved my life and, what is more important, my hair.

I was just about trying the packet capture stage when I found this thread. Now it all drills down to Hetzner being the cuplrit!! And blaming their customers for a fault in their switches and banning them.

They should pay for your time because you have actually done what should have been their job.

Thanks for your time and for sharing with the community your findings. I'll be grateful to you for the centuries to come.

Thank you for the kind words.

I will say that, after troubleshooting this, Im not sure you can fault Heztner here (though I might have hinted at that earlier). Actually, it works as designed. The problem is that the Proxmox config Im running (and properly you as well) turns your normal network card into a switch, allowing all packets in, which does require some rethinking on how return traffic should be handled.

The flooded frames happens all the time in a network and is how switches learn. There are two major reasons for floods. Either the server has been silent for a while (which will mean the switch will prune the mac/interface information from its mac table and therefore have to flood to find it again). The other reason is that the server simply doesnt exist. In that scenario all frames sent to the server will be flooded as the switch doesnt know where the server is (nowhere in this case).

 

[doymer]

Thank you for the kind words.

I will say that, after troubleshooting this, Im not sure you can fault Heztner here (though I might have hinted at that earlier). Actually, it works as designed. The problem is that the Proxmox config Im running (and properly you as well) turns your normal network card into a switch, allowing all packets in, which does require some rethinking on how return traffic should be handled.

The flooded frames happens all the time in a network and is how switches learn. There are two major reasons for floods. Either the server has been silent for a while (which will mean the switch will prune the mac/interface information from its mac table and therefore have to flood to find it again). The other reason is that the server simply doesnt exist. In that scenario all frames sent to the server will be flooded as the switch doesnt know where the server is (nowhere in this case).

Thanks for taking the time to reply. Not only kind words; I think that you pointed me to the right direction.

May be this is true in the case you expose where you are receiving whois packets and you (as me, as maybe all) have a REJECT rule in the firewall. But in my case I am receiving traffic to destination IPs (78.xx.xx.xx and others) that are *out of the scope* of my server addressing (176.xx.xx.xx/27) in port 80, 445 and many others. Network scanners? May be...

But this should not happen, I think. May be I am wrong but I think that the upper level routing system should had never injected that traffic into the switch I am connected to because my network segment is out of that scope by network mask (176.xx.xx.xx/27). That /27 imply that only packets destined to an IP in the 176.<something> range should reach my network segment. But I am receiving traffic destined to IP addresses 78.<something>, 5.<something>, etc. and I have no explanation for that. Does that mean that Hetzner makes no routing at all inside their network and all is a mess of traffic directed by ARP and MAC addressing? I expect not...

Anyway I still do not fully understand why I may be sending out packets with that MACs (my dump capture is still empty after several hours) but they say I do and me, as you, has nothing in the server that explain that behavior. So I cannot avoid it so far except perhaps using your ebtables solution blindly. I am constantly receiving traffic in the public interface that is not directed to any of my IPs nor MAC addresses like this one:

17:38:33.242902 84:c1:c1:76:a9:a2 > 00:50:56:00:39:9e, ethertype IPv4 (0x0800), length 60: 79.124.62.86.48386 > 5.9.9.140.24129: Flags , seq 806294571, win 1024, length 0

But the system, normally, does not reply to them. Finding the ones that force a reply is being harder in this case.

Best regards.


(edited) Latest addition: I started a new capture and, at last, I got packets (outgoing):

17:48:04.991198 90:1b:0e:93:25:30 > 84:c1:c1:76:a9:a2, ethertype IPv4 (0x0800), length 54: 5.9.170.34.43 > 46.17.102.78.46368: Flags [R.], seq 0, ack 3876508740, win 0, length 0
18:04:08.296780 00:50:56:00:9b:c8 > 84:c1:c1:76:a9:a2, ethertype IPv4 (0x0800), length 54: 5.9.170.36.43 > 46.17.102.78.46368: Flags [R.], seq 0, ack 1619227194, win 0, length 0
18:07:34.628512 00:50:56:00:9f:a3 > 84:c1:c1:76:a9:a2, ethertype IPv4 (0x0800), length 54: 5.9.170.37.43 > 46.17.102.78.46368: Flags [R.], seq 0, ack 3103376507, win 0, length 0

Looks like they are R[eset], of course from MACs not owned by me so Hetzner will, for sure, complaint. Now to find out why (and who) is sending this kind of packets only sometimes and not always or never.

More on this: I saw that I was sendig resets for some packets (5 now). Noticed that destination MACs were the same for all so I asked the server if it knew it.

arp -a: static.225.xx.xx.176.clients.your-server.de (176.xx.xxx.225) at 84:c1:c1:76:a9:a2 [ether] on vmbr0

(edit 2)The reply is that this MAC is the one for my network broadcast address (the one ending in 255) but, as it can be seen in the capture the destination IP is from outside of my network (46.17.102.78). Someone can help me clarify this mess?
I read wrongly the IP address. Not 255 but 225, the gateway adress and, of course, that is fully correct. All incoming trafic must come from the network gateway MAC address as it does. Sorry for the confusion.

 

[spirit]

Okay. I will try to explain ..

A switch works under a couple of main rules:

• Whenever a packet comes in:
  • Rule1: Record the source mac and the interface it came from in a table
  • Rule2: Lookup the destination mac in the table created in rule 1:
    • Rule2A: If the destination mac is found in table, send the packet out that interface
    • Rule2B: if the destination mac is NOT found in table, flood the packet to all interfaces but the interface the packet came from

yes, no problem, I known how it's works (I still think that hertner should fix statically mac address on their switches mac table to avoid this)

What I am seeing on my server is that get ALOT of packets not destined for me like:

No. Time Source Destination Protocol Length Info
1 0.000000 162.62.9.237 78.47.101.73 TCP 60 47741 → 43 [SYN] Seq=0 Win=65535 Len=0

Frame 1: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Ethernet II, Src: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb), Dst: IntelCor_0c:5f:c3 (00:1b:21:0c:5f:c3)
Internet Protocol Version 4, Src: 162.62.9.237, Dst: 78.47.101.73
Transmission Control Protocol, Src Port: 47741, Dst Port: 43, Seq: 0, Len: 0

This packet is from 162.62.9.237 (not me) to 78.47.101.73 (not me either). As I don't have access to the switch I'm connected to I can only assume why I'm getting this packet which is Rule2B - flood the packet. 78.47.101.73 has properly been silent for a while (or not online) meaning the switch doesn't know out of which interface its mac address is located.
In a normal server setup this packet would have been dropped coming into my servers network card as it is neither my mac address (as destination) and its not multicast/broadcast.
The problem here is that my server is also a switch (linux bridge) which means I will allow all packets coming in and filter them through my firewalls (ebtables/iptables).

Ah ok, got It ! I didn't known that you use proxmox firewall with reject. I understand now.

In iptables I have a clear Proxmox rule saying: If the packet destination port is TCP/43 (WHOIS) then reject it.
REJECT is an active reply (contrary to DROP) meaning I would send the following packets:

No. Time Source Destination Protocol Length Info
2 0.000039 78.47.101.73 162.62.9.237 TCP 54 43 → 47741 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Frame 2: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: IntelCor_0c:5f:c3 (00:1b:21:0c:5f:c3), Dst: JuniperN_47:b9:cb (f4:cc:55:47:b9:cb)
Internet Protocol Version 4, Src: 78.47.101.73, Dst: 162.62.9.237
Transmission Control Protocol, Src Port: 43, Dst Port: 47741, Seq: 1, Ack: 1, Len: 0

ok, I didn't known that src mac was forget too here.

This is were it all breaks. I'm now sending out a packet that I didn't "own". This conflicts with Heztner because of switch Rule1. Their switch will now store that the source mac (IntelCor_0c:5f:c3 (00:1b:21:0c:5f:c3)) that I sent out is now located down the interface towards my server, thereby blackholing traffic to the "real" server connected somewhere else.

So, there are ways to combat this, I choose the dropping of outbound packets not from me. You could possibly also locate all rules in iptables and make sure they either a) drops silently (not sending an active reply) and/or b) only reacts if the packet includes your servers IP addresses inbound.

Hope this makes sense.

I wonder if we could drop incoming packets with wrong destination mac with ebtables.
before their are reject by iptables

currently, with proxmox firewall, with have ebtables rules for outgoing traffic only

-A FORWARD -j PVEFW-FORWARD
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
-A PVEFW-FWBR-OUT -i tap100i0 -j tap100i0-OUT
-A tap100i0-OUT -s ! 2:2:ec:b9:34:da -j DROP
-A tap100i0-OUT -p ARP -j tap100i0-OUT-ARP
-A tap100i0-OUT -j ACCEPT
-A tap100i0-OUT-ARP -p ARP --arp-ip-src 10.59.100.103 -j RETURN
-A tap100i0-OUT-ARP -j DROP

but we could add incoming rules too

-A FORWARD -j PVEFW-FORWARD
-A PVEFW-FORWARD -i fwln+ -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
-A PVEFW-FWBR-IN -i tap100i0 -j tap100i0-IN
-A PVEFW-FWBR-OUT -i tap100i0 -j tap100i0-OUT
-A tap100i0-IN -d ! 2:2:ec:b9:34:da -j DROP
-A tap100i0-OUT -s ! 2:2:ec:b9:34:da -j DROP
-A tap100i0-OUT -p ARP -j tap100i0-OUT-ARP
-A tap100i0-OUT -j ACCEPT
-A tap100i0-OUT-ARP -p ARP --arp-ip-src 192.168.0.1 -j RETURN
-A tap100i0-OUT-ARP -j DROP

(where 2:2:ec:b9:34:da is the mac of my vm100 net0)

 

[alxgarder]

yes, no problem, I known how it's works (I still think that hertner should fix statically mac address on their switches mac table to avoid this)



Ah ok, got It ! I didn't known that you use proxmox firewall with reject. I understand now.


ok, I didn't known that src mac was forget too here.



I wonder if we could drop incoming packets with wrong destination mac with ebtables.
before their are reject by iptables

currently, with proxmox firewall, with have ebtables rules for outgoing traffic only

-A FORWARD -j PVEFW-FORWARD
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
-A PVEFW-FWBR-OUT -i tap100i0 -j tap100i0-OUT
-A tap100i0-OUT -s ! 2:2:ec:b9:34:da -j DROP
-A tap100i0-OUT -p ARP -j tap100i0-OUT-ARP
-A tap100i0-OUT -j ACCEPT
-A tap100i0-OUT-ARP -p ARP --arp-ip-src 10.59.100.103 -j RETURN
-A tap100i0-OUT-ARP -j DROP

but we could add incoming rules too

-A FORWARD -j PVEFW-FORWARD
-A PVEFW-FORWARD -i fwln+ -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
-A PVEFW-FWBR-IN -i tap100i0 -j tap100i0-IN
-A PVEFW-FWBR-OUT -i tap100i0 -j tap100i0-OUT
-A tap100i0-IN -d ! 2:2:ec:b9:34:da -j DROP
-A tap100i0-OUT -s ! 2:2:ec:b9:34:da -j DROP
-A tap100i0-OUT -p ARP -j tap100i0-OUT-ARP
-A tap100i0-OUT -j ACCEPT
-A tap100i0-OUT-ARP -p ARP --arp-ip-src 192.168.0.1 -j RETURN
-A tap100i0-OUT-ARP -j DROP

(where 2:2:ec:b9:34:da is the mac of my vm100 net0)

I guess to could have the rules inbound on ebtables. Just remember that multicast and broadcast also needs to get through. I like it outbound as this is where I actually have the issue and I know packets in this direction will have to been sourced from me.

 

[alxgarder]

Thanks for taking the time to reply. Not only kind words; I think that you pointed me to the right direction.

May be this is true in the case you expose where you are receiving whois packets and you (as me, as maybe all) have a REJECT rule in the firewall. But in my case I am receiving traffic to destination IPs (78.xx.xx.xx and others) that are *out of the scope* of my server addressing (176.xx.xx.xx/27) in port 80, 445 and many others. Network scanners? May be...

But this should not happen, I think. May be I am wrong but I think that the upper level routing system should had never injected that traffic into the switch I am connected to because my network segment is out of that scope by network mask (176.xx.xx.xx/27). That /27 imply that only packets destined to an IP in the 176.<something> range should reach my network segment. But I am receiving traffic destined to IP addresses 78.<something>, 5.<something>, etc. and I have no explanation for that. Does that mean that Hetzner makes no routing at all inside their network and all is a mess of traffic directed by ARP and MAC addressing? I expect not...

Anyway I still do not fully understand why I may be sending out packets with that MACs (my dump capture is still empty after several hours) but they say I do and me, as you, has nothing in the server that explain that behavior. So I cannot avoid it so far except perhaps using your ebtables solution blindly. I am constantly receiving traffic in the public interface that is not directed to any of my IPs nor MAC addresses like this one:

17:38:33.242902 84:c1:c1:76:a9:a2 > 00:50:56:00:39:9e, ethertype IPv4 (0x0800), length 60: 79.124.62.86.48386 > 5.9.9.140.24129: Flags , seq 806294571, win 1024, length 0

But the system, normally, does not reply to them. Finding the ones that force a reply is being harder in this case.

Best regards.


(edited) Latest addition: I started a new capture and, at last, I got packets (outgoing):

17:48:04.991198 90:1b:0e:93:25:30 > 84:c1:c1:76:a9:a2, ethertype IPv4 (0x0800), length 54: 5.9.170.34.43 > 46.17.102.78.46368: Flags [R.], seq 0, ack 3876508740, win 0, length 0
18:04:08.296780 00:50:56:00:9b:c8 > 84:c1:c1:76:a9:a2, ethertype IPv4 (0x0800), length 54: 5.9.170.36.43 > 46.17.102.78.46368: Flags [R.], seq 0, ack 1619227194, win 0, length 0
18:07:34.628512 00:50:56:00:9f:a3 > 84:c1:c1:76:a9:a2, ethertype IPv4 (0x0800), length 54: 5.9.170.37.43 > 46.17.102.78.46368: Flags [R.], seq 0, ack 3103376507, win 0, length 0

Looks like they are R[eset], of course from MACs not owned by me so Hetzner will, for sure, complaint. Now to find out why (and who) is sending this kind of packets only sometimes and not always or never.

More on this: I saw that I was sendig resets for some packets (5 now). Noticed that destination MACs were the same for all so I asked the server if it knew it.

arp -a: static.225.xx.xx.176.clients.your-server.de (176.xx.xxx.225) at 84:c1:c1:76:a9:a2 [ether] on vmbr0

(edit 2)The reply is that this MAC is the one for my network broadcast address (the one ending in 255) but, as it can be seen in the capture the destination IP is from outside of my network (46.17.102.78). Someone can help me clarify this mess?
I read wrongly the IP address. Not 255 but 225, the gateway adress and, of course, that is fully correct. All incoming trafic must come from the network gateway MAC address as it does. Sorry for the confusion.

Do you have the capture file from your last dump? I'm super interested in seeing it so I can compare to mine.

 

[spirit]

Another possibility:

sysctl -w net.netfilter.nf_conntrack_tcp_loose = 0

it should protect from unknown ACK, or SYN-ACK. (but not SYN), marking packets from unestablished connection as invalid.

They will drop packets early in iptables with the default rule.
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP


BUT, this currently breaking live migration. (but I don't think you need it anyway at hetzner)

 

[doymer]

Do you have the capture file from your last dump? I'm super interested in seeing it so I can compare to mine.

If you are refering to my complaint that I am receiving traffic not addressed to me it is as easy as taking the tcpdump line and invert the direction of the capture:

tcpdump -Q in -n -e not ether host <your first MAC> and not ether host <your second MAC> and not ether host <your other MAC>

This will reveal traffic you receive not addressed to your MACs. It is correct to see traffic not addressed to your specific IPs but inside the shared addressing space of your subnet. As an example if you IP is 192.168.1.5/27 traffic to 192.168.1.20 that is is the same subnet as you (in fact all that is between 192.168.1.1 and 192.168.1.30) is theoretically valid. But you should never, ever, see trafic to 192.168.2.x or any other else. Of course this is if routing is well configured. But I guess that in your capture you will see lots of packets sent to an IP address that is outside your scope. I am very interested too in knowing if that happens to you too. I have other servers at Hetzner and they also get this irregular traffic though not so intense. In this short capture of one other servers you can see:

1.- Wrong? traffic even inside their vSwitch (my server is using one of their virtual switches to connect with other servers internally. That 33:33:ff:* does not exist in that server though the IPv6 addresses are correct)
10:19:18.735110 84:c1:c1:76:af:b0 > 33:33:ff:00:00:02, ethertype 802.1Q (0x8100), length 90: vlan 4000, p 0, ethertype IPv6, fe80::86c1:c100:76:aff0 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has 2a01:xxx:xxx:xxx::2, length 32
10:19:18.735274 84:c1:c1:76:af:b0 > 33:33:ff:b2:13:b8, ethertype 802.1Q (0x8100), length 90: vlan 4000, p 0, ethertype IPv6, fe80::86c1:c100:76:aff0 > ff02::1:ffb2:13b8: ICMP6, neighbor solicitation, who has fe80::xxx:xxx:xxx:13b8, length 32

2.- Valid traffic, not addressed to any of my servers but inside the network addressing scope (I am node 211)
10:19:18.870997 84:c1:c1:76:a9:50 > 00:50:56:00:76:cd, ethertype IPv4 (0x0800), length 66: 14.229.20.246.50526 > 176.xx.xx.210.445: Flags , seq 2037627499, win 8192, options [mss 1452,nop,wscale 2,nop,nop,sackOK], length 0
10:19:19.176027 84:c1:c1:76:a9:50 > 08:60:6e:45:c2:65, ethertype IPv4 (0x0800), length 60: 43.242.128.20.21 > 176.xx.xx.213.21: Flags , seq 23592960, win 0, length 0

3.- Invalid traffic shot to anywhere else inside the Hetzner cyberspace
10:19:19.237437 84:c1:c1:76:a9:50 > 00:50:56:00:62:1d, ethertype IPv4 (0x0800), length 60: 46.148.20.13.60000 > 5.9.253.229.1722: Flags , seq 1770544271, win 1024, length 0

For the initial server I did not capture to a file but to the screen so I lost the captured lines as soon as I closed the session but it is much about the same: From time to time (all along the day yesterday) the vmbr switch (I think it is the only one that is able to generate that kind of traffic because none of the packet data -MACs, IPs- belongs to any of my hosts and if it was the host the one that generated it, at the very least, the source IP should be one of the right IPs) sent a reset packet back.

And I think this is malware because no one is so incompetent to, constantly, try to map a SMB drive (port 445) that does not own and not notice it is not the right IP. I think these are tries to find holes in servers exposed to internet and thus it is malware. And, should this be true, Hetzner is somehow collaborating to expand this malware beause they should be discarding that traffic in first place and blaming us of their misconfiguration.

But this is only my view and I may be wrong. Don't know if this was what you were expecting. If not do not hesitate to tell me and I'll try to grab the data you need.

 

[doymer]

I have a new theory about why this may be happenning. Please correct me if I am wrong.

The edge switch of the proxmox server (vmbr0) is being constanly bombed with packets going to multiple MAC addesses. As alxgarder explained in one prior message:

Whenever a packet comes in:

• Rule1: Record the source mac and the interface it came from in a table
• Rule2: Lookup the destination mac in the table created in rule 1:
  • Rule2A: If the destination mac is found in table, send the packet out that interface
  • Rule2B: if the destination mac is NOT found in table, flood the packet to all interfaces but the interface the packet came from

So I imagine thousands of different MAC addesses hiting my vswitch. It save the MAC, look for the destination, does not find it because it is not one of the configured and flood the packet to the rest of interfaces, flooding them to your guests too.

But what if the number of that MAC table reaches and exceed the maximum number of MACs that the switch can handle before the age timer of the first ones expire? In physical switches that number is usually 8000 MAC addresses but do not know the actual number in proxmox/debian vswitch . The next different MAC cannot be inserted in the table and then (specullating in option b):

a) the switch may behave like a hub flooding the packet to all interfaces *including* the one it received it if no port protection is in place.
b) The switch send back a reset packet and forgets about it or any other mechanism if port protection is in place.

Looks like option b) may be ocurring here if that option exist. Do not know the details so may be some help from the proxmox/debian staff will be a great help. Right now my vmbr0 switch has lots of MACS:

root@pve6n2 ~ # brctl showmacs vmbr0 | wc -l
12289

Excessive for a single server with two guests and not too much traffic. And ALL external MACs has an ageing timer of 0 (do not know the actual implications of that).

Hetzner reply to my request for and explanation is:

>> Your server is receiving broadcast traffic from neighbor IPs this is an normal behavior on switch. You can block such traffic on your FW.

So now it is clear that received traffic is being broadcasted in a L2 fashion not taking into account L3 info and they expect to be us the ones that have to fix that mess.

Waiting for your comments get warm regards.

 

[doymer]

So I looked a lot at all these filters and was really unsure what they did (and more worried how they actually would behave) so I took a little different approach in order to troubleshoot.
ebtables does layer2 rules, so I started with just seeing if I could log every time the server was sending out offending frames. I ended with the following:

ebtables -I OUTPUT -o enp2s0 -p IPv4 --among-src ! d4:3d:7e:da:f0:03 --log-level info --log-prefix "MAC-FLOOD-O" --log-ip -j CONTINUE
ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j CONTINUE

As explained I have two MAC addresses. d4:3d:7e:da:f0:03 which is the BIA of the main NIC and 00:50:56:15:14:e5 which is paired with my secondary IP.
This behaves the same as the tcpdump I did earlier - basically saying: log if the source mac is not any of my mac addresses. That caught the next wave of frames (5 to be exact). All of them came on the FORWARD chain.

Therefore, I have now changed the FORWARD chain to drop:

ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j DROP

.. and I'm running the tcpdump also to see if this stops the packets.

Until tomorrow..

If you have two MAC addresses I think that your ebtables should be

ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! 00:50:56:15:14:e5,d4:3d:7e:da:f0:03 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j DROP

instead to allow traffic from the other MAC address also.

 

[dendi]

Hello,
I had three packets overnight, after a new abuse mail on two servers (I have more then 10 running servers) that are the same of last week.
So the problem does not exist on most servers, for now.

tcpdump -Q out -n -e not ether host a8:a1:59:3e:b9:39
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp35s0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:00:37.646080 30:5a:3a:75:5a:c2 > cc:e1:7f:07:e1:a7, ethertype IPv4 (0x0800), length 54: 136.243.81.4.43 > 144.217.24.7.21264: Flags [R.], seq 0, ack 914352, win 0, length 0
06:29:14.766505 30:5a:3a:75:5a:c2 > cc:e1:7f:07:e1:a7, ethertype IPv4 (0x0800), length 54: 136.243.81.4.43 > 80.82.77.139.10943: Flags [R.], seq 0, ack 2699844537, win 0, length 0
09:01:31.147134 30:5a:3a:76:b7:5d > cc:e1:7f:07:e1:a7, ethertype IPv4 (0x0800), length 54: 136.243.81.33.43 > 45.146.164.182.49133: Flags [R.], seq 0, ack 475803466, win 0, length 0

tomorrow I will try with the suggested ebtable rule:

ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! a8:a1:59:3e:b9:39 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j DROP

But my stupid question is: if the problem is the REJECT in the proxmox firewall, is it not enough to replace it with DROP?

 

[spirit]

But my stupid question is: if the problem is the REJECT in the proxmox firewall, is it not enough to replace it with DROP?

yes, you can use DROP if you want.

 

[dendi]

yes, you can use DROP if you want.

do you think it makes sense?
I admit, I'm not an iptables expert and I'm afraid of going to change system files, I think I have to modify /usr/share/perl5/PVE/Firewall.pm and reload pve-firewall, do you have some tip?

For the ebtable solution, is it enough to type it on the console (of course it will not survive to reboot) or I need to reload or restart something?

 

[doymer]

Hello,
I had three packets overnight, after a new abuse mail on two servers (I have more then 10 running servers) that are the same of last week.
So the problem does not exist on most servers, for now.

tcpdump -Q out -n -e not ether host a8:a1:59:3e:b9:39
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp35s0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:00:37.646080 30:5a:3a:75:5a:c2 > cc:e1:7f:07:e1:a7, ethertype IPv4 (0x0800), length 54: 136.243.81.4.43 > 144.217.24.7.21264: Flags [R.], seq 0, ack 914352, win 0, length 0
06:29:14.766505 30:5a:3a:75:5a:c2 > cc:e1:7f:07:e1:a7, ethertype IPv4 (0x0800), length 54: 136.243.81.4.43 > 80.82.77.139.10943: Flags [R.], seq 0, ack 2699844537, win 0, length 0
09:01:31.147134 30:5a:3a:76:b7:5d > cc:e1:7f:07:e1:a7, ethertype IPv4 (0x0800), length 54: 136.243.81.33.43 > 45.146.164.182.49133: Flags [R.], seq 0, ack 475803466, win 0, length 0

tomorrow I will try with the suggested ebtable rule:

ebtables -I FORWARD -o enp2s0 -p IPv4 --among-src ! a8:a1:59:3e:b9:39 --log-level info --log-prefix "MAC-FLOOD-F" --log-ip -j DROP

But my stupid question is: if the problem is the REJECT in the proxmox firewall, is it not enough to replace it with DROP?

Take into account that your interface is not enp2s0 but enp35s0 as from your capture.

And for your question perhaps it better is answered by the proxmox team because that line is appended by the server itself, not something that we configure by hand. Perhaps adding an explicit drop in the datacenter firewall is all that is needed to override that behavior but I have not tried it.

 

[doymer]

do you think it makes sense?
I admit, I'm not an iptables expert and I'm afraid of going to change system files, I think I have to modify /usr/share/perl5/PVE/Firewall.pm and reload pve-firewall, do you have some tip?

For the ebtable solution, is it enough to type it on the console (of course it will not survive to reboot) or I need to reload or restart something?

Changing Firewall.pm, and not breaking anything, may not survive an update/upgrade so, in the long term, it may be worse as you will have forgotten but it and will not redo the change .

 

[spirit]

do you think it makes sense?
I admit, I'm not an iptables expert and I'm afraid of going to change system files, I think I have to modify /usr/share/perl5/PVE/Firewall.pm and reload pve-firewall, do you have some tip?

why do you want to modifiy FIrewall.pm ? you can choose drop in rules or default actions.

It could be great to test : "sysctl -w net.netfilter.nf_conntrack_tcp_loose = 0" , to see if it's help too.
this will drop ACK, SYN-ACK from unknown established connection in the firewall. I don't known if hertzner is flooding SYN packets too ?

 

[dendi]

The problem here is that my server is also a switch (linux bridge) which means I will allow all packets coming in and filter them through my firewalls (ebtables/iptables). In iptables I have a clear Proxmox rule saying: If the packet destination port is TCP/43 (WHOIS) then reject it.
REJECT is an active reply (contrary to DROP) meaning I would send the following packets:

No. Time Source Destination Protocol Length Info
2 0.000039 78.47.101.73 162.62.9.237 TCP 54 43 → 47741 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

I have reject rules too but I have default IN policy to DROP and not create REJECT rules, I think theese are defaults of proxmox that I can't see or edit.

# grep -i reject /etc/pve/firewall/*
#

I see this line in my iptables -L for example:

PVEFW-reject  tcp  --  anywhere             anywhere             tcp dpt:whois

 

[spirit]

I have reject rules too but I have default IN policy to DROP and not create REJECT rules, I think theese are defaults of proxmox that I can't see or edit.

# grep -i reject /etc/pve/firewall/*
#

I see this line in my iptables -L for example:

PVEFW-reject  tcp  --  anywhere             anywhere             tcp dpt:whois

ok, they are a default rule in DROP action, rejecting port 43.


"-A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject"

not sure why this rule exist. I'll ask to pve-devel mailing list.

Edit:
as workaround, you can add a rule to drop port 43, it should work.

 

[spirit]

https://git.proxmox.com/?p=pve-firewall.git;a=commit;h=d9e7522b561ceb323e93affb29c9fced89fed967

fix the reject tcp/43 in default drop rule
(will be available in pve-firewall 4.2-3)

 

[spirit]

Hi Guys,

I think I have found a cleaner way, with a simple:

echo 0 > /sys/class/net/tap100i0/brport/unicast_flood

or even

echo 0 > /sys/class/net/fwpr100p0/brport/unicast_flood


Like this, the unknown macs are not flooded to the fwbr bridge, so no firewall needed.

I have tested it, forging destination mac packets, and it seem to works fine

Can you test it ?
(vm need to be started first)