Proxmox generate 2 mac address visibile on the switch not allowed by the data center

[openaspace]

This settings?

 

[spirit]

the patch to avoid bad traffic with wrong mac to vm && firewall is not yet available in proxmox7. I just send the patch to proxmox devel mailing list friday

https://lists.proxmox.com/pipermail/pve-devel/2021-September/050090.html
https://lists.proxmox.com/pipermail/pve-devel/2021-September/050093.html
https://lists.proxmox.com/pipermail/pve-devel/2021-September/050095.html
https://lists.proxmox.com/pipermail/pve-devel/2021-September/050094.html

(If you have a proxmox7 server to test, I can provide you some test .deb packages)



The mac address prefix option at datacenter level, is prefix when you autogenerate mac address for the vms.

 

[openaspace]

the patch to avoid bad traffic with wrong mac to vm && firewall is not yet available in proxmox7. I just send the patch to proxmox devel mailing list friday

https://lists.proxmox.com/pipermail/pve-devel/2021-September/050090.html
https://lists.proxmox.com/pipermail/pve-devel/2021-September/050093.html
https://lists.proxmox.com/pipermail/pve-devel/2021-September/050095.html
https://lists.proxmox.com/pipermail/pve-devel/2021-September/050094.html

(If you have a proxmox7 server to test, I can provide you some test .deb packages)



The mac address prefix option at datacenter level, is prefix when you autogenerate mac address for the vms.

Really thank you for your work.
I will migrate to 7.

 

[spirit]

if you want test on proxmox7, download this deb:

https://mutulin1.odiso.net/libpve-common-perl_7.0-6_all.deb
https://mutulin1.odiso.net/libpve-network-perl_0.6.1_all.deb (optionnal, only if you use the sdn beta feature)
https://mutulin1.odiso.net/pve-container_4.0-9_all.deb (only if you use containers)
https://mutulin1.odiso.net/qemu-server_7.0-13_amd64.deb (only if you use vms)

and install them with "dpkg -i *.deb".

then, edit /etc/network/interfaces,

and under the vmbrX bridge, add "bridge-disable-mac-learning 1"

auto vmbr0
iface vmbr0 inet manual
        bridge ports ...
       ...
        bridge-disable-mac-learning 1

Then, reboot.

That's all.

 

[fuck]

I'm tired of hetzner mac issues and thinking to switch provider.
But seriously, what the reason they implemented a mac address monitoring? How it's helping them? What issues solving?

 

[openaspace]

I'm tired of hetzner mac issues and thinking to switch provider.
But seriously, what the reason they implemented a mac address monitoring? How it's helping them? What issues solving?

Nooo is one of the best companies in Europe. Cheap servers, flat 1gbps bandwidth. I will try a new setup soon with Routed networking instead of bridged.

 

[JavierG]

I have a problem, i am looking for help. I pay to help.


Hetzner send me this:
We have detected that your server is using different MAC addresses from those allowed by your Robot account.

Please take all necessary measures to avoid this in the future and to solve the issue.
We also request that you send a short response to us. This response should contain information about how this could have happened and what you intend to do about it.
In the event that the following steps are not completed successfully, your server can be locked at any time after 2022-02-23 16:47:06 +0100.

How to proceed:
- Solve the issue
- Please note, in case you have fixed the problem, please wait at least 10 minutes before rechecking: https://abuse.hetzner.com/retries/?token=a6a5a33cfd39acb2ca672841935a53a
- After successfully testing that the issue is resolved, send us a statement by using the following link: https://abuse.hetzner.com/statements/?token=a6a5a33cfd39acb2ca672841935a53a

Please visit our FAQ here, if you are unsure how to proceed:
https://docs.hetzner.com/robot/dedicated-server/faq/error-faq/#mac-errors

Important note:
When replying to us, please leave the abuse ID [AbuseID:A0CE8B:1F] unchanged in the subject line. Manual replies will only be handled in the event of a lock.
Please note that we do not provide telephone support in our department. If you have any questions, please send them to us by responding to this email.

Allowed MACs:
44:8a:5b:2c:31:ea
Unallowed MACs:
2a:ba:40:71:83:a2
2c:96:40:71:83:a2


I dont fix this. can anybody help me? I will pay for the help

 

[openaspace]

After migrate to proxmox 7 I don't have received any notifications about mac address.

regards.

 

[Undergrid]

I have a problem, i am looking for help. I pay to help.


Hetzner send me this:
We have detected that your server is using different MAC addresses from those allowed by your Robot account.

<snip>

If you aren't already on proxmox 7, upgrade. That fixed this issue for me at Hetzner.

Edit: Be careful and read the notes before upgrading! You have to set your MAC address in the IPv4 network config (you probably want Solution B) otherwise you'll end up locked out (unless you've also configured IPv6)

 

[pro-ite]

Nice, almost >2 years and on PVE 7.4-17 I got this warning from hetzner.
In the pve-firewall.log i see unallowed MACs to DEST=notmyIP (but neighbor)
Could it be that PVE means to forward this to the "notmyIP"?

 

[silverstone]

Nice, almost >2 years and on PVE 7.4-17 I got this warning from hetzner.
In the pve-firewall.log i see unallowed MACs to DEST=notmyIP (but neighbor)
Could it be that PVE means to forward this to the "notmyIP"?

Same experience here ...

The main Fix (besides changing a bunch of sysctl, disable mac bridge learning & unicast/multicast flood in bridge settings) was to add a Firewall Rule to Drop Incoming traffic with Destination = NOT_MY_IPs.

Unfortunately I did NOT find an easy way to Invert the sense of Matching, so I went ahead and build a "not_assigned_public_ips" list consisting of the entire /26 Subnet MINUS my Own IPs and the Gateway ....

That seems to help quite a bit (knock on wood), but without I'm pretty sure I would still have issues .

EDIT 1: Otherwise ... Routing Setup OR, most likely, PCIe passthrough of the NIC into OPNSense VM and handle everything from there.

The Big Issue would be if OPNSense has some kind of Failure or breaks down, Recovery would be very difficult. Well, I guess just a matter of Booting into Recovery mode and revert to the currently kind-of-working Bridged Setup .

 

[spirit]

"bridge-disable-mac-learning 1" still apply for bridged setup at hetzner

+ if you use proxmox firewall, dont use "reject", use "drop"

 

[silverstone]

"bridge-disable-mac-learning 1" still apply for bridged setup at hetzner

+ if you use proxmox firewall, dont use "reject", use "drop"

I also have that implemented (and disabled multicast flood & unicast flood).

I also tried to use "port isolation" for the member of a Bridge, that also does NOT solve this issue.

Neither does ebtables rules (because actually the traffic is addressed to my MAC but NOT my IP - weird stuff at Hetzner).

So if I don't have a firewall rule in place I am getting INPUT traffic which is destined to other Servers in my Subnet, NOT my Servers.

So from what I can see, the problem is NOT solved.

 

[Undergrid]

Unfortunately I did NOT find an easy way to Invert the sense of Matching, so I went ahead and build a "not_assigned_public_ips" list consisting of the entire /26 Subnet MINUS my Own IPs and the Gateway ....

Define an IPSet and add each of your IP addresses making sure the "nomatch" check box is ticked for each item. Then you just add your DROP rule with destination set to your IPSet.

 

[silverstone]

Define an IPSet and add each of your IP addresses making sure the "nomatch" check box is ticked for each item. Then you just add your DROP rule with destination set to your IPSet.

I originally tried that but I think it was not behaving correctly.

Maybe a OR(NOT(A) , NOT(B)) not working as expected and not being equivalent to NOT(A OR B), when you do your ipset or something like that.

 

[Undergrid]

I also have that implemented (and disabled multicast flood & unicast flood).

I also tried to use "port isolation" for the member of a Bridge, that also does NOT solve this issue.

Neither does ebtables rules (because actually the traffic is addressed to my MAC but NOT my IP - weird stuff at Hetzner).

So if I don't have a firewall rule in place I am getting INPUT traffic which is destined to other Servers in my Subnet, NOT my Servers.

So from what I can see, the problem is NOT solved.

So what's the problem with having a firewall rule that drops everything not addresses to one of your IP addresses?

And yeah, the whole my MAC other persons IP is something weird Hetzner does, but they aren't going to fix it, so we have to work around it.

 

[silverstone]

So what's the problem with having a firewall rule that drops everything not addresses to one of your IP addresses?

And yeah, the whole my MAC other persons IP is something weird Hetzner does, but they aren't going to fix it, so we have to work around it.

Besides the fact that the ipset with the nomatch didn't appear to work correctly (at least for me), what feels wrong is that we filter on the IPv4 Level, rather than MAC Address. Then again it's true that I cannot really filter by MAC Address since apparently I am marked as the destination MAC, even though the IP address is not mine ...

Whether it works, that remains to be seen (knock on wood). I'm already VERY happy that you brought that Firewall Rule up to my Attention, as I believe most/all of the Traffic not addressed to me was dropped .

 

[silverstone]

So what's the problem with having a firewall rule that drops everything not addresses to one of your IP addresses?

And yeah, the whole my MAC other persons IP is something weird Hetzner does, but they aren't going to fix it, so we have to work around it.

So you agree that the problem is NOT fixed by:

• bridge-disable-mac-learning 1
• any of my sysctl Configuration
• any of the other Linux Bridge Options such as

bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096

• manually disabling unicast flooding

#!/bin/bash

mapfile -t list < <( find /sys -iname *unicast_flood* )

for item in "${list[@]}"
do
    # Echo
    if [[ -n "${verbose}" ]]
    then
       echo "Executing: echo 0 > ${item}"
    fi

    # Disable unicast flooding on all interfaces
    echo 0 > $item
done

• Attempting to force Linux Bridge Port Isolation (see e.g. https://forum.proxmox.com/threads/vmbr1-vm-communication-isolation.73885/, https://forum.proxmox.com/threads/network-bridge-where-all-member-vms-are-isolated.145607/ or https://bugzilla.proxmox.com/show_bug.cgi?id=4300)

So despite all the claims, I'd still say that the Interface eth0 (main Physical Interface of the Server) is STILL in Promiscous Mode, even though I put all those settings on the Bridge vmbr0. Maybe this is needed on the uplink port, but this matches the behavior that I can see (getting traffic destined to other Server IP addresses, not mine).

Because, from what I can read on this Forum, everybody believes (probably inaccurately) that the Problem has been fixed by a Proxmox VE Software Update in PVE 7.x or so, but my feeling (and Hetzner MAC Abuse Emails) on Proxmox 8.2.2 tell otherwise ...

The ONLY thing that most likely helps is your Firewall Rule Recommendation (although I still am NOT sure if the IPSet with nomatch is working, I'll probably just do a "double" DROP Rule, one with the explicit Exclusion of the Hosts in the Subnet like I am currently doing, and one extra rule with the nomatch Option (which I don't really trust to be Honest ...).

I had added also this to /etc/network/interfaces to filter based on MAC Address but it's most likely dropping stuff destined to me by the Gateway, so NOT a good Solution (because anyway the Destination MAC Address is mine, even though the IP is NOT):

auto vmbr0
iface vmbr0 inet static
        ...
        pre-up ebtables -I OUTPUT -o eth0 --among-src ! MY_MAC_01,MY_MAC_02 --log-level info --log-prefix "MAC-FLOOD-OUTPUT" --log-ip -j DROP
        pre-up ebtables -I FORWARD -o eth0 --among-src ! MY_MAC_01,MY_MAC_02 --log-level info --log-prefix "MAC-FLOOD-FORWARD-OUTBOUND" --log-ip -j DROP
        pre-up ebtables -I FORWARD -i eth0 --among-dst ! MY_MAC_01,MY_MAC_02 --log-level info --log-prefix "MAC-FLOOD-FORWARD-INBOUND" --log-ip -j DROP
        pre-up ebtables -I INPUT -i eth0 --among-dst ! MY_MAC_01,MY_MAC_02 --log-level info --log-prefix "MAC-FLOOD-INPUT" --log-ip -j DROP

And I feel like the ipfilter setting on VM Level with ipfilter-net0 is NOT sufficient, because the Host does not have such a setting (which IMHO is what your Firewall Rule helps with) . On VM Level adding the IPs to is most likely sufficient (but I also added the Firewall Rule on VM Level as well to drop block_wrong_dest there as well just in case), but the Host Part is NOT taken care of "by default".

This should be MUCH better documented IMHO. It would spare MANY Hetzner Customers (and potentially other) A LOT of Frustration.

And IMHO there should be a ipfilter Rule of the Host as well e.g. ipfilter-eth0 or ipfilter-vmbr0, similarly to what is done to VMs,

Maybe we should do a Wiki Page Ourselves, if that is possible, to try to help other Customers .

@proxmox team: any chance of getting at least some Description implemented in the Wiki ?

@Undergrid: the other thing I can think of is ... Can we make a Rule / Service / etc that DISABLES ALL TRAFFIC, in case (for whatever reason) pve-firewall fails/gets down ? Like a "lockdown mode" on a VPN for Instance. Because otherwise, if the Firewall for whatever Reason fails, the Server becomes vulnerable. Not sure what happens during pve-firewall restarteither (am I letting some Packets go through while the Firewall is restarting/reloading the rules ?) ....

 

[spirit]

Here a summary to help people:


bridge-disable-mac-learning 1

This is disabling unicast flood && bridge learning on the bridge. (proxmox register manually vm mac on ports)

That mean that traffic incoming to the server is not forwarded to the vms if the destination mac is different than the vm mac. (It'll be dropped when entering the bridge)

If bridge-vlan-aware is enabled && bridge-disable-mac-learning 1 , the linux kernel should disable promisc mode on the bridge. (all vms interfaces need to be started with unicast flood disable, to that need at minimum a full restart of the server or a full restart of all vms after disabling bridge-disable-mac-learning)

This should block incoming packets with dst mac address unknown, the packet will be dropped at physical interface level, before going into the server.




Now, about the vm firewall rules REJECT vs DROP.
This is because of current iptables implementation, where we need an extra fwbrX bridge. The reject packet are send by proxmox itself, with a random mac of the fwbr bridge.
This should be finally fixed with the new nftables implementation, because we don't need the extra fwbr bridge.
I don't have hetzner server to test, but if somebody could test it, it could be great



BTW,if you are a hetzner customer, please ask them to fix their fucking networking layer2 filtering.
(basically, they don't filter/register mac on their physical switch, I think because they use cheap switch. So they use some kind of async deamon, looking traffic after bad packets are flooded, then block their switch ports)

 

[silverstone]

Here a summary to help people:


bridge-disable-mac-learning 1

This is disabling unicast flood && bridge learning on the bridge. (proxmox register manually vm mac on ports)

That mean that traffic incoming to the server is not forwarded to the vms if the destination mac is different than the vm mac. (It'll be dropped when entering the bridge)

If bridge-vlan-aware is enabled && bridge-disable-mac-learning 1 , the linux kernel should disable promisc mode on the bridge. (all vms interfaces need to be started with unicast flood disable, to that need at minimum a full restart of the server or a full restart of all vms after disabling bridge-disable-mac-learning)

This should block incoming packets with dst mac address unknown, the packet will be dropped at physical interface level, before going into the server.




Now, about the vm firewall rules REJECT vs DROP.
This is because of current iptables implementation, where we need an extra fwbrX bridge. The reject packet are send by proxmox itself, with a random mac of the fwbr bridge.
This should be finally fixed with the new nftables implementation, because we don't need the extra fwbr bridge.
I don't have hetzner server to test, but if somebody could test it, it could be great



BTW,if you are a hetzner customer, please ask them to fix their fucking networking layer2 filtering.
(basically, they don't filter/register mac on their physical switch, I think because they use cheap switch. So they use some kind of async deamon, looking traffic after bad packets are flooded, then block their switch ports)

As I said, unfortunately, with Hetzner at least, this does NOT seem to be sufficient .

I'm starting to wonder if I should transition to a Brouter / Routed setup and avoid all of these Issues with IPv4 Single IPs, just like I am doing with IPv6, although I am not sure if I can do it without losing some of the IPs (or using 1 dedicated IP for Proxmox VE just like I am using right now is all I need in order to get that working).

IPv6 Subnet must be Routed. And, apparently, in OPNSense VM both the /64 and the Additional /56 IPv6 Subnet I purchased work correctly, it's just that one Gateway appears down / unpingable, if the other Gateway is used instead (I guess OPNSense cannot have 2 "default" Gateways, which is fair, although it's weird that they don't allow to monitor an External IP when the Gateway isn't being used ...).

As to Hetzner fixing their L2 Filtering ... Well, if they didn't bother for more than 2 Years, why would you expect they do now ?