Proxmox with hetzner, what a love story.

[T0xicFX]

So, i decided to install proxmox on a Hetzner dedicated, i have additional IPs with mac addresses, and i got a whole /29 subnet without any mac addresses as hetzner doesn't provide any for subnets.

They ask to do a routed configuration but as soon as i try to make one, i fuck my entire network, i'm actually so bad to do that
so, i'm sharing my network configuration here and i hope a kind soul can help me making a routed configuration so hetzner doesn't bother me anymore using unauthorized mac addresses.

Mac Abuse mail i receive every time i make a vm on proxmox with an subnet IP & random mac address

Dear Mr XXXXXXXXXXXXX,

We have detected that your server is using different MAC addresses from those allowed by your Robot account.

Please take all necessary measures to avoid this in the future and to solve the issue.
We also request that you send a short response to us. This response should contain information about how this could have happened and what you intend to do about it.
In the event that the following steps are not completed successfully, your server can be locked at any time after 2023-05-11 12:41:10 +0200.

How to proceed:
- Solve the issue
- Please note, in case you have fixed the problem, please wait at least 10 minutes before rechecking: https://abuse.hetzner.com/retries/?token=582cbc557815718efa91435ed2f19ee
- After successfully testing that the issue is resolved, send us a statement by using the following link: https://abuse.hetzner.com/statements/?token=582cbc557815718efa91435ed2f19ee

Please visit our FAQ here, if you are unsure how to proceed:
https://docs.hetzner.com/robot/dedicated-server/faq/error-faq/#mac-errors

Important note:
When replying to us, please leave the abuse ID [AbuseID:C65CDA:37] unchanged in the subject line. Manual replies will only be handled in the event of a lock.
Please note that we do not provide telephone support in our department. If you have any questions, please send them to us by responding to this email.

Kind regards

Network department

Hetzner Online GmbH
Industriestr. 25
91710 Gunzenhausen / Germany
Tel: +49 9831 505-0
Fax: +49 9831 505-3
abuse@hetzner.com
www.hetzner.com

Register Court: Registergericht Ansbach, HRB 6089
CEO: Martin Hetzner, Stephan Konvickova, Günther Müller

For the purposes of this communication, we may save some
of your personal data. For information on our data privacy
policy, please see: www.hetzner.com/datenschutzhinweis

> #1766669 (168.119.12.xxx)
>    Allowed MACs:
>        00:50:56:00:e1:6c
>        00:50:56:00:e1:8e
>        00:50:56:00:e1:86
>        00:50:56:00:e1:80
>        00:50:56:00:e1:8f
>        d4:5d:64:bb:c5:4e
>        00:50:56:00:d6:8f
>    Unallowed MACs:
>        1a:84:8b:e2:f0:2f
>        4a:46:58:5a:4f:a3
>        86:20:09:38:91:1e
>

There's my network configuration

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

iface enp195s0 inet manual

iface enp195s0 inet6 static
        address 2a01:4f8:242:42d8::2/64
        gateway fe80::1

auto vmbr0
iface vmbr0 inet static
        address 168.119.12.xxx/26
        gateway 168.119.12.xx
        bridge-ports enp195s0
        bridge-stp off
        bridge-fd 0
        hwaddress d4:5d:64:bb:c5:4e
        up ip route add 168.119.12.xxx/32 dev vmbr0 (additional ip with mac address)
        up ip route add 168.119.12.xxx/32 dev vmbr0 (additional ip with mac address)
        up ip route add 168.119.12.xxx/32 dev vmbr0 (additional ip with mac address)
        up ip route add 168.119.12.xxx/32 dev vmbr0 (additional ip with mac address)
        up ip route add 168.119.12.xxx/32 dev vmbr0 (additional ip with mac address)
        up ip route add 168.119.12.xxx/32 dev vmbr0 (additional ip with mac address)
        up ip route add 136.243.186.xxx/29 dev vmbr0 (subnet without any mac address provided)

iface eth0 inet manual

Thanks !

 

[gurubert]

1. You should put the IPv6 address also on vmbr0. enp195s0 is just a bridge port.

2. VMs attached to vmbr0 have direct Ethernet access to the Hetzner network (it's a bridge). You do not need to add host routes for the 168.119.12.xxx addresses.

3. The network 136.243.186.xxx/29 should be routed over a separate bridge interface (vmbr1) not connected to Hetzner's Ethernet. Just add a bridge without any ports, make the first IP of your /29 the address of the bridge interface on the Proxmox node and then attach the VMs to that bridge and give them addresses out of that /29.

 

[T0xicFX]

1. You should put the IPv6 address also on vmbr0. enp195s0 is just a bridge port.

2. VMs attached to vmbr0 have direct Ethernet access to the Hetzner network (it's a bridge). You do not need to add host routes for the 168.119.12.xxx addresses.

3. The network 136.243.186.xxx/29 should be routed over a separate bridge interface (vmbr1) not connected to Hetzner's Ethernet. Just add a bridge without any ports, make the first IP of your /29 the address of the bridge interface on the Proxmox node and then attach the VMs to that bridge and give them addresses out of that /29.

so i would need to do something like that?

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

iface enp195s0 inet manual

iface enp195s0 inet6 manual

auto vmbr0
iface vmbr0 inet static
        address 168.119.12.xxx/26
        gateway 168.119.12.xx
        bridge-ports enp195s0
        bridge-stp off
        bridge-fd 0
        hwaddress d4:5d:64:bb:c5:4e

iface vmbr0 inet6 static
        address 2a01:4f8:242:42d8::2/64
        gateway fe80::1
        
iface vmbr1 inet static
        address 136.243.186.xxx/29
        gateway 168.119.12.xxx
        bridge-ports none
        bridge-stp off
        bridge-fd 0

iface eth0 inet manual

 

[gurubert]

Yes, quite. Remove the gateway from vmbr1. It is already set for vmbr0.

 

[T0xicFX]

alright but how can i get network on my ct/vm ? cause i tried that one he got me with that error

error: vmbr1: cmd '/bin/ip route add default via 168.119.12.103 proto kernel dev vmbr1 onlink' failed: returned 2 (Error: Nexthop has invalid gateway.)

so i tried without gateway but how i make my ct and vm have network?

 

[counteradditional]

Depending on your use case, I've done similar (not a /29 but I've only used 1 IP address from Hetzner) although I didn't mess around with routing tables / translation tables.

I never wanted the host directly exposed to the internet (even though it is physically). So just used PCIe passthrough and sent the whole NIC to an opnsense VM.

config can be done if you use terminal mode (you can use a Hetzner KVM to get onto the opnsense VM), and from there you can drop into a command line and run "pfctl -d" to temporarily disable the firewall and config as needed.

For me I don't see huge amounts of bandwidth or anything so this is currently working fine for me. I use Zerotier to create site to site connections and dial in from my home network.

 

[T0xicFX]

i just want a simple thing like gurubert try to explain to me lol

 

[gurubert]

alright but how can i get network on my ct/vm ? cause i tried that one he got me with that error

error: vmbr1: cmd '/bin/ip route add default via 168.119.12.103 proto kernel dev vmbr1 onlink' failed: returned 2 (Error: Nexthop has invalid gateway.)

so i tried without gateway but how i make my ct and vm have network?

The VM on vmbr1 has an address out of 136.243.186.xxx/29 with the default gateway 136.243.186.xxx, the adress of vmbr1 of the Proxmox node. The Proxmox node has to have IP forwarding on and presumably a firewall active.

This is basic IP networking.

 

[T0xicFX]

so basically in my ct settings i put
136.243.186.xxx (a subnet ip) and 136.243.186.xxx (the first subnet ip) as gateway?

 

[gurubert]

so basically in my ct settings i put
136.243.186.xxx (a subnet ip) and 136.243.186.xxx (the first subnet ip) as gateway?

Yes, for different values of xxx.

 

[T0xicFX]

wait
do you have discord? will be better for speeds of answers and i can screenshare so you see, cause it's not working ):

 

[gurubert]

This is a forum for async community support.

If you need hands on support and are inside the EU or Switzerland you can have a look at https://www.heinlein-consulting.de/ and request a quote.