Network trouble with Hetzner subnet

[silverstone]

Debug it step by step...

Run a ping from your VM to the outside world. Now check where it gets lost. On the host, sniff traffic on the bridge that your VM is connected to, to see if the correct MAC for that IP shows up or not. If it reaches the bridge, you can be sure, that the problem is on host or network-level.

tcpdump -i vmbr0 ether host xx:xx:xx:xx:xx:xx

Thank you for your reply.

I was leaning towards either a sysctl setting, but maybe it's just the Proxmox VE Firewall and their "Security Group Policies" defined on Datacenter Level, applied on VM Level, yet ONLY showing on NODE level, without ANY indication whether the traffic was DROPPED or ACCEPTED: https://forum.proxmox.com/threads/p...ups-and-impossible-to-understand-logs.147404/

I also asked on the OPNSense Forum to see if they had a clue, if that maybe was intended behavior.
https://forum.opnsense.org/index.php?topic=40585.0

Come to think of it, I recently moved some of the VM Rules towards the "Group Security Policy" in Datacenter, then "loaded" the "Group Security Policy" into the VM. I thought it was maybe a sysctl change that did that, but since then ... no logs inside the VM Firewall Logs, ONLY on NODE level.

EDIT 1: running your command I can see a ton of UDP OpenVPN Traffic (at least that !), but nothing about ping/ICMP. I now added EXPLICITELY a "DIRECT" ICMP IN/OUT Rule (not using this "Security Group") to see if that fixes it (after rebooting the VM).

EDIT 2: Rebooted the OPNSense VM, tried to ping again. NOTHING. tcpdump register lots of ICMP going TO the OPNSense VM, but if I try to pin 8.8.8.8, run tcpdump, register everything, then look for 8.8.8.8, nothing is found .

Heck I cannot even piung the hetzner gateway for some reason ....

It's like everything is reversed. INBOUND everything (or at least most stuff) works, OUTBOUND everything is broken .

EDIT 3: I tried to run brct vmbr0 setageing 0 and the System "Crashed" (I had to issue a reboot from Hetzner Robot)

Maybe I have to add "bridge-ageing 0" to /etc/network/interfaces instead ?

Nope, I'd probably be locked out of the Server if I did that. I did another attempt with brctl setageing vmbr0 0 but it would just hang ....

EDIT 4: this is getting ridicolous ... I can ping the OpenVPN Client I'm connecting from (Home) from the OPNSense VM. But I cannot ping the Gateway / DHCP Server I'm connected to, even though ... I could get an IP Address .

EDIT 5: Not sure if applicable here, but I have NOT bound the Firewall Rules to any Interfaces. According to https://forum.proxmox.com/threads/v...tgoing-internet-connection.51542/#post-239172 it might be required (although in that case it's for a Routed Setup / NAT most likely).

And Hetzner is quite special with their Network Configuration, especially on the Host:
https://docs.hetzner.com/robot/dedi...fig-debian-ubuntu/#etcnetworkinterfaces-eni-1

Proxmox VE Host has this weird 255.255.255.255 or /32 Netmask with the pointopoint Setting.

Whereas the OPNSense VM gets via DHCP the /26 Subnet IP Address ...

EDIT 6: This is the latest /etc/network/interfaces for the affected Interfaces

auto lo
iface lo inet loopback

iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
    hwaddress XX:XX:XX:XX:XX:XX
    address xxx.xxx.xxx.proxmox
    netmask 255.255.255.255
    gateway xxx.xxx.xxx.gateway
    pointopoint xxx.xxx.xxx.gateway
    bridge-ports eth0
    bridge-stp off
    bridge_waitport 0
    bridge-fd 0
    bridge-disable-mac-learning 1
    bridge-unicast-flood off
    bridge-multicast-flood off
    bridge-vlan-aware yes
    bridge-vids 2-4096
    pre-up ip addr flush dev eth0
    post-up ip addr flush dev eth0

 

[wowo]

This was all I had to do to make bridged containers with public IPs working @ hetzner:

# Physical interface
iface eth0 inet manual


# Bridge for the hosts IP and additional public floating IPs

auto vmbr0
iface vmbr0 inet static

# This is the public IP of the host
address  37.x.x.86/26
gateway  37.x.x.1
bridge-ports eth0
bridge-stp off
bridge-fd 0

After that i set the correct IP, MAC and Gateway for the NIC in the VM as seen on the Hetzner info tab for that server and it worked just fine.

Maybe remove all your additional configs/firewalls, etc. for a test. I think it should work.

 

[silverstone]

This was all I had to do to make bridged containers with public IPs working @ hetzner:

# Physical interface
iface eth0 inet manual


# Bridge for the hosts IP and additional public floating IPs

auto vmbr0
iface vmbr0 inet static

# This is the public IP of the host
address  37.x.x.86/26
gateway  37.x.x.1
bridge-ports eth0
bridge-stp off
bridge-fd 0

After that i set the correct IP and MAC for the NIC in the VM and it worked just fine.

It should be quite straight forward. But with all the MAC Abuse Emails that I got (and many other people got ... just look at e.g. Reddit) I don't know what to think .

You are however using Containers you said ? I wonder if that makes a difference (since you don't have an "extra" [the KVM] kernel on the Bridge, essentially, since Containers share the Kernel with the Host).

 

[wowo]

It should be quite straight forward. But with all the MAC Abuse Emails that I got (and many other people got ... just look at e.g. Reddit) I don't know what to think .

You are however using Containers you said ? I wonder if that makes a difference (since you don't have an "extra" [the KVM] kernel on the Bridge, essentially, since Containers share the Kernel with the Host).

Yes I am using containers. But it should be the same for KVM. It is important, that you set the correct MAC to the virtual interface attached in your VM.

You get abuse mails from Hetzner when your IP uses other MAC-adresses than the ones shown on the IP-tab in hetzner config for that IP.

It's easy to check. If you sniff the bridge your VM is attached to and you see the correct mac-address for your IP when generating traffic, everything should be fine.

 

[silverstone]

Yes I am using containers. But it should be the same for KVM. It is important, that you set the correct MAC to the virtual interface attached in your VM.

You get abuse mails from Hetzner when your IP uses other MAC-adresses than the ones shown on the IP-tab in hetzner config for that IP.

It's easy to check. If you sniff the bridge your VM is attached to and you see the correct mac-address for your IP when generating traffic, everything should be fine.

I know how you are thinking. I thought the same.

VM NIC was set to the correct MAC.

Sniffing with TCPdump showed that the "Abuse MAC" were linked to an address within that /26 Subnet, but NOT my Servers- One of my "Neighbors" so to speak.

Yet, Hetzner sent me several MAC Abuse Emails as well as, after I complained, a Human Response which was even more chilling:


After 4 days of almost restless nights I'm starting to think to cancel this Server before I even get it up and running properly.
It shouldn't be Rocket Science ...

 

[silverstone]

Maybe the only Option is to order an additional NIC and LAN (WAN) Connection fore roughly 5 EUR Extra per Month in Total, then setup PCIe Passthrough in Proxmox VE to pass the NIC directly to OPNSense.

https://docs.hetzner.com/robot/dedi...formation/root-server-hardware/#miscellaneous

Seriously, the OPNSense VM originally worked (although it generated these MAC Abuse Emails for which Hetzner threatened to cut off my Server), but I have no clue why it doesn't work anymore .

EDIT 1: The only (new ???) Stuff I see in NODE -> Firewall -> Logs or /var/log/pve-firewall.log is some

0 7 PVEFW-HOST-OUT 21/May/2024:08:34:12 +0200 policy DROP: OUT=vmbr0 SRC=XX.XXX.XXX.proxmox DST=185.12.64.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43053 DF PROTO=TCP SPT=48492 DPT=53 SEQ=3071275364 ACK=0 WINDOW=32120 SYN

But that is Hetzner DNS Server.

Maybe that's due to the "net0" Interface I recently introduced in OPNSense VM -> Firewall for all their Rules

 

[silverstone]

I didn't really make much progress.

I'm trying to play with the Proxmox Firewall, but it's really tricky, as some stuff is no logged no matter what.

I tried to "force" that behavior by adding some Catch-All "DROP" Rules for Inbound & Outbound, but it brings more questions than it answers.

It seemed *sometime* that disabling OpenVPN in the OPNSense VM helped to ping **a few times** e.g. 8.8.8.8.
Similarly it seemed *sometime* that adding a Floating Rule Pass in/out ANY ANY helped *sometimes* to be able to perform DNS Lookups from the VM. And pinging but very rarely.

I still don't get if the iptables firewall, in the way that Proxmox implemented it, is a "first match" or "last match" (or if we can/should customize that).

I assumed the Firewall to be first-match, but I also noticed that some traffic (particularly SYN Packets) get dropped even from my Management Network IP Addresses.

Any tips ?

EDIT 1: Checking "Disable Firewall" in OPNSense Firewall -> Settings seems to be the ONLY thing to let OPNSense ping the outer World ...

EDIT 2: Checking "Disable reply-to on WAN rules" in OPNSense Firewall -> Settings also works (better than Disabling the whole Firewall)

With this Setting in place, I can finally (try to) Update OPNSense. Weird ... this was like the last Setting I would have thought it would have an impact ... It stopped while trying to download the first Package

EDIT 3: and just when I thought I found the Solution, OPNSense decided to crash on me .


EDIT 4: this fortunately could be (quite easily) fixed by running pkg upgrade -f which will either reinstall the current version of a Package (if no newer Package is available) or install an updated version of the Package (in case a newer Package is available).

 

[silverstone]

I'd still say there is something weird going on with IPv6 now.

Strangely, for IPv4, I see the logs in the VM -> Firewall -> Logs-

But for IPv6, even though I added the Security Group Rule and associated it with "net0" on the VM, I see "allow-ping-in" in the Host Logs, *not* in the VM (like I did for IPv4).

Either the IP/CIDR Notation is NOT working as expected (I added the Allowed IPv6 Address in VM -> Firewall -> IP SET -> ipfilter-net0) or something else is at play here ...

Right now I have the IPv6 Target Address for the Subnet /64 in Hetzner Robot Configured to the OPNSense VM.

Thus the Host does NOT have any IPv6 Addresses (well, besides link-local fe80:xxxx:etc), which all vmbr* Bridges indeed do have.