One NIC 2 IPs - OPNsense on additional IP

[crankshaft]

Been working on this the whole day and unable to get it to work.

This is the recommended setup where you have a single NIC and 2 Ip addresses using routed configuration.

This is on a bare-metal machine hosted by a data centre.

I need to isolate the proxmox web management console (enp1s0f0np0) from the guests such that the guest public IP cannot access the host.

The intention is for vmbr0 to route to a opnsense guest on vmbr0 and then it will manage routing to the other guests.

https://pve.proxmox.com/wiki/Network_Configuration

auto lo
iface lo inet loopback

auto enp1s0f0np0
iface enp1s0f0np0 inet static
    address 1XX.235.225.170/32
    gateway 1XX.64.0.1
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv4/conf/eno0/proxy_arp

auto vmbr0
iface vmbr0 inet static
  address  1XX.99.84.229/32
  bridge-ports none
  bridge-stp off
  bridge-fd 0

While I can ping on the main interface enp1s0f0np0, I cannot ping using vmbr0:

# ping -I enp1s0f0np0 8.8.8.8                                                                                                                         
PING 8.8.8.8 (8.8.8.8) from xxx.235.225.170 enp1s0f0np0: 56(84) bytes of data.                                                                                              
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=0.858 ms                                                                                                                     
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=0.856 ms                                                                                                                     
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=0.852 ms                                                                                                                     

ping -I vmbr0 8.8.8.8                                                                                                                               
PING 8.8.8.8 (8.8.8.8) from xxx.99.84.229 vmbr0: 56(84) bytes of data.
>> No Reply

I have also added the routes:

# ip route                                                                                                                                            
default via xxx.64.0.1 dev enp1s0f0np0 proto kernel onlink                                                                                                                 
10.0.0.0/24 dev mynet proto kernel scope link src 10.0.0.1                                                                                                                
xxx.64.0.1 dev vmbr0 scope link                                                                                                                                            
192.168.100.0/24 dev xnet proto kernel scope link src 192.168.100.1 
`

I have also had lots of suggestions from chatGPT and googled lots of threads and gitub examples and tried many of them, but none succeed in routing vmbr0 to the main gateway.

Any help / suggestions much appreciated

Thanks

 

[crankshaft]

also tried adding pointopoint: 1XX.64.0.1 to vmbr0 as read elsewhere but that did not help.

 

[weehooey-bh]

Hello

I am going to assume the following:

• You are using OVHcloud (it looks like it is based on the information you provided).
• The IP address for your server is 1XX.235.225.170/32
• The IP address for your OPNsense instance in PVE is 1XX.99.84.229/32
• Your OPNsense IP address is an OVH "Additional IP"

Here is a reworked version of your interfaces file. Look it over before using it; it is based on the information you provided, but it still may not be right. You need to validate it.

auto lo
iface lo inet loopback

up echo "1" > /proc/sys/net/ipv4/ip_forward
up echo "1" > /proc/sys/net/ipv4/conf/enp1s0f0np0/proxy_arp
# Note: you were applying this to interface "eno0" and
# you do not have an "eno0". Also, not sure you want these to
# run "post-up", so I changed to "up"

auto enp1s0f0np0
iface enp1s0f0np0 inet manual
# Moved the server IP address to vmbr0

auto vmbr0
iface vmbr0 inet static
    address 1XX.235.225.170/32
    gateway 1XX.64.0.1
    bridge-ports enp1s0f0np0 # You had "none" so traffic would never reach this bridge.
    bridge-stp off
    bridge-fd 0
# Host bridge and OPNsense WAN interface

post-up /sbin/ip route add 1XX.99.84.229/32 dev vmbr0
pre-down /sbin/ip route del 1XX.99.84.229/32 dev vmbr0
# You do not want your server answering traffic for 1XX.99.84.229/32 so
# do not put it on any of the interfaces or bridges here.

auto vmbr1
iface vmbr1 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
# OPNsense LAN interface

OPNsense Configuration

• WAN interface
  • On vmbr0
  • 1XX.99.84.229/32
  • Upstream gateway 1XX.64.0.1 You may need to look this up with OVH and the additional IP. I have found that sometimes, the VMs will need a gateway different from the host.
• LAN interface
  • vmbr1
  • You assign the IP addresses and VLANs
• In PVE, create an SDN VLAN Zone that uses vmbr1
• The guests will use the VLAN Vnets

Packet flow (note, it is helpful to think of the bridges as network switches).

• Host traffic
  • Traffic for your PVE host will arrive on enp1s0f0np0
  • enp1s0f0np0 is a bridge-port on vmbr0 so the packets will flow to vmbr0
  • vmbr0 has an IP address, which means the PVE host has access to that traffic as that IP address (1XX.235.225.170/32)
• Guest traffic
  • Traffic for your guests will arrive on enp1s0f0np0 destined for 1XX.99.84.229/32
  • The host networking has a route for it and forwards it to vmbr0
  • Your OPNsense instance will have its WAN interface on vmbr0 and will respond to traffic for 1XX.99.84.229/32

I hope this helps you get setup.

 

[crankshaft]

Thank you so much.

I am indeed on a new bare-metal server on OVH and I have genuinely been working on this since I woke up this morning until now, almost midnight, and been around in endless circles with chatGPT reminding it to stop asking questions I have already answered and to stop repeating things I already told it don't work

The OVH control console is not very informative, it shows I have an additional IP but does not show the subnet/mask or gateway (as you mentioned) and so I have to guess or open a ticket.

I will try your solution in the morning, and I do like the way that you have commented each change and explained why you did it.

And your thought process at the end where you explain the traffic flow is most helpful in understanding how I can approach this again in the future.

Thank you so much again, I will try this first thing tomorrow and feedback on the results.

 

[andrew-transparent]

I feel your pain, crankshaft, the OVH documentation can be lacking and and difficult to find the right answer. I struggled with this as well.

Adding to weehooey-bh excellent advise there are a few ways to configure this depending on if you're using an 'Additional IP' ( their documentation sometimes references this as a 'Faiover IP' - what they used to call it), or if you're routing an IP block (either to the dedicated machine or into the vRack). Routing an IP block is more straightforward and uses standard networking practices than using 'Additional IPs' and virtual macs. From their documentation here's how you can determine your gateway:

Your gateway IP address will be determined independently of your additional IP block. At OVHcloud, we reserve the last available IP in each /24 subnet (not including the broadcast address) for a gateway address. Additionally, all Dedicated Servers are given an IP on a /24 subnet. This means a Dedicated Server's gateway will always be the first three octets of its IP followed by .254. For example, if your Dedicated Server's IP was 203.0.113.34, its gateway would be 203.0.113.254. Since additional IPs are routed through the host, any VM or other service using the additional IP would also use this as the gateway address.
https://support.us.ovhcloud.com/hc/en-us/articles/360011668000-Getting-Started-with-Additional-IPs

In a nutshell if you look at your dedicated server's configuration page in the control panel it will show what its default gateway is. Use that as your gateway for your Additional IP, depsite it being outside of your Additional IPs subnet (I'm not aware of any other data center that does this, its an OVH thing).

To add to this Pfsense and OPNSense by default do not like having a gateway address outside of an IPs subnet the way OVH handles gateways for single IPs. There is an option in the settings to "use non-local gateway address" that needs to be enabled in this case. In OPNSense they call that option 'Far Gateway' that needs to enabled. Again, if you're using IP blocks you will be assigned a GW inside the block and don't need this option.

One potential 'gotcha' to look out for is if you move that Additional IP to another dedicated server down the road, technically you will have a different default gateway which is the default gateway of the new server you assigned it to. If you don't change that in your configuation it will work fine as long as you maitain that dedicated server and its respective gateway in your account. At somepoint your gateway can stop working if you don't keep that up-to-date with the respective dedicated/virutal mac you've assigned it (we learned that the hard way). To avoid all this we now just use IP blocks and route them into our vRack and have our Firewalls assigned IPs/gateways in the block and can move them around as we see fit. Routing handles all of that without having to worry about special settings for 'far gateways'.

Good luck!

PS as weehooey-bh shows, /32 is your subnet for a single IP.

 

[crankshaft]

@andrew-transparent
==================
Thanks for that info, been looking at OVH for months, the Black Friday deals pushed me into setting up a test / evaluation server.

Not impressed with their support though, the chatbot only states the obvious, and when I attempt to chat with a human, it says we are very busy now (canned message) and after 30 minutes times out and just says submit a ticket, and deletes your chat history.

Thanks for the recommendation regarding Opnsense, the problem I have is the default install has ssh disabled and as this is on the local proxmox nets it's not accessible yet, except for via the serial console and that is quite difficult to do anything productive with.

So instead I am using a local install and will configure it and backup and push it back to OVH.

I have enabled the far gateway as you suggested but have yet to verify that it works.

@weehooey-bh
============
Thanks, your config brought up all the interfaces without any modifications, but I have yet to verify that opnsense will connect on vmbr0.

Once that works, I then need to figure out how to only allow access to it's web console from the primary IP address, but I believe that will be managed using iptables, not even attempted that yet

 

[crankshaft]

OK, the interfaces are up and OPNsense is installed with WAN > vmbr0 and LAN > vmbr1.

I have also setup and assigned interfaces, and set the IP and gateway for OPNsense, and also used the far gateway to allow it to use a gateway with a different subnet.

But it wont connect, and I cant really debug anything or make changes as I cannot access the web console as OPNsense is completely contained in the pve environment.

How would I go about setting up a route or rule that will allow me to access OPNsense either via it's wan or lan connection ?

EDIT

guests on the vmbr1 interface can access opnsense and ping, ssh etc to it, but I cannot figure out how to allow myself to access it via ssh and http via the primary ip on vmbr0.

Tried this but it does not work:

iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 192.168.100.1:80
iptables -t nat -A POSTROUTING -o vmbr1 -j MASQUERADE

 

[crankshaft]

I now think that the cause may be that the mac address used to send packets from vmbr0 from host and from opnsense are different and ovh are rejecting them:

tcpdump -i vmbr0 -e arp or icmp | awk '{print $1, $2}'
12:00:35.193305 xx:6b:00:66:eb:xx
12:00:35.507900 xx:ed:de:ad:be:xx

i have already tried setting the vmbr0 to the mac address in the original config and the opnsense mac using the console but it does not make any difference:

auto vmbr0
iface vmbr0 inet static
    address 1XX.235.225.170/32
    gateway 1XX.64.0.1
    bridge-ports enp1s0f0np0 # You had "none" so traffic would never reach this bridge.
    bridge-stp off
    bridge-fd 0
    hwaddress XX:6B:00:66:EB:XX

ifconfig vtnet1 down
ifconfig vtnet1 ether xx:6b:00:66:eb:xx
ifconfig vtnet1 up

I cannot access the web console so cannot verify if it is correctly being forced to use the same mac

does OVH reject packets that do not originate from the given mac address of the single NIC ?

 

[andrew-transparent]

OVH support is abysmal for any type of 'how do I..." or "this isn't working as expected" questions. However, if you have physical issues with your dedicated server (bad drive, power supply, loss of connectivity, bandwidth/latency issues etc) they are very quick and responsive. Their pricing reflects this. Right or wrong there is an expectation on their end that your just supposed to 'know' how all of this works and their idiosyncrasies.

As far as rejecting packets from an IP that doesn't match the assigned MAC - yes, I'm almost positive they do and I forgot about that part. You need to assign the MAC OVH gave you to the respective interface you want to use on OPNSEnse to enable MAC spoofing. Also ensure PVE allows that guest to do that as well. It's called MAC Filtering in PVE and you need to make sure its disabled. I can't remember its default state off-hand.

MAC filter under Datacenter -> MyNode -> MyVM -> Firewall -> Options

As far as accessing any of the web gui panels on PVE or OPNSense (or anything else you add downt the road) I highly recommend you block everything WAN and use some sort of VPN, Wireguard/Tailscale/Cloudflare to access any of that from your managment LAN subnet.

We have no public IPs assigned to any of our PVE hosts. Everything sits behind OPNSense and all ports are closed. We have a small container (or VM doesn't matter) to run a cloudflared daemon that connects to our Cloudflare tenant and anybody that needs access has a Warp client and has been authenticated at the user/device level. From there we restrict traffic to just GUI ports and SSH. You can accomplish the same with Tailscales and that seeems to be a very popular choice.

 

[crankshaft]

Thanks, spent another 1.5 days on this still not working.

Am using zerotier and can access the pve console, but 8006 and 22 are only open to my source IP.
I dont really understand your setup, you mean that you have one baremetal running opnsense and another running PVE ?

 

[andrew-transparent]

No, we have clusters setup and all of our opensense are vm's with various VLANs running behind them. No dedicated hardware for opensense. But the concept is the same when we only had one server.

Essentially I would have vmbr0 be setup with no VLAN tagging and opnsense's WAN interface would be on vmbr0. The opnsense WAN interface would mac spoof the OVH virtual mac address and have the additional IP on that interface (with the 'far gateway' config). The PVE host would not have any ip addressing on vmbr0. In fact we never use the actual IP of the dedicated host that OVH assigns anywhere, its basically ignored in our setup.

Then I'd setup vmbr10 which is tagged as VLAN 10. This would be the management (or LAN) interface. Opnsese would have another adatper here called LAN, and it would have an internal IP range such as 192.168.100.1/24. Our PVE host would have IP addressing on this bridge, eg 192.168.100.10/24 with GW set to 192.168.100.1

Now our PVE host can get out to the internet via opnsense and the only WAN ip exposed is the Addtional IP on the WAN interface of opnsense. Of course you're blocked to the PVE admin internface unless you have an inbound/NAT rule on your opnsense to allow your source IP -> 8006 -> 192.168.100.10.

Once that is established I'd create my cloudflared tunnel into the 192.168.100.0/24 network and once I can confirm I can connect via SSH / webgui to PVE through the tunnel I remove the inbound rule for 8006 on opnsense and not have any ports open at all.

Naturally there is an order of operations needed to ensure you don't get locked out, otherwise you're using IPMI access to the console to unstuck yourself. You can leave that dedicated public IP for your PVE host on the vmbr0 until you get all of the above sorted and tested, then remove it at the very end.

 

[crankshaft]

unfortunately another day of unsuccessful attempts.

I submitted support log 8 hours ago, no reply, but not surprised.

I have the same config that @weehooey-bh suggested, but OPNsense refuses to connect.

In OPNsense I configured as per the screenshots, but when I try and ping a public IP, I just get no response or no route to host.

Dont suppose you can have a look / compare with yours ?

Will see what OVH says, when they eventually respond.

Also just noticed that I am getting these messages:

[92879.481497] vmbr0: received packet on tap1000i1 with own address as source address (addr:xx:6b:00:66:eb:xx, vlan:0)

# ip route
default via XXX.64.0.1 dev vmbr0 proto kernel onlink
10.0.1.0/24 dev public proto kernel scope link src 10.0.1.1
XXX.99.84.229 dev vmbr0 scope link
XXX.24.0.0/16 dev vmbr1 proto kernel scope link src XXX.24.212.177
192.168.50.0/24 dev public proto kernel scope link src 192.168.50.1
192.168.50.0/24 dev vnet1 proto kernel scope link src 192.168.50.1

also no reply when pinging the public address externally. firewall is not blocking.

 

[weehooey-bh]

@andrew-transparent good stuff!

@crankshaft We use pfSense, not OPNsense. It's similar but not exactly the same.

OVH has published new (updated?) instructions which may be helpful to you:

https://help.ovhcloud.com/csm/en-ca...?id=kb_article_view&sysparm_article=KB0043904

These differ from what @andrew-transparent and we have set up, but they might resolve your issue.

Some notes:

• There are instructions for High Grade, Scale and Advance. They are substantially the same.
• The server's IP address is hung on the bond (Scale) or the actual interface (Advance).
• vmbr0
  • Assign a private IP address from a private subnet you assign to it (you pick this). In their example. this is 192.168.0.0/24.
  • NO bridge ports
• The bridge's IP address is a gateway for the VMs (for you, this would be your OPNsense's WAN gateway).
• In this private subnet, each VM would get an IP address. In your case, only your OPNsense would get an IP address.
• Each VM would also get assigned a public IP address (i.e. the OVH Additional IP). In your case, you would assign the public IP address as a VIP on the WAN interface. You will need to configure outgoing NAT for all traffic on the WAN to use the public IP address.

Let us know how this works for you.

 

[andrew-transparent]

I'll have a look again when I have some time today/tomorrow and compare. Those new instructions that weehooey-bh provided are probably the best thing to work from given the issues you're having. We have a different setup so our configs may not be the best fit for what you're trying to do (all of our links are bonded into the vRack and our IP blocks are routed into that - no mac spoofing involved).

the error:
vmbr0: received packet on tap1000i1 with own address as source address (addr:xx:6b:00:66:eb:xx, vlan:0)

Is most certainly giving you grief. There is some sort of loop or your using the same MAC address twice somewhere in your setup (either on the host or opensense).