[SOLVED] 1 NIC, 2 public IPs (MAC binding) + pfSense VM HELP!

darkguy2008 · Mar 30, 2019

Hello all,

First time poster here, I've been following Proxmox for a while and now I've set up a VPS in an external provider. My current provider gives me a Supermicro server with only 1 NIC plugged in. I have no control over this as I've already made a ticket to ask my provider to plug another wire into the 2nd NIC and they said it's not possible. They do MAC binding which means that a single MAC can have/use two (or more) IP addresses at the same time.

As a matter of fact, I had this setup working in VMWare ESSXi 6.5, but sometimes it's a bit unflexible so I've decided to move. In VMWare, all I would have to do is to make two networks and two switches (1 of each is already created) - one for LAN, as the existing ones are for WAN. Then, I create my pfSense VM, set the WAN adapter's MAC address to the ESXi host's MAC address on NIC 1, and it would work fine and dandy - pfSense would get an IP from DHCP in the WAN interface. All good there.

Now, I'd like to replicate this in Proxmox and I'm having a really hard time. I've tried countless things, like setting dhcp on eno1 and setting the same details or IPs static in there and in vmbr0, I've tried to set the MAC of the pfSense interface in the WAN adapter to the same on the host and it doesn't work either. I'm honestly frustrated at how complicated something so simple could be (and is, in ESXi) so let's say I'm starting out with a blank canvas:

What should I do to archieve my goal? My idea is this:

Internet <-> MAC1=WAN IP 1 (DHCP) <-> pfSense <-> VMs
Internet <-> MAC2=WAN IP 2 (STATIC) <-> Proxmox

I had this working in ESXi, so it should be possible on Proxmox as well. The issue is... how? I've lost 2 days on this already. Some help would be great.

Edit: My provider says this generic message when I get a new IP: "We apply MAC-filter to your server. If you are going to use it as a hypervisor, please, use a bridge mode for your network interface." - Using a bridge mode sounds good for the... linux bridges? How do I archieve that?

Thanks!
-DARKGuy

TL;DR: Just use ovs (openvswitch) instead of linux bridge. Check post #20.

darkguy2008 · Mar 30, 2019

Haider Jarral · Mar 31, 2019

Is pfsense setup as a VM or physical box ?

Haider Jarral · Mar 31, 2019

Do not set any IP on vmbr0 interface and let pfsense VM use that interface in bridge mode, with DHCP pfsense should get an IP. Then you would probably need to set another interface that distributes NAT ip to your VMs.

darkguy2008 · Mar 31, 2019

Hello Haider, thanks for your reply!

Haider Jarral said:
Is pfsense setup as a VM or physical box ?

VM

Haider Jarral said:
Do not set any IP on vmbr0 interface and let pfsense VM use that interface in bridge mode, with DHCP pfsense should get an IP. Then you would probably need to set another interface that distributes NAT ip to your VMs.

Yeah, I don't have issues with the 2nd bridge, but the 1st one has been giving me issues. I'm no Linux expert, but I tried to do what you meant (I guess) and it's not working. Any ideas?

Notice the em0 MAC is the same as the server's MAC on LAN1 (LAN2 is always down). This is how I always used to set it up in ESXi, but it's not working here.

Here's my /etc/network/interfaces:

Code:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address  XX.XX.107.X61
        netmask  255.255.254.0
        gateway  XX.XX.106.1
        bridge_ports eno1
        bridge_stp off

auto vmbr0
iface vmbr0 inet dhcp
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address  192.168.100.1
        netmask  255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0

With my previous setup, ESXi (now Proxmox) would be statically assigned in the XX.XX.107.X61 address and the pfSense would take DHCP by default on the XX.XX.107.X51 address, which is what I'm aiming for.

If I'm doing it wrong let me know, I'm willing to test anything.

I also tried this with no avail either, I'm really lost...

Code:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet dhcp
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-hw 0c:c4:7a:12:25:5e

auto vmbr1
iface vmbr1 inet manual
        bridge-ports vmbr0
        bridge-stp off
        bridge-fd 0
        bridge-hw 0c:c4:7a:12:25:5e

auto vmbr2
iface vmbr2 inet static
        address  192.168.100.1
        netmask  255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0

Thanks!

Haider Jarral · Mar 31, 2019

Do not set IP on eno1, dont set ip on vmbr0, only set IP on pfsense.

darkguy2008 · Mar 31, 2019

Haider Jarral said:
Do not set IP on eno1, dont set ip on vmbr0, only set IP on pfsense.

Okay, like this? I tried to set it manually on pfSense (although I'm looking for it to use DHCP instead) and it doesn't work, no WAN access:

What else can I try?

Haider Jarral · Mar 31, 2019

Keep pfsense WAN at DHCP and see if it gets the IP.

Haider Jarral · Mar 31, 2019

Have you looked at this article ?

https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-proxmox.html

darkguy2008 · Mar 31, 2019

Haider Jarral said:
Keep pfsense WAN at DHCP and see if it gets the IP.

Just did, no workie:

Haider Jarral said:
Have you looked at this article ?

https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-proxmox.html

I did, but it requires two NICs, I only have one plugged in (don't have control over this). Other than that, the same install procedure is the one I'm following. My provider is GTHost, if that rings any bell.

Haider Jarral · Mar 31, 2019

Wait, if you have only 1 NIC, how is there any IP for LAN ?

1 Nic means you can only use that NIC in one direction either LAN or WAN. Here is what I envision would work

ISP
|
Proxmox
|
eno1-vmbr0 (NO IP)
|
Pfsense - WAN - vmbr0 (DHCP or static IP)

Pfsense at this point should get IP from ISP directly.

darkguy2008 · Mar 31, 2019

Haider Jarral said:
Wait, if you have only 1 NIC, how is there any IP for LAN ?

1 Nic means you can only use that NIC in one direction either LAN or WAN. Here is what I envision would work

ISP
|
Proxmox
|
eno1-vmbr0 (NO IP)
|
Pfsense - WAN - vmbr0 (DHCP or static IP)

Pfsense at this point should get IP from ISP directly.

The LAN IP is given by the 2nd VM NIC attached to vmbr1:

Code:

auto vmbr1
iface vmbr1 inet static
        address  192.168.100.1
        netmask  255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0

Okay, so to avoid giving an IP to eno1 and vmbr0, what do I have to put in /etc/network/interfaces?

Also, I have two IPs, ending in 151 (by DHCP) and 161 (Static). I plan to use 151 for pfSense but 161 for Proxmox so I can manage it. If I remove the IP for eno1 and vmbr0, how do I manage it? I need to reach it somehow, no?

Your idea is what I want to archieve, yes, but somehow pfSense is not getting DHCP from the WAN or even ping a host if it's specified manually.

Here's some background info on the IP addresses I have:

IP-address (by DHCP): XX.XX.107.151
Additional IP addresses: XX.XX.107.161
Default gateway: XX.XX.106.1
Subnet mask: 255.255.254.0

If I'm setting any route (watching the GIF it seems that it can't find a route?), I'd rather have it in proxmox than in pfSense because it's really messy to deal with.

darkguy2008 · Mar 31, 2019

This is the network config (visually):

Host:

VM:

/etc/network/interfaces:

Code:

root@vps01:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet dhcp
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address  192.168.100.1
        netmask  255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0
root@vps01:~#

Haider Jarral · Mar 31, 2019

ok so vmbr0 get this IP settings, you can set it via gui as well.

IP-address (by DHCP): XX.XX.107.151
Default gateway: XX.XX.106.1
Subnet mask: 255.255.254.0

The pfsense VM uses vmbr0 as bridge and you set manual IP on it

Additional IP addresses: XX.XX.107.161
Default gateway: XX.XX.106.1
Subnet mask: 255.255.254.0

After setting it, goto proxmox shell of server and ping gateway and see if server can reach it, if that works meaning your server is good,

Next from shell of pfsense ping gateway and the other IP, share the results.

darkguy2008 · Mar 31, 2019

Okay, first of all thanks for bearing with me on this issue

I applied the changes as requested, yet the result is still the same:

I also did the pings as requested. From pfSense, it seems I can't ping the gateway or the host either

Haider Jarral · Mar 31, 2019

To isolate further keep only one NIC on pfsense bridged to vmbr0.

Do you have firewall turned on by any chance on Proxmox ?

DataCentre > Firewall > Options

darkguy2008 · Mar 31, 2019

Alright!

Firewall is disabled on the datacenter, this is a clean Proxmox install. I can also ping both IPs in my tests and they answer to my server:

However, it's enabled on the node. I'll disable it:

Firewall is also disabled on the VM:

No firewall on VM's net0 either:

Initial boot on network change:

It seems it can't detect link-up:

Single WAN - same settings, nope:

And DHCP doesn't work either:

My network config and routes:

Code:

root@vps01:~# ip route
default via XX.XX.106.1 dev vmbr0 onlink
XX.XX.106.0/23 dev vmbr0 proto kernel scope link src XX.XX.107.151
192.168.100.0/24 dev vmbr1 proto kernel scope link src 192.168.100.1 linkdown

root@vps01:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=125 time=0.550 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=125 time=0.643 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1011ms
rtt min/avg/max/mdev = 0.550/0.596/0.643/0.052 ms

root@vps01:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address  XX.XX.107.151
        netmask  255.255.254.0
        gateway  XX.XX.106.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address  192.168.100.1
        netmask  255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0

darkguy2008 · Mar 31, 2019

So I fired up XCP-ng, first time I try that hypervisor... I log in with the UI app they have, create a pfSense VM, give it the same MAC address on Network 0 (WAN), leave 1 with automatic MAC address, install and... voilá, automatic DHCP, I didn't even have to touch anything after installation aside from specifying which interface is WAN and which one is LAN.

Plain bliss, why Proxmox can't be like this?

Considering this is working and is linux (?) based, what can I do to simulate this same behavior in Proxmox?

Haider Jarral · Apr 1, 2019

I have been using Proxmox and multiple public IPs on my VMs I did not have any issues.

All I did was using vmbr0 in bridge which you are essentially doing as well.

In your case it might require further troubleshooting like at pcap level using tcpdump on each interface.

I can offer a remote desktop session if you want just to see if I can help, must be something very basic.

darkguy2008 · Apr 2, 2019

Haider Jarral said:
I have been using Proxmox and multiple public IPs on my VMs I did not have any issues.

All I did was using vmbr0 in bridge which you are essentially doing as well.

In your case it might require further troubleshooting like at pcap level using tcpdump on each interface.

I can offer a remote desktop session if you want just to see if I can help, must be something very basic.

Thanks Haider, however the differences between XCP-NG and Proxmox got me thinking a lot, and I decided to compare both versions to see why it worked there and it didn't in my setup.

Lo and behold, for future readers, my solution was to use OVS (openvswitch) instead of Linux Bridge. Holy cow, how complicated Proxmox can be sometimes.

There, it's working fine and dandy:

Network config:

Caveat: Since I'm using pfSense as a firewall VM in DHCP (107.151 instead of 107.161), starting it up will disconnect the management interface (or maybe block it). I think I have to open the port 8006 in pfSense to allow it to work - which is good, anyways, extra security and works as a failover: If the pfSense VM doesn't start up the default port will be open anyways, and through pfSense I can apply port forward to use other a different port than 8006.

Thanks a lot for your help and insights, it was really helpful as well and I learned a lot!

[SOLVED] 1 NIC, 2 public IPs (MAC binding) + pfSense VM HELP!

Member

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Member

Member

Well-Known Member

Member

Well-Known Member

Member

Attachments

Member

Well-Known Member

Member