[SOLVED] Help with network setup (1 public IP, 2 bridges)

S

Stokkes

Member
Nov 16, 2020
5
3
8
45
Hello,

Hoping some people with more experience with Proxmox than me can assist. I've had this setup working for years in ESXi, but I'm not sure how to recreate it in Proxmox.

Setup:

  • 1 Public IP (for testing, let's call this 192.168.1.165)
  • I want to assign that public IP to a pfSense VM running on Proxmox
  • I do NOT want Proxmox webUI available on the internet, nor do I want it to have a routable IP on the internet
  • I want Proxmox WebUI to be available within the pfSense LAN IP Range (let's call it 10.0.0.0/16)
  • OpenVPN Server is running on pfSense, which is what I use to "connect" to the pfSense network, where I would also want to access the Proxmox UI on https://10.0.0.250:8006/
I've attached a visual diagram of what I'm talking about. With ESXi, I'd simply have 2 vswitches, each vSwitch would have a VMKernel port. For the vSwitch that is connected to the WAN port on the server, I'd set a bogus IP (999.999.999.999) and also have a VMnetwork port that only had pfSense which was assigned my WAN IP (let's say 192.168.1.165 in this example).

I've started with this in /etc/network/interfaces, which DOES work, but Proxmox can't be reached on 10.0.0.250.

Code: 
auto lo
iface lo inet loopback

iface enp4s0 inet manual
iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 999.999.999.999/24
    bridge-ports enp4s0
    bridge-stp off
    bridge-fd0
#WAN

auto vmbr1
iface vmbr1 inet static
    address 10.0.0.250/24
    gateway 10.0.0.1
    bridge-ports none
    bridge-stp off
    bridge-fs 0
#LAN

Hope someone can help.

Thanks!
 

Attachments

  • Screenshot 2020-12-20 18.44.41@2x.png
    Screenshot 2020-12-20 18.44.41@2x.png
    199.4 KB · Views: 64
First of, address 999.999.999.999/24 is not a valid IP address. Just don't configure it and remove that line.

Can you reach the GUI from one of the VMs or the pfSense in the 10.0.0.0/24 network? nc -zv 10.0.0.250 8006 will tell you if the port is open.

I hope that you do have another way to access the PVE node besides the VPN to the pfSense. If the pfSense is having any problems, for example not starting, you will have locked yourself out.
 
aaron said:
First of, address 999.999.999.999/24 is not a valid IP address. Just don't configure it and remove that line.

Can you reach the GUI from one of the VMs or the pfSense in the 10.0.0.0/24 network? nc -zv 10.0.0.250 8006 will tell you if the port is open.

I hope that you do have another way to access the PVE node besides the VPN to the pfSense. If the pfSense is having any problems, for example not starting, you will have locked yourself out.
Click to expand...
I think the address is just for the post... so we do not know the real one ;)
 
aaron said:
First of, address 999.999.999.999/24 is not a valid IP address. Just don't configure it and remove that line.

Can you reach the GUI from one of the VMs or the pfSense in the 10.0.0.0/24 network? nc -zv 10.0.0.250 8006 will tell you if the port is open.

I hope that you do have another way to access the PVE node besides the VPN to the pfSense. If the pfSense is having any problems, for example not starting, you will have locked yourself out.
Click to expand...

Yes I am aware 999.999 isn't valid, I actually have this commented out (as you suggested).

So here's the weird problem (and I've seen other threads on this forum with the same thing, but no real solution):

When I'm connected to the pfSense OpenVPN (it should be noted I have a static entry for Proxmox in the DHCP assignment):

  1. SSHing into any other VM on 10.0.0.0 works
  2. SSHing into Proxmox on 10.0.0.250 does not work, nor does the WebUI work
  3. SSHing from a VM on the 10 network to Proxmox works
  4. WebUI from a VM on the 10 network to Proxmox works
  5. Pinging the Proxmox host from pfSense doesn't work and results in 100% packet loss
Now if I go directly on the Proxmox host (local KVM)
  1. Pinging the pfSense gateway (10.0.0.1) doesn't work
  2. Pinging any other VM on the 10 network doesn't work
Doing an
Code: 
ip route show
shows this:
Code: 
default via 10.0.0.1 dev vmbr1 proto kernel onlink
10.0.0.0/24 dev vmbr1 proto kernel scope link src 10.0.0.250

This is what makes me think it's a Proxmox issue, because it really looks like the Proxmox host isn't connected to the pfSense LAN and the only reason the VMs can connect to it are because there's a static entry the ARP table attached to the MAC address of the vmbr1

Hope some of this helps shed a bit more light.. I'm really scratching my head on this one.
 
Here's the result of nc -zv 10.0.0.250 8006 from 4 places:

  1. From pfSense: nc: connect to 10.0.0.250 port 8006 (tcp) failed: Operation timed out
  2. From Proxmox host itself: proxmox.localdomain [10.0.0.250] 8006 (?) open
  3. From my computer connected to the OpenVPN server: nc: connect to 10.0.0.250 port 8006 (tcp) failed: Operation timed out
  4. From a VM on the 10 network: proxmox.localdomain [10.0.0.250] 8006 (?) open
 
Okay, so to sum it up. pfSense (vpn) <-> PVE does not work.

other VMs -> PVE works
Stokkes said:
Pinging any other VM on the 10 network doesn't work
Click to expand...
Any firewalls running inside the VMs that would block a ping? Can you ssh from the PVE node into one of the other VMs? Because it seems that the connection there is possible and since the VMs are also using vmbr1 that traffic is not passing through the pfSense but directly to the PVE node.


For the pfSense related stuff, including the VPN, this seems like some kind of routing issue and I would investigate on the pfSense side.
 
aaron said:
Okay, so to sum it up. pfSense (vpn) <-> PVE does not work.

other VMs -> PVE works

Any firewalls running inside the VMs that would block a ping? Can you ssh from the PVE node into one of the other VMs? Because it seems that the connection there is possible and since the VMs are also using vmbr1 that traffic is not passing through the pfSense but directly to the PVE node.


For the pfSense related stuff, including the VPN, this seems like some kind of routing issue and I would investigate on the pfSense side.
Click to expand...

Actually I can't hit ANYTHING on 10.x network from the PVE at all. I can't ping the pfSense gateway, I can't ping a VM, nothing.

Which is why I don't think it's a pfSense issue.
 
But you can get a connection from the other VMs to the PVE node right?
Can you reach anything on the internet from the PVE node?

Can you post the output of ip a on the PVE node and one of the other VMs?
 
aaron said:
But you can get a connection from the other VMs to the PVE node right?
Can you reach anything on the internet from the PVE node?

Can you post the output of ip a on the PVE node and one of the other VMs?
Click to expand...

I think I have figured it out, thanks to running the ip -a. It looks like the vmbr1 used to have a MAC address of X, which I had assigned to the pfSense DHCP server to provide a static IP of 10.0.0.250

However, after a recent update of PVE, the vmbr1 MAC address changed to Y.

Removing the entry from the ARP table in pfSense, restarting the network ifup -a and I can now ping the pfSense gateway.

So to prevent it, I've now forced a bridge_hw xx:xx:xx:xx:xx:xx for each of the vmbr's to ensure they never change.

Wow frustrating!

For people who see this in the future and want a similar setup, I have created 2 interfaces files:

  1. Normal access (no internet to the PVE): interfaces.local
  2. Internet access to the PVE, in case of emergency: interfaces.inet
If there is an emergency, I just do a cp /etc/network/interfaces.inet /etc/network/interfaces && ifup -a, fix whatever is wrong, then restore via cp /etc/network/interfaces.local /etc/network/interfaces && ifup -a

/etc/network/interfaces.local
Code: 
auto lo

iface lo inet loopback
iface enp4s0 inet manual
iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
#    address 192.168.1.160/24 # ISP IP
#    gateway 192.168.1.1 # ISP Gateway
    bridge-ports enp4s0
    bridge-stp off
    bridge-fd 0
    bridge_hw xx:xx:xx:xx:xx
#WAN

auto vmbr1
iface vmbr1 inet static
    address 10.0.0.250/24
    gateway 10.0.0.1
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    bridge_hw yy:yy:yy:yy:yy:yy
#LAN

/etc/network/interfaces.inet
Code: 
auto lo

iface lo inet loopback
iface enp4s0 inet manual
iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.160/24 # ISP IP
    gateway 192.168.1.1 # ISP Gateway
    bridge-ports enp4s0
    bridge-stp off
    bridge-fd 0
    bridge_hw xx:xx:xx:xx:xx
#WAN

auto vmbr1
iface vmbr1 inet static
    address 10.0.0.250/24
#    gateway 10.0.0.1
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    bridge_hw yy:yy:yy:yy:yy:yy
#LAN
 
  • Like
Reactions: myzamri, manusysadmin and aaron
Ah, old ARP tables... those are seldom a problem and thus easily overseen as a possible cause...

Thanks for sharing this!

Please consider adding the "Solved" prefix next time by editing the first post and selecting it from the drop down next to the title. I already did it for this thread. Thanks :)
 
  • Like
Reactions: Stokkes
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back