Default network config with eth0/vmbr0 on LAN, can't get eth1 up with public static.

[garymaurizi]

Hello, I have a default proxmox install going on a supermicro xeon rack server, it has two onboard NIC's, right now the default configuration has eth0 active and bridged to vmbr0 with a local LAN ip address (this physically plugs into another rackmount server that acts as a linux iptables router/firewall providing dhcpd). I can access the proxmox webUI using the private rfc1918 LAN ip address.

By default eth1 exists but is not autostarted or active. eth1/second onboard NIC is physically plugged straight into my cable modem which provides public static ip addresses to its clients, but does not have a dhcp server running.

I need to bring eth1 live with a public static IP address so that I can access the proxmox webUI over the internet, and be able to assign containers/VM's public static IP addresses from the modem through eth1.

I have tried to manually edit /etc/network/interfaces and make eth1 active with a public static ip address, and I have also tried leaving eth1 autostarted with manual, and making a bridge for it with vmbr1 bridge ports eth1, I have also tried doing this in the proxmox webUI, in all cases the local network can reach the webUI and ping over the assigned public static, but I can not ping it from outside, or reach the server in any way with the public static assigned to eth1 or vmbr1.

I figured it must be a routing problem, so I tried changing my default route from a LAN default route to the modems gateway and this worked and made the proxmox webUI reachable over the internet, however, it was not consistent, and changed everytime networking was restarted, when I tried to make this change persistent by putting "up route add default gw 25.44.145.184 vmbr1" at the end of /etc/network/interfaces, even though it looks like the route is added when I do 'service networking restart' it is not pingable, this routing fix only works when I take that line out at the end of /etc/network/interfaces and issue the route command by hand after networking has been restarted?

Can anyone help with this? what exactly is going on here? I just want eth0 attached to my LAN/linux router/firewall and eth1 attached to my cable modem so that I can access the webUI and give some CT's/VM's public IP's? please help?

Thank you so very much for your time,
GM.


For informational purposes here is an altered example of the /etc/network/interfaces file:

-------------------root@server:~# cat /etc/network/interfaces-------------------------------
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.0.0.3
netmask 255.255.0.0
gateway 10.0.0.1
bridge_ports eth0
bridge_stp off
bridge_fd 0

auto vmbr1
iface vmbr1 inet static
address 32.33.144.196
netmask 255.255.255.248
bridge_ports eth1
bridge_stp off
bridge_fd 0

#default route fix/alteration -- does not work here, but works when issued by hand
up route add -net 32.33.144.192 netmask 255.255.255.248 gateway 32.33.144.193 dev vmbr1
up route add default gw 32.33.144.193 vmbr1
---------------------------------------------------------------------------------------------------

 

[spirit]

Re: Default network config with eth0/vmbr0 on LAN, can't get eth1 up with public stat

maybe

[COLOR=#333333][I]auto vmbr1[/I][/COLOR]
[COLOR=#333333][I]iface vmbr1 inet static[/I][/COLOR]
[COLOR=#333333][I]address 32.33.144.196[/I][/COLOR]
[COLOR=#333333][I]netmask 255.255.255.248[/I][/COLOR]
[COLOR=#333333][I]bridge_ports eth1[/I][/COLOR]
[COLOR=#333333][I]bridge_stp off[/I][/COLOR]
[COLOR=#333333][I]bridge_fd 0[/I][/COLOR]
post-up [COLOR=#333333][I]route add -net 32.33.144.192 netmask 255.255.255.248 gateway 32.33.144.193 dev vmbr1[/I][/COLOR]
[COLOR=#333333][I]post-up route add default gw 32.33.144.193 vmbr1[/I][/COLOR]

?

 

[garymaurizi]

Re: Default network config with eth0/vmbr0 on LAN, can't get eth1 up with public stat

maybe

[COLOR=#333333][I]auto vmbr1[/I][/COLOR]
[COLOR=#333333][I]iface vmbr1 inet static[/I][/COLOR]
[COLOR=#333333][I]address 32.33.144.196[/I][/COLOR]
[COLOR=#333333][I]netmask 255.255.255.248[/I][/COLOR]
[COLOR=#333333][I]bridge_ports eth1[/I][/COLOR]
[COLOR=#333333][I]bridge_stp off[/I][/COLOR]
[COLOR=#333333][I]bridge_fd 0[/I][/COLOR]
post-up [COLOR=#333333][I]route add -net 32.33.144.192 netmask 255.255.255.248 gateway 32.33.144.193 dev vmbr1[/I][/COLOR]
[COLOR=#333333][I]post-up route add default gw 32.33.144.193 vmbr1[/I][/COLOR]

?

This has now been tried, and it works, but breaks any container/VM's ability to reach the internet(they can still reach 10.0.0.X LAN).

I did exactly what you said, but the second I change the hardware nodes default route out, the vmbr1 public static becomes ping-able, but all containers lose internet access(but still continue to have LAN access.)

I reset all the routes and interfaces back to the way the proxmox installation set them up(a complete re-install actually), and then I tried simply adding eth1 with my public static IP, and pinging the public static -- this did NOT work, I then ran tcpdump on eth1 to try to diagnose the issue and discovered that the public static ip on eth1 is getting the icmp echo requests in this setup, but the host that is pinging is not getting replies back from that public static ip so the ping fails on the external hosts end?!? this has got to be a relatively common proxmox setup, I just want my local lan to have access to private containers, and still be able to create public ones that local clients can access without internet traversal...

My guess is the icmp requests are coming in on eth1 public static and leaving out through eth0/vmbr0 as that is the default gateway? I know about policy based routing and know how to set this up, so please do not reply with that as a requirement. I should not have to use that in this instance and have solved this exact problem in the past on ym centOS vpn gateway/firewall/router without it, I should be able to have 10.0.0.1 remain the default gateway, and add an additional NOT 'default' gateway for eth1, I believe?

Any help is highly appreciated, I have tried just about everything I can think to try at this point. If it helps get the ball rolling - This is a professional installation in a medical facility, we are evaluating proxmox and considering a support contract. I am the IT administrator, and gravitate towards open source solutions when ever possible.

Signed,
GM

 

[garymaurizi]

Re: Default network config with eth0/vmbr0 on LAN, can't get eth1 up with public stat

I have also tried deleting eth1 entirely, so proxmox is set with the default networking eth0 started and blank with vmbr0 as 10.0.0.3, routing shows the default gateway as 10.0.0.1 which is my CentOS router/firewall that connects to the cable modem.

I then try to create a VM/container with a public static IP address and this does not work either as I anticipated, as their are no routes for it.

I'm at a complete. loss.

 

[screenie]

Re: Default network config with eth0/vmbr0 on LAN, can't get eth1 up with public stat

I don't think you will get it to work this way;
If i understand it correctly you have a public subnet from your ISP which you want to assign to your VM's, protected by your firewall;

Three possible solutions:

1. VM's can have private ip's but traffic from/to internet should come/go to their own public ip:
Firewall has a public static ip - forward your remaining public ip's per static nat to private ip's of your VM's;

2. VM's need to have the public ip configured directly;
Ask your ISP for a /30 network where your connect the cable modem and external interface of your firewall together;
The ISP should route an additional public subnet to the public ip of your firewall, configure one internal interface of the firewall with the public ip, connect this interface to one nic on your PVE host and bridge it to the VM's with firewall internal ip as default gateway;
A second interface to an additional nic of your PVE host with private ip for management - do port forwarding from firewall external public ip;
(you can do that also with secondary ip's if you do not have enough interfaces);

3. The ISP won't assign you a /30 and route you another subnet or you don't want to work with private ip's
You need to configure a bridged firewall - no routing, all traffic is L2 based from PVE host and VM's from/to Internet
You mentioned to have your firewall based on CentOS - first result from Google: http://blog.connexeon.com/sysadmin/linux/install-a-bridged-firewall-iptables-on-centos/

hth,
Alex