Help with networking model (concepts)

lpallard · Jun 26, 2016

Hello,

I am migrating from a virtualized pfsense router running as a KVM machine in PVE, to a hardware one, and so far, I managed to build the real one and its working fine.

Next step is to reconfigure the networking stack in PVE node. I want the following setup, but somehow all I managed to do was to render the webUI unrecheable and lock myself out of PVE... I have reseted root's password via IPMI and the procedure available in PVE's wiki page.

The setup I want to achieve:

Port eth0: physically connected to the LAN interface on the router, will serve PVE webUI, most VM's will be using this port to connect to LAN both ways (in & out), and will have IP 192.168.0.5 to access the webUI.

Port eth1: physically connected to a separate interface on the router, some VM's will be using this port to connect to a separate LAN both ways (identical as with eth0).

The reason why I have to LAN's on the router with different ports is for easier management at the firewall level.

I tried creating a bridge vmbr0 that used eth0 so my VM's could use eth0, and another bridge called vmbr1 that used eth1 so the other VM's could use eth1. All that managed to do was to render the webUI inacessible, and ssh no longer working.

Obviously I am doing something wrong. Can a single physical interface (eth0) be used simultaneously for both the webUI and for a bridge for several VM's??

Looking forweard to some pro's input here... Im lost!

A sincere thank you!

lpallard · Jun 26, 2016

OK I just managed to make the webUI accessible once again by commenting out everything in /etc/networking/interfaces except the eth0 block.

Now the file looks like:

Code:

# network interface settings
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address  192.168.0.5
        netmask  255.255.255.0
        gateway  192.168.0.1

auto eth1
iface eth1 inet manual

#auto vmbr0
#iface vmbr0 inet manual
#       bridge_ports eth0
#       bridge_stp off
#       bridge_fd 0

#auto vmbr1
#iface vmbr1 inet manual
#       bridge_ports eth1
#       bridge_stp off
#       bridge_fd 0

hodo · Jun 26, 2016

The following interface-script should work:

Code:

# network interface settings
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
        address  192.168.0.5
        netmask  255.255.255.0
        gateway  192.168.0.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
       bridge_ports eth1
       bridge_stp off
       bridge_fd 0

The UI should accessible via https://192.168.0.5:8006

hodo · Jun 27, 2016

Does it work for you? I can explain the options for you, if you need further assistance.

lpallard · Jun 27, 2016

Hello hodo

Based on your reply and the suggested content for /etc/network/interfaces, I have made a few modifications to match my user case and needs. Namely, I have added another bridge block for the second ethernet port to be used as a separate route to my router.

The file now looks like this:

Code:

# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
    address  192.168.0.5
    netmask  255.255.255.0
    gateway  192.168.0.1
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet static
    address  192.168.0.6
    netmask  255.255.255.0
    bridge_ports eth1
    bridge_stp off
    bridge_fd 0

Two issues:

1. Network communication on eth0 (vmbr0) is fine, I can SSH to the node, I can access the webUI via 192.168.0.5:8006 and all VM's assigned to vmbr0 have full LAN communication. All is good, except Proxmox itself cannot access the internet. I cannot ping google.com, and when I issue an apt-get update, I get

Code:

root@proxmox:~# apt-get update
Err http://security.debian.org wheezy/updates Release.gpg                   
  Could not resolve 'security.debian.org'
Err http://hwraid.le-vert.net wheezy Release.gpg                             
  Could not resolve 'hwraid.le-vert.net'
Err http://ftp.ca.debian.org wheezy Release.gpg                     
  Could not resolve 'ftp.ca.debian.org'
Reading package lists... Done     
W: Failed to fetch http://ftp.ca.debian.org/debian/dists/wheezy/Release.gpg  Could not resolve 'ftp.ca.debian.org'

W: Failed to fetch http://security.debian.org/dists/wheezy/updates/Release.gpg  Could not resolve 'security.debian.org'

W: Failed to fetch http://hwraid.le-vert.net/debian/dists/wheezy/Release.gpg  Could not resolve 'hwraid.le-vert.net'

W: Some index files failed to download. They have been ignored, or old ones used instead.

2. The VM I assigned to vmbr1 has no networking capability. I cannot SSH to it, and from the VM, I cannot access the DMZ, LAN, or the internet. I can however ping 192.168.0.6 (vmbr1) from LAN, but not the VM behind it.

I have attached a picture to show the current network config. Very simple, but perhaps I am doing something wrong?

hodo · Jun 27, 2016

hello lpallard,

let us solve ur problems step for step.

Your first problem seems to be related to your dns configuration. Can you give me an output for cat /etc/resolv.conf?

For your second problem: I dont understand, why you have assigned an ip on vmbr1.
Nonetheless, your Problem seems related to your pfsense configuration.

Can you ping the bridge or vm (vmx in your chart) from the pfsense box?

lpallard · Jun 27, 2016

You got it right! /etc/resolv.conf was pointing to the old virtualized router....

Code:

root@proxmox:~# cat /etc/resolv.conf
search localdomain
nameserver 192.168.0.100

Once I changed it to the new pfsense router's LAN IP, apt-0get worked! Damn, I didnt think changing network configuration would be so intensive. Can I use a hostname instead of the actual IP for the pfsense LAN port? That way if the LAN port changes (192.168.0.1), I will not have to manually change it in /etc/resolv.conf...

For the second issue, you are right. I forgot to delete the IP assignment on vmbr1. Can I make the modification on /etc/network/interfaces without rebooting the proxmox node? I have 13 VM's currently running...

I can ping the bridge from pfsense

Code:

PING 192.168.0.6 (192.168.0.6): 56 data bytes
64 bytes from 192.168.0.6: icmp_seq=0 ttl=64 time=0.419 ms
64 bytes from 192.168.0.6: icmp_seq=1 ttl=64 time=0.136 ms
64 bytes from 192.168.0.6: icmp_seq=2 ttl=64 time=0.220 ms

--- 192.168.0.6 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.136/0.258/0.419/0.119 ms

but I cannot ping the VM behind it (VM-x)

Code:

PING 192.168.1.101 (192.168.1.101): 56 data bytes

--- 192.168.1.101 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

hodo · Jun 27, 2016

hello again,

nice to hear, that i could help you!

It is possible, to configure dns names in resolv.com but not recommend. Then you have to configure the hostname for the ip in the /etc/hosts file or via dhcp as dns server or some kind of domain.

EDIT: To be precise: There is no benefit in setting hostnames in resolv.conf. When you type myhostname.localdomain your system needs to resolve this name to an IP.
In Simple Words: First your system looks in the /etc/hosts file, if there is a matching entry, it uses the ip. If there is no matching entry it asks the configured dns server configured in resolv.conf.

For your second issue:
If you can ping the 192.168.0.0 network from the dmz port you have to configure a route pointing to the 192.168.1.0 network on this interface. Then you should be able to reach the 192.168.1.0 network.
As far as i know, you should separate the dmz network with a unique network. Its "cleaner" to use a seperate network and avoids some weird pitfalls.

In debian you can restart the network stack with the following command:

Code:

systemctl restart networking.service

This reloads the contents of the /etc/network/interfaces file. I dont know, if it is safe to use this in proxmox.

The safest way is to restart the node via proxmox interface.

lpallard · Jun 28, 2016

Hello hodo,

OK I will not make resolv.conf find the gateway automatically via hostname... I will try to remember that if I change the router's IP, to modify resolv.conf.

For the other issue, I believe the problem is with pfsense. I have 3 interfaces on pfsense, em0 (WAN), em1 (LAN) and em2 (OPT1).

My goal is to somehow split my Procurve 24 ports switch into two "independent" virtual switches with pfsense's LAN port connected to the first "Virtual switch" so I can access Proxmox's WebUI via 192.168.0.5, and have pfsense's OPT1 port connected to the second "virtual switch" with Proxmox's eth1 port connected to it.

I am thinking about using VLAN's to perform this. It would be like splitting the physical switch into 2 independant switches more or less.

Attached is a diagram of what I want in the end. I managed to create 2 VLAN's on the procurve switch (VLAN1 with ports 1 to 18, and VLAN2 with the remaining ports), but I got locked out of the switch and network collapsed. Had to factory reset the switch. I believe it had something to do with management port or port tagging but I am not sure.... I have 0 experience with this. Do you have any experience or a better way to achieve what is shown on the diagram? I have posted on pfsense's forums, but the usual way is that either I dont get any replies, or the replies are usually harsh and out of topic.

My assumptions:
-VLAN's are the right feature for what I want to do
-I do NOT have to do anything on pfsense or proxmox at ALL. (only to config the switch with 2 VLAN's)
-The switch webUI will still be accessible via LAN (VLAN1) since it will reside in the same subnet (192.168.0.X). VLAN2 will have subnet 192.168.1.X

If someone has a bit of experience with VLAN's please share! I would appreciate and this may help others achieve a more robust network stack.

hodo · Jun 29, 2016

Hello lpallard,
VLAN is one way to do, what you want to do.

May i ask what for other systems are involved in your setup? Why do you wanna split your switch like this? Maybe its easier to virtualize your pfsense on the proxmox host?

I have no experience with procurve switches, but the brands we are using there are options to restrict the switchports for communication with other ports.

example: i can set for port 1, that packets are only forwarded to port nr. 1-12. this i can do for every port.

You can achieve this with vlans, too. But you have to set the vlans at your interfaces.

More Info on tagged vs untagged VLAN: http://community.hpe.com/t5/Switches-Hubs-Modems-Legacy/Tagged-vs-Untagged/td-p/4004330

lpallard · Jun 29, 2016

Hello Hodo

I am happy to provide more details on my setup. BTW, pfsense is not virtualized. It is a physical server.

I currently have several devices that I want to "isolate" from my LAN for security reasons, and easier manageability. By using separate interfaces in pfsense, I can run 2 DHCP servers for different subnets, run different snort and firewall rulesets tailored to specific uses (LAN is more secure and more restrictive, OPT1 is more relaxed). I use OPT1 for my wifi access point, my nexus phone, my ipod, my GF's phone & tablets, visitors laptops or tablets, etc. In other words, for devices that are not entirely under my control.

VM-X is actually a small virtualized FTP server that I seldomly access from the outside. If it gets compromised, by being on OPT1 I have another set of barriers to prevent someone from attacking my LAN. This is why I am using vmbr1 on the proxmox node.

OPT1 being a single port on the pfsense router, this doesnt allow me from connecting more than one physical device (either my wifi AP, or proxmox's vmbr1 port). Thats why I am looking to "split" my procurve switch into 2 switches with VLANs. I reckon this could be easily solved by using 2 physical switches, but having a nice procurve 24 port switch, I asked myself "why not?"..

I guess I went with setting up VLAN's on the switch and created 2 VLAN's on pfsense, VLAN1 on pfsense was physically connected to a port belonging to VLAN1 on the switch, and same for VLAN2... That didnt work. It was like the switch was still a single 24 port switch. Then I decided to activate tagging on the ports. This is when I lost all network connectivity and had to factory reset the switch...

I will read a bit more on the VLAN stuff at the link you provided. When you said "VLAN is one way to do" did you meant that there are other simpler ways to do so? (of course using another switch)..

hodo · Jun 29, 2016

Hello lpallard,

thank you for the explanation. With the information you provided i think the simplest solution is a VLAN separation on your switch.

I take the following assumptions:
10 Port (Port 1-10) Procurve Switch (got from your drawing)
1 LAN-Net (192.168.0.0/24)
1 OPT-Net (192.168.1.0/24)

Now you have to do the following on your switch:
Set VLAN-ID 1 on the ports you want to connect the LAN-Net devices.(incl. the port of the PFSENSE-LAN-Cable)
Set VLAN-ID 2 on the ports you want to connect the OPT-Net devices.(incl. the port of the PFSENSE-OPT-Cable)

Attention: Don`t use VLAN-ID 0, and don`t use VLAN-IDs greater 4094.

You don`t need tagged VLAn in your scenario. Tagged VLAN is used, when you need more than one vlan on 1 port.

Please give me a feedback.

lpallard · Jun 29, 2016

Hey Hodo

So I already had configured my stuff more or less like you suggested:

I created 2 VLANs at the switch level (see screenshot 1)

I set VLAN1 with ports 1-16 in this VLAN, including the port to which the cable to pfsense LAN is located (port 1). (see screenshot 2)

I set VLAN2 with ports 17-24 in this VLAN, including the port to which the cable to pfsenbse OPT1 is located (port 17). (see screenshot 3)

On screenshot 4 you can see the per-port configuratiuon if it matters. I left everything factory default.

Still, everything works perfectly on LAN, but I just cannot access clients on OPT1.

I noticed in pfsense, the OPT1 interface says "No carrier" in status. This is not good...

hodo · Jun 30, 2016

Hello lpallard.

short answer this time:

Uncheck the boxes "vlan aware enabled" on the 444 screenshot.

lpallard · Jul 2, 2016

Finally its working!

Ports 1-16 are assigned to VLAN1, ports 17-24 are assigned to VLAN2. pfsense LAN is connected to VLAN1, OPT1 to VLAN2. Clients on both VLAN's can get IP's in their respective subnets from pfsense, and after a bit of tweaking on the FW rules, traffic is being routed between subnets so as to the outside (to WAN). VM's on both vmbr0 and vmbr1 can get IP's from pfsense depending on which VLAN they are connected to.

Setting the "VLAN aware" checkbox to NO definitely helped.

Thanks Hodo for the help and input! Learnt a lot about networking and the way PVE deals with networking stack internally....

Cheers!

Search

Search

Help with networking model (concepts)

lpallard

Renowned Member

lpallard

Renowned Member

hodo

Well-Known Member

hodo

Well-Known Member

lpallard

Renowned Member

Attachments

hodo

Well-Known Member

lpallard

Renowned Member

hodo

Well-Known Member

lpallard

Renowned Member

Attachments

hodo

Well-Known Member

lpallard

Renowned Member

hodo

Well-Known Member

lpallard

Renowned Member

Attachments

hodo

Well-Known Member

lpallard

Renowned Member