Advice on network layout for home router and file server on 4 NIC PC

[Mencius]

Hi everyone

Sorry if my questions or terminology are a bit basic, I am starting from a low and totally self-taught knowledge base.

For fun and education I am trying different home router and server setups on a small 4 NIC embedded PC. For now I have proxmoxve hosting a pfsense VM (firewall/router) and a turnkey linux file server in a container. This works great but after I muddled my way through setup I read some blogs/comments suggesting it is unwise to have a virtualised router in the same network space as the host's management IP. This is my simple network setup:

auto lo
iface lo inet loopback

iface enp1s0 inet manual

iface enp2s0 inet manual

iface enp3s0 inet manual

iface enp4s0 inet manual

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
#pfsense WAN marked 1 on case

auto vmbr2
iface vmbr2 inet static
        address 192.168.1.100/24
        gateway 192.168.1.1
        bridge-ports enp2s0 enp3s0 enp4s0
        bridge-stp off
        bridge-fd 0
#Software switch for LAN - all ports marked 2 to 4 on case. Also handles proxmoxve management, pfsense management, file server management

The comments describe what is going on. vmbr1 is assigned to net0 which is the WAN interface in the pfsense VM and vmbr2 is net1 as LAN in pfsense. 192.168.1.1 is the address of the pfsense router and 192.168.1.100 is a static address outside the range that pfsense hands out on the LAN.

What I am trying to understand is whether I should remove the proxmox management IP from vmbr2 and separate it somehow. Is this better security practice? Would I use a vlan to do this and should it be a different IP range and/or subnet? If that is the right approach, how would I reach promox from a device which is on the LAN behind the pfsense router VM?

I would be very grateful for any pointers or suggestions for things I could read about this. I have spent a long time staring at the networking page of the proxmox documentation, particularly the section titled "Example: Use VLAN 5 for the Proxmox VE management IP with VLAN aware Linux bridge" but I can't quite seem to get my head around how it would map to my setup.

 

[Mencius]

I suppose to ask this in a simpler way: should I have any concerns (security or otherwise) about running this setup with the configuration above or is it fine?

Here is the blog post with concerns about virtualised router/firewalls: Things That Make Pat Nervous: Running Your Firewall in a Virtual Machine

 

[Mencius]

I'm fairly stuck with this but will update this thread as I work towards an answer, perhaps at some point an answer that could help someone looking at the same thing in the future.

I tried creating a VLAN for the proxmox management IP and set vmbr2 to vlan aware like this:

auto vmbr2.1
iface vmbr2.1 inet static
    address 192.168.1.100/24
    gateway 192.168.1.1
#VLAN for proxmox management IP

auto vmbr2
iface vmbr2 inet manual
     bridge-ports enp2s0 enp3s0 enp4s0
     bridge-stp off
     bridge-fd 0
     bridge-vlan-aware yes
     bridge-vids 2-4094
#Software switch for LAN - all ports marked 2 to 4 on case. Also handles router management, file server management

But I must be misunderstanding something fundamental because the result was that the pfsense VM continued to work fine but I was no longer able to access the promox management interface. Though the proxmox management IP showed up in the pfsense ARP table it was inaccessible.

 

[guletz]

What I am trying to understand is whether I should remove the proxmox management IP from vmbr2 and separate it somehow. Is this better security

Yes.

Would I use a vlan to do this and should it be a different IP range and/or subnet? If that is the right approach, how would I reach promox from a device which is on the LAN behind the pfsense router VM?

As you are declare yourself a beginer, is difficult to manage VLANs, I would suggest to you this:
- the WAN bridge (pfsense) is OK
- create a new bridge(vmbr3) using let say enp2s0, add a IP different from LAN IP space(let say 172.16.1.100/24, def gw=172.16.1.1)
- create a new iface(with 172.16.1.1/24=mgm) in pfsense using vmbr3
- in pfsense create some rules to isolate wan from mgm:
- new connections from wan -> mgm DENIED
- new connections from mgm -> LAN DENIED
- port-forwarding from LAN from your IP Laptop/Desktop/what-ever -> 192.168.1.100:8006 -> 172.16.1.100:8006 ACCEPT

Note, that as fallback, you can plug your device with a cable in enp2s0 and access Proxmox IP(with a IP from the same IP space on your device)
Even more tricks:
- install on pfsense proxy-server, and setup that Proxmox updates to use this proxy(google apt proxy)
- disable in pfsense mgm interface, and enable only you need to access Proxmox IP

Good luck / Bafta!

 

[Mencius]

Yes.




As you are declare yourself a beginer, is difficult to manage VLANs, I would suggest to you this:
- the WAN bridge (pfsense) is OK
- create a new bridge(vmbr3) using let say enp2s0, add a IP different from LAN IP space(let say 172.16.1.100/24, def gw=172.16.1.1)
- create a new iface(with 172.16.1.1/24=mgm) in pfsense using vmbr3
- in pfsense create some rules to isolate wan from mgm:
- new connections from wan -> mgm DENIED
- new connections from mgm -> LAN DENIED
- port-forwarding from LAN from your IP Laptop/Desktop/what-ever -> 192.168.1.100:8006 -> 172.16.1.100:8006 ACCEPT

Note, that as fallback, you can plug your device with a cable in enp2s0 and access Proxmox IP(with a IP from the same IP space on your device)
Even more tricks:
- install on pfsense proxy-server, and setup that Proxmox updates to use this proxy(google apt proxy)
- disable in pfsense mgm interface, and enable only you need to access Proxmox IP

Good luck / Bafta!

Thank you for your very helpful reply. You helped me clear up where I was getting stuck, I was imagining I could do all of this in proxmox alone but it makes sense that I need to manage things like the separate interface within pfsense.

I was thinking of using VLANs so I could keep as many physical ports free as possible but you are right, this is the easier and better way for me to learn. Of course, having a NIC reserved to manage proxmox in emergencies is appealing too.

I have started testing your suggestions out in VMs before I implement them.

I am also on the lookout for a linux based router/firewall distribution as I am not entirely happy with pfsense. Mostly because it does not currently support multiple queues on virtio network adapters.

Does anyone know of a good alternative? Perhaps ipfire?

 

[guletz]

Does anyone know of a good alternative? Perhaps ipfire?

Hi again,

I am happy that I was able to help you. Now because you want a firewall, maybe You could considered a cheap hardware firewall. For sure you will have better performance and security, and you can learn a lot of things. Any software firewall can fail for many reason, in a such case you will have a big problem. Using a hardware firewall you will lower the probability to face a bad situation.

A cheap and powerful hw router could be Mikrotik Hex S. I also use this as a border firewall in front of Proxmox servers.
.... and you can learn a lots of things.

Good luck and fast learning