How to protect ProxMox with a virtual virewall?

[Ovidiu]

I'm looking to run a virtual FW appliance on ProxMox and would like it to protect all incoming traffic - for the hypervisor and all guests but am not sure how the networking part has to look like. I have attached a drawing of the current situation. Any hints on how to achieve this are very welcome.
At a later stage I'd like outgoing traffic to be also routed through this FW but I guess that would be achieved by simply using the FW as the gateway for the hypervisor + guests?

 

[manu]

I don't have this setup in my testlab, but I would try the following way.

* give the virtual firewall device direct access to the nic, by doing a pci passthough of the PCI device to the firewall
this will be the WAN interface of the firewall
* assign a virtual NIC to be the LAN device of the firewall and add this device to the br0 bridge.
Then you have to make sure that you set up the right routing or port forwarding to the Proxmox Host from the Firewall.

Note that this is of for the faint of heart, so make sure you have proper testing here.

 

[Ovidiu]

thanks for pitching in and for the pointers.
this sounds complicated. I had assumed I wouldn't need direct access to the nic but could do this with some clever bridging. I'll rethink this unless I get some other feedback.

 

[manu]

Hi Ovidiu
I think it should be possible to do that with clever bridging, it's just I haven't had a thought about that yet. Let see if other users can chime in.

 

[Denny]

I described my setup in some detail here for another user who was trying to do what you are asking. In short I use two bridges and also utilize vlans but the concept isn't dependant on using vlans. If you are using pfsense for instance, you could setup an HA pair using the basic design I am suggesting for redundancy.

 

[manu]

Actually on reading Denny's point it is not as complicated as I first thought.

Basically I would do as such:

* remove the eth0 device from the vmbr0 bridge ( and also deconfigure the IP from it if you want the PVE host to be behind the firewall)
* create a bridge called wan0, where you add the eth0 device
* create a VM for your firewall. I have a sweet spot for pfSense as I use it personnaly, but it does not matter here (VyOS, NetBSD .. )
When creating the VM for your firewall make sure it has two NICS:
* a WAN NIC to be bridged to the wan0 bridge
* a LAN NIC to be bridged the the vmbr0 bridge

Then if all your VMs have their NICs in the vmbr0, you would only need to change their gateway.

All of this is not so complicated if you have local access to the PVE host. If you need to this remotely you really have *to know* what you are doing.

 

[Ovidiu]

@Denny that was exactly what I was asking for, I'm still busy reading through that post but would you mind explaining these two sentences from your linked post?

My UTM gateway is the only device that connects to vmbr1 (internet) and everything else connects to vmbr0 (internal). By doing it this way I can migrate the UTM to any host in the proxmox cluster.

and #2

If your switch doesn't support vlans you would need two ethernet interfaces to attain the same functionality.

Doesn't this next quote state the exact opposite - unless I'm missing a point somewhere here?

In short I use two bridges and also utilize vlans but the concept isn't dependant on using vlans.

 

[Denny]

@Denny that was exactly what I was asking for, I'm still busy reading through that post but would you mind explaining these two sentences from your linked post?

and #2

Doesn't this next quote state the exact opposite - unless I'm missing a point somewhere here?

Sorry I wasn't more clear. If you don't have a switch capable of separating internal from external traffic (ie vlan aware) you will need separate switches for each if you intend to be able to move your firewall virtual machine from one host to the other seamlessly.

 

[Denny]

Here is what my /etc/network/interfaces file would look like if I weren't using vlans and instead had two ethernet cards plugged into an internal and external facing switch.

# network interface settings                                                                                                                                                                                                                 
auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual


auto bond0
iface bond0 inet manual
        slaves eth0
        bond_miimon 100
        bond_mode 4



auto bond1
iface bond1 inet manual
        slaves eth1
    bond__miimon 100
    bond_mode 4

auto vmbr0
iface vmbr0 inet static
   address  192.168.1.20
   netmask  255.255.255.0
   gateway  192.168.1.1
   bridge_ports bond0
   bridge_stp off
   bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
   bridge_ports bond1
   bridge_stp off
   bridge_fd 0

Note that bond0.9 becomes bond1 and that eth1 is slaved to it.

 

[Ovidiu]

Would me diagram now look correct? The top one is the original, current setup, the bottom one would be what I understood from your explanations.

I think I got it right, a bridge doesn't need an IP, so with the bottom diagram, incoming requests for the IP of the hypervisor or any of the guests would end up on the wan0 bridge from where they would be routed by the Virtual FW through br0 to their respective NICs, right?

Looks way to complicated to implement in live mode I'm just trying to educate myself here. Btw. would something like this also work for any other hypervisor? i.e. vmware or would that need to be handled differently.

 

[Denny]

This is what I mean



In this drawing BR0 is the external (internet) bridge. It is associated with eth0 and connected into your demarc. BR1 represents the internal network and it is associated with eth1. The only machine (virtual) connected to BR0 is the firewall/UTM device (such as pfsense) and has the public IP address. The firewall also has a connection to the internal network to allow inspected traffic in and out. All regular vms have a connection to BR1 only and therefore must travel through the firewall to get to the internet.

It is worth noting that BR1 has an internal IP address assigned to it to allow access to the hypervisor.
If you are trying to setup a DMZ environment I will modify the drawing to show that.

Hopefully that helps, I sometimes can get too complicated in my explanations.

Would me diagram now look correct? The top one is the original, current setup, the bottom one would be what I understood from your explanations.

I think I got it right, a bridge doesn't need an IP, so with the bottom diagram, incoming requests for the IP of the hypervisor or any of the guests would end up on the wan0 bridge from where they would be routed by the Virtual FW through br0 to their respective NICs, right?

Looks way to complicated to implement in live mode I'm just trying to educate myself here. Btw. would something like this also work for any other hypervisor? i.e. vmware or would that need to be handled differently.

 

[boopzz]

Just for some confidence, this is almost exactly how I have mine setup. I have one interface that my cable modem plugs into that goes to one bridge, and another bridge that connects to a trunk on my VLAN switch. pfSense is the L3 device that routes between (and Firewalls and NATs) them.

I've also a few more VLANs, one for clients, one for servers, another one as a L3 transit VLAN. To do this I tick the "VLAN Aware" checkbox on the bridge and then add the VLANs onto the pfSense box.

 

[Ovidiu]

Thanks for all the info guys, the drawing is awesome BUT: this looks like it will only work with 2 physical nics, I was looking to figure out how this can be done with only 1 physical nic on the hypervisor. Am I misreading something here or is that not possible at all?

###edit###
sorry if I lead this discussion into the wrong direction, I assumed the 1 physical nic in my first drawing was noticed :-(

 

[Denny]

I think I am headed for an answer that you won't like. You either need multiple nics or multiple vlans. There must be some way to separate the internal and external traffic. This matches my layout as described earlier and detailed in my interfaces file.

 

[Denny]

Needless to say, my internet demarc (fiber) plugs into port 2 on my switch. Proxmox hosts plug into ports 1,4,5, and 7 on the switch. The connection to the internet therefore rides on vlan 9. Internal traffic rides on vlan 1 and is untagged. Complicated maybe for a home environment but not too much so.

 

[Denny]

Rereading your question at the beginning of this thread it occurs to me that I am envisioning an internal physical network being supported by the virtual machines on the network and protected by the firewall vm. You never mentioned that actually so I need to ask, is this the situation or did you only need to protect the virtual machines on the host but no need to provide communication to an internal network beyond the host? The later is possible without vlans or a second nic.

 

[Ovidiu]

Hi @Denny and thanks so much for the details which have been very helpful. This was a theoretical question I had because at the company I currently work for, someone had the idea of moving a few linux servers from our DMZ onto a server in the cloud. I had a look at our needs and realized we would only need a small to medium sized root server and a few VMs, there are no critical services running on these VMs. Since I prefer ProxMox I looked at how this can be accomplished although I found we still have a spare VMware license floating around so that is also an option.

So basically I was looking at placing a hypervisor into the cloud which hosts a few VMs and wanted to protect them. No other internal network needed. I know hypervisors usually have some firewall and security baked in but I saw this as an opportunity to learn and I was keen to check out opnsense

Curious to see the solution you mentioned here:

only need to protect the virtual machines on the host but no need to provide communication to an internal network beyond the host? The later is possible without vlans or a second nic.

 

[Denny]

It is pretty simple, first of all you don't need any bond interfaces and the internal network bridge (vmbr0 above) looks like this

auto vmbr0
iface vmbr0 inet manual
bridge_ports none
bridge_stp off
bridge_fd 0

keep in mind that with this setup you lose access to the hypervisor if the firewall vm goes down (unless you have drac access or similar).

Are you considering colocation or cloud infrastructure? I can see where Proxmox fits in a colocation design but if you were using the providers cloud facilities you wouldn't need your own hypervisor. Rackspace (I am intimately familiar as I worked for them) AWS and many others provide everything you need including load balancer/waf facilities.

 

[Ovidiu]

@Denny: thanks, that looks pretty straight forward. Will test in a lab first

I was looking renting a basic server which covers my needs at around 150$ / month (I can use ProxMox or VMware which I have a spare license for). Not sure what you can get for that money with regards to cloud facilities. Feel free to pm me, I had a look at Rackspace's website and can't find any prices regarding my needs. Its all "contact us for prices" - but I assume its way over what I am looking at.

 

[Denny]

@Ovidiu , Of the adjectives that I would use to describe Rackspace, inexpensive isn't among them. It is easy to set this up but it does hamstring you a little unless you are provided out of band management such as Drac. I noticed an error in my statement above but the importantt part is bridge_ports none. I left out putting an IP address on the internal interface of the hypervisor.