Network and other config questions around building a Sophos UTM + extra services 'box'

[jaytee129]

Am an ex-ESXi user that's new to Proxmox, and who's unix days are far behind him

I'm trying to set up a Sophos UTM+ 'box' that has some extra services built in. Here's what I have so far:

- got latest version of Proxmox installed on an Intel i5 2.3GHz NUC (2 cores, 4 threads) no problem

- was able to setup a Sophos VM (after a few redo's) plus a Windows 10 Pro VM. The latter is to run Unifi controller, a UPS agent, and to allow remote access to the subnet (via Sophos IPSec Gateway VPN)

- I've assigned 1 socket / 2 cores to each VM, and bumped up the cpuunits to 1536 for Sophos VM to give it higher scheduling priority over Windows VM (left at default 1024 cpuunits)

- I setup PCI Passthrough of the built-in Gb NIC to Sophos VM for dedicated WAN port, and I have a USB 2.5Gb NIC (no PCI passthrough possible) as a shared LAN port. I don't expect to get (or need) 2.5Gb so I'll probably change this if/once I get this puppy working.

I want all the WAN traffic to go through the Sophos VM and have it be my LAN GW/DHCP/DNS server.

My questions:

1. I'm looking first and foremost for help with the networking. What should my '/etc/network/interfaces' files look like? See diagram. I think I need routing rather than the bridging it gave me out of the box. I'm worried 'trial and error' will lead to not being able to do anything and having to start over.



I'm also looking for advice/opinion on the following, if anyone has any:

2. Is using a USB NIC problematic?
- I've read reports of lost connections and performance problems but don't have a sense of whether it's isolated incident or common issue

3. I read that built-in virtual networking can be slow so some deploy Open vSwitch (though for one post I read that didn't help). How much of an issue is this? What should I look at to ensure performance is decent for internet speed of 150Mbps and 5-10 concurrent internet users?

4. I was thinking of turning off Proxmox firewall so I only have one firewall to deal with and to eliminate Proxmox PVE resource use for that purpose. Any reason I should leave it on? Anything else I should turn off (or turn on) given my use for this Promox host?


Any info would be appreciated

 

[shrdlicka]

Hi,

You can hand over usb ports/devices to a VM as well, so you could give the USB NIC to the Sophos VM. But you would need another NIC for proxmox itself.

1.) I'm not sure why on the USB NIC side you would need a routing setup.
2.) I as far as I know, lots of people use USB NICs. As always, on the forum you will read mostly from the people that have some problems with their setup .
3.) Till now, I didn't try the OVS setup. I would keep it simple for the beginning with a bridge setup. This shouldn't be a problem with 150 Mbps internet and a intel nuc. What I read mostly is people not getting the full gigabit of their connection.
4.) Firewall does the usual things. I don't think it takes up a lot of resources. It always depends on what you want to do if you want to use it or not, i.e. some service should not be accessible by some IP range ....

 

[jaytee129]

Hi,

You can hand over usb ports/devices to a VM as well, so you could give the USB NIC to the Sophos VM. But you would need another NIC for proxmox itself.

1.) I'm not sure why on the USB NIC side you would need a routing setup.
2.) I as far as I know, lots of people use USB NICs. As always, on the forum you will read mostly from the people that have some problems with their setup .
3.) Till now, I didn't try the OVS setup. I would keep it simple for the beginning with a bridge setup. This shouldn't be a problem with 150 Mbps internet and a intel nuc. What I read mostly is people not getting the full gigabit of their connection.
4.) Firewall does the usual things. I don't think it takes up a lot of resources. It always depends on what you want to do if you want to use it or not, i.e. some service should not be accessible by some IP range ....

Thanks for reply. The Sophos VM has two NICs - the on-board NIC dedicated to it for WAN port, and the USB NIC is shared by Sophos, Windows and Proxmox and that's working okay within the host with static IP addresses. What isn't happening yet is that the computer I connect to the USB NIC does not get a DHCP address from Sophos. I'm okay with giving static IP addresses to the internal components but I need to have Sophos give dynamic IP addresses to everything outside, If the issue is not related to using a bridge setup, what's preventing Sophos DHCP from giving out an IP address?

Also the WAN side of sophos needs a dynamic address from the ISP router. That looked like the routing config use case in the promox documentation at https://pve.proxmox.com/wiki/Network_Configuration.

 

[shrdlicka]

Hi,

the WAN side is the NIC directly given to the Sophos VM. It should just use it as it would a bare metal network card, you don't need any config for that on the Proxmox host.

The USB NIC/LAN side would be best for a bridge. A brdige will act like a switch inside your Proxmox host. So handing out DHCP leases from your VM to the LAN should work as well. I would try this without the Proxmox firewall first, to make the first testing easier.

Did you configure the default Proxmox bridge interface with a static IP and is it connected to the usb nic?

 

[jaytee129]

OK I reconfigured things a bit (sticking with bridges) and some key things now work but a problem remains.

First of all, I learned that if I pull the USB NIC out and put it back in, I can't reach anything anymore. Appears Promox doesn't support USB hot plugging - is that right, or is it because I need to change something for that work?

So I resolved that it's a bad idea to use USB NIC for internal access. Now instead of passing through the on-board NIC, I created a vm bridge for it and use it for internal access. I assigned a new vm bridge using USB NIC to Sophos WAN. After rebooting, everything that needs a dynamic IP address worked fine (Sophos WAN, Windows VM and connected PC), however the WinVM and PC can't reach the internet. (Sophos VM sees it fine). I don't think it's a Sophos issue as I'm using a restored config that works fine on another Sophos UTM currently in use.

Attached is updated network diagram, and below is my interfaces file (all generated automatically). Anything look strange or that might explain why traffic is not going through Sophos VM out to internet?

auto lo
iface lo inet loopback

auto enp0s25
iface enp0s25 inet manual

auto enx000fc91ff664
iface enx000fc91ff664 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.4.254/24
        gateway 192.168.4.1
        bridge-ports enp0s25
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enx000fc91ff664
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

 

[jaytee129]

Getting mixed signals from Sophos. It may not be reaching the internet after all. While some network IP lookups appear to be getting resolved, am also seeing 'unable to reach destination' messages in DNS log. I'm not actually connected to internet - am connected to my main router. maybe something there but really it should work - it's just doing what the ISP router would do.

Question is - should the config (in previous post) for vmbr1 work as a path to the internet

Also, I pulled USB NIC out and Sophos still showed WAN "up" after >5 mins. Should have changed to "down" by then. Proxmox did change status from Active='Yes' to 'No', and it did not go back to 'Yes' after I plugged it back in. I had to reboot to get it back to active=yes.

FWIW, I set the NICs for Sophos VM to Virtio (paravirtualized) from Intel E1000 when I rebuilt it this morning. Would that make a difference in any of this? I read virtio performs better but E1000 is more compatible.

 

[jaytee129]

So Sophos had lost it's "default gateway" setting on DHCP server (despite that setting being part of the restored config) don't know what happened there but when I put it back in the internet worked/

So ALL IS GOOD, at least connectivity wise. Now need to set up performance testing.