Virtualising OPNSense on a Proxmox cluster – Chicken vs Egg

[tipex]

I would like to virtualise my OPNSense router, but I can’t get my head around how the network and my 3 node Proxmox cluster would deal with this.

For example to setup a Proxmox cluster you need the machines to already have IP addresses and a working DNS. In my case the servers are also dropped into a VLAN by my switch which is trunked to my OPNSense machine. Without my OPNSense router I don’t think any of this would be possible i.e. I need the router there before I can create my cluster.

Then on the other hand I need my cluster in place so I can create the OPNSense VM.

So in my mind I am now stuck with a chicken and egg scenario.

Part of me thinks leaving it as a separate device is best but it would be real nice to virtualise it so I get all the benefits of migration, high availability, snapshots etc.

Any thoughts?

 

[wbk]

Hi tipex,

Would you not start with a single Proxmox with a fixed IP, get that running, install OPNSense, and then add the other nodes? How much DNS would you need? If it is only for apt, you could get away with a couple of entries in /etc/hosts until things run.

If you don't have a router yet, you could of course use one of the other machines for a temporary bare metal installation of OPNSense. You could export the configuration and later import it to the virtualized OPNSense.

 

[Deleted member 205422]

For example to setup a Proxmox cluster you need the machines to already have IP addresses and a working DNS.

Why do you think you need a DNS to have the cluster? I was just wondering why it's all based on static IPs in my other post - anything hostname resolution related is basically pulled from the hosts file locally, the cluster config holds IP addresses by default.

In my case the servers are also dropped into a VLAN by my switch which is trunked to my OPNSense machine. Without my OPNSense router I don’t think any of this would be possible i.e. I need the router there before I can create my cluster.

VLANs are L2, your nodes are on the same VLAN and same L3 subnet, there's nothing to route for them, is it? The (assuming L2) switch should be sending it to the right place.

Then on the other hand I need my cluster in place so I can create the OPNSense VM.

Well your cluster does not need to have anything L3 routed.

Any thoughts?

I might be wrong, but it's all good in my mind on the L2 switch as long as the VLAN tagging works well.

 

[Deleted member 205422]

I would just add that for having DNS for downloading the OPNSense, etc. of course you either could pre-prepare that or use another router while you are working on creating the cluster, then change the settings on the switch so that the VLAN for routing it all out goes into your OPNSense instead of the original router.

Be sure to not cut yourself off from the management of the switch, be extra careful if it's something like controller managed and have a plan B with that switch. I would also set low <5min lease time on (both the old and new) DHCP server before trying to transition bigger network like this.

 

[tipex]

ok yes I see what you are both saying. With regards to the hostnames I think I was confusing things with stuff I read online where the hostname of the Proxmox node must be fixed and cannot change once the cluster is formed. Looking through my install notes the hostnames are not used when forming the cluster - just IP addresses like you say. Plus I know what I want the hostnames to be upfront regardless of there being a DNS server or not.

OPNSense defines the VLANs and network address ranges so I was thinking for the Proxmox nodes to be in a VLAN OPNSense would need to be present. Thinking about it more I also define the VLANs in my switch only without any network address range. Using my switch I could probably get the Proxmox nodes into the correct VLAN without needing OPNSense. Without OPNSense dishing out DHCP I would need to fix the IP address of each node which is doable.

So in summary it sounds possible but could be a bit fiddley. Ideally, I would try it but it sounds a lot of work when things are already built and running. It’s something to try if I ever rebuild the whole network from scratch.

I do like the idea of running OPNSense on a separate device just to get the network up and running. Then once the Proxmox cluster is formed move the settings over to a virtualised OPNSense. I might try this soon as this is the position I’m in now.

 

[Deleted member 205422]

ok yes I see what you are both saying. With regards to the hostnames I think I was confusing things with stuff I read online where the hostname of the Proxmox node must be fixed and cannot change once the cluster is formed. Looking through my install notes the hostnames are not used when forming the cluster - just IP addresses like you say. Plus I know what I want the hostnames to be upfront regardless of there being a DNS server or not.

I think it uses what is on the /etc/hosts it does not use DNS lookups, so even you see it is referencing names, they do not need to be in your networks e.g. dnsmasq.

OPNSense defines the VLANs and network address ranges so I was thinking for the Proxmox nodes to be in a VLAN OPNSense would need to be present. Thinking about it more I also define the VLANs in my switch only without any network address range. Using my switch I could probably get the Proxmox nodes into the correct VLAN without needing OPNSense. Without OPNSense dishing out DHCP I would need to fix the IP address of each node which is doable.

VLANs are layer 2, IP is layer 3, the switch is switching, not routing. All you do is basically make every port you want for Proxmox to serve that VLAN as untagged (for the start at least), no? Regarding the IPs, PVE ISO install literally requires you to set static IPs, there's no DHCP baked in there.

So in summary it sounds possible but could be a bit fiddley. Ideally, I would try it but it sounds a lot of work when things are already built and running. It’s something to try if I ever rebuild the whole network from scratch.

I do not think it's fiddly at all, it's just at some point you will put that VLAN together with your OPNSense, up to that point it will be without router or you will use your other router. I would however not want to have this running like this in "production" at all. Why not route with OPNSense only some of your VMs? Not the cluster itself?

I do like the idea of running OPNSense on a separate device just to get the network up and running. Then once the Proxmox cluster is formed move the settings over to a virtualised OPNSense. I might try this soon as this is the position I’m in now.

See one para above, at least that would be my take. But nothing wrong with running router as a VM.

 

[Deleted member 205422]

On a separate note, when you do these things with you hardware switches and you don't want to cut a branch from under yourself, make sure you keep one port on a VLAN on which the switch can be managed, with static IP, then if things go south you can literally plug patch cable to that one port and set static IP on your computer (or WiFi to eth adapter) and fix it without having to reset and restore (from backup, right?) ...

 

[tipex]

Yes I already have an emergency access port on the switch and have needed to use it already hahaha! Luckily I realised the need for such a thing.

Everything I am doing is just for home use. Just a playground to learn but I do like to try and do things the best practice way and pretend its production like. I get what you saying with regards to having the router on as a VM. Its makes for a somewhat fragile network. I don’t need to virtualise it. The idea came from two things:
- My current device does not have enough CPU power to handle Gbit fibre. I bought it back when I only had an ADSL connection. Rather than buying new kit I thought why not virtualise it instead.

- Backing up physical hosts I find a pain in the bum. If I had OPNSense virtualised I can set automated snapshots and then easily restore. Like say I do an update but it goes horribly wrong.

I guess in large production environment routers would be on dedicated hardware and you would probably have two for high availability. You could then update one and if the update went wrong at least the other is still running.

 

[Deleted member 205422]

- My current device does not have enough CPU power to handle Gbit fibre. I bought it back when I only had an ADSL connection. Rather than buying new kit I thought why not virtualise it instead.

You will hate me for this bot for a supposedly gigabit fibre ONT I would buy an EdgeRouter-X for $50, it definitely routes Gigabit with HW offload, it can do much more, but I get it, you like OPNSense. The backup with ER is just a config file and that's it. Now before you hate me entirely, that can be your router, it does not have to be your firewall, you can set DNAT/SNAT rules (yes, the ER is also VLAN aware) as you want to the OPNSense and for instance have your VMs then routed via that OPNSense.

- Backing up physical hosts I find a pain in the bum. If I had OPNSense virtualised I can set automated snapshots and then easily restore. Like say I do an update but it goes horribly wrong.

Will be still possible.

I guess in large production environment routers would be on dedicated hardware and you would probably have two for high availability. You could then update one and if the update went wrong at least the other is still running.

It's different topology then with multiple WANs, you would want to multihome and BGP and all. But anyhow, I think if you like tinkering with routing you might even give VyOS (if you google, you'll find out why) alongside the OPNSense. I think one is a router (albeit with iptables), the other is a firewall (but it can route too).

 

[Deleted member 205422]

Oh I forgot to add - you can then have OSPF between the ER-X and the OPNSense too. Anyhow my point was, in case your VMs went down, it would take down the OPNsense too, but if they were the only ones routed out by it, no big issue for anything else in the network that basically wants to be simply TCP/UDP all outbound allowed and inbound only established/related, dropping invalid and maybe reply basic ICMP by the router itself. That's it. In case the OPNSense is down, the DNAT rules will simply not have any traffice routed anywhere where it bothers you.

Bonus point: you will learn a lot about routing having 2 routers instead of one. And for those "I have done something horrible" moments on the hardware router, it supports commit-confirm or simply commit before save commands like it was some kind of CISCO...

 

[tipex]

Oh man this is information overload... I love it.

I need some time to digest and google all these words you are telling me

I apreciate all the info. This gives me loads of stuff to think about.

I've always considdered a network as a firewall/router at the top and then everything hanging off it. Maybe I need to forgot this and start from a blank peice of paper.

 

[Deleted member 205422]

Oh man this is information overload... I love it.

I need some time to digest and google all these words you are telling me

I apreciate all the info. This gives me loads of stuff to think about.

I've always considdered a network as a firewall/router at the top and then everything hanging off it. Maybe I need to forgot this and start from a blank peice of paper.

No worries, it's good to have stuff virtualized for testing concepts and then of course the common logic is first barrier needs to be firewall, then router, etc., but in home setup, your services are basically your DMZ, you do not really need a Cisco ASA to be the first thing that gets hit from WAN inbound, you can have simple router with basic iptables (SNAT/DNAT is just iptables too), then for the part you are about to expose, you put it behind e.g. OPNSense including with VPN endpoints etc. You can't really set up e.g. split horizon DNS easily on that little edgerouter, but frankly you may as well route everything going to access your services as if coming from the outside, it will have to pass through the OPNSense anyhow. Or you can IPsec "in" even when you are at home (into your lab environment).

Just keep in mind, if you were to set up both IPv4 (supposedly with NAT) and IPv6, then the v6 is best done via CLI on ER-X, but that should not stop you. You absolutely want to have IPv6 iptables set up because those are all internet routable addresses. Fun fact, if you do not have IPv6, you can set up Protocol 41 (i.e. 6in4 tunnel, for instance at tunnelbroker.net) right on that Edgerouter that will feed the rest of your network including that OPNSense, you can also only route it to the OPNSense. Many possibilities, but from there it becomes more about networking than VMs/CTs ...

Oh and one more thing, you may even port mirror your WAN port (but I would use a proper switch for that, not the little router) into some sort of Snort or Suricata that runs as yet another discreet VM.