pfSense in Proxmox with no external switch

[UniverseX]

Hi,
I have a box with 8 built-in NICs on the motherboard which I'd like to be used as a firewal and switch for my network. I have pfSense and Ubuntu Server installed inside of the Proxmox. I'm struggling to configure pfSense/Proxmox in such a way that pfSense (or/and Proxmox) will be used as a switch so I don't need to have an external switch (I know it's easier to configure pfSense when there is one). My undestanding that I need to create 1 Linux Bridge for pfSense WAN port which will be physically connected to the modem. Other 7 ports will be managed by Proxmox? Or they also need to be bridged to pfSense so it will be routing it? Probably diagram will make more sense what I'm trying to achieve.
Would anyone able to advise on this? Much appreciate.

 

[Alwin]

You can passthrough the WAN interface to the pfsense directly and use hardware features, thus eliminating the need for a bridge for only the pfsense. Secondly you could create a bridge with all remaining ports connected and add an interface from the ubuntu server and one from the pfsense.

[WAN]  <->  eth0 : pfsense : eth1  <->  vmbr0 [ETH2-ETH8]  <->  eth0 : Ubuntu

I hope the illustration makes it more clear (then confusing ).

 

[jim.bond.9862]

I think the way it would go is like,
First bridge is vmbr0 with 7 ports on it. That will be your LAN and switch.
Next vmbr1 port 8 for WAN, connected to your modem.
On pfsence vm add vmbr1 as WAN and vmbr0 as LAN.
The only issue I can see is ip address on pfsence and proxmox host.
The pfsence must be your gateway an thus would be .0.1 ip.
Proxmox need to be static but where do you assign it.
Or you will need to do extra bridge for host and do proxmox routing.

 

[UniverseX]

Thanks for the replies. @Alwin I've tried as you've advised, but it doesn't seem working. Ubuntu machine doesn't get connected to the network (can't ping it and can't seem from the inside of the Ubuntu). Unless I do something wrong

 

[jim.bond.9862]

This will not work. You need 2 bridges.
Vmbr0 with 7 ports and vmbr1 with 1 port.
Plug the port on vmbr1 to your WAN(cable modem)
Than assigne vmbr1 to your pfsence WAN port.
And vmbr0 to tour pfsence lan port. Your pfsense should be your router and gateway. On proxmox host you need statuc ip and gateway.
All your VMs will be connecting to vmbr0.

 

[Alwin]

Depending on your network setting, everything on the vmbr1 should be able to ping one another.

For the pfsense, if you didn't already, here the docs for the passthrough.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

 

[jim.bond.9862]

There is no need to passthrough.
It is much easier and simpler to just create a bridge on the port you want to use amd attach vm to that bridge. I bave been running my pfsence like this for almost a year now.
Why complicate things with passthrough.

 

[UniverseX]

Thanks guys. I've tried as you've advised.
However, after pfSense installation and moving to the newly created network(attaching my PC), I wasn't able to open anything in the browser. PC gets IP address assigned successfully. pfSense web interface opens but extremely slow. pinging external addresses in ssh session fails. It's not always possible to connect to the SSH. It's seems that routing and/or DNS resolution isn't working.
I've also disabled Disable hardware checksum offload as advised here - docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-proxmox.html
I've tried to remove vmbr0 but connection to the Proxmox gets lost, so I guess it should stay (had to edit interfaces file to bring connection back).

 

[Astraea]

When you open the console for that vm what nics are assigned to what function, check the macs to make sure pfsense is testing the wan and lan correctly.

 

[Alwin]

Do you have enough resources on your host? I can see two VMs with a total of 8 cores and 16 GB RAM. May you have overcommited?

 

[UniverseX]

When you open the console for that vm what nics are assigned to what function, check the macs to make sure pfsense is testing the wan and lan correctly.

NICs are correctly assigned to the corresponding interfaces, MAC addresses inside of the pfSense matches to what is assigned in Proxmox to pfSense VM.

 

[UniverseX]

Do you have enough resources on your host? I can see two VMs with a total of 8 cores and 16 GB RAM. May you have overcommited?

So far only 50% RAM assigned to pfSense (and only 6Gb used by VM when running) and CPU load is quite low all the time. I don’t see any shortage of resources when VM running.

 

[Alwin]

NICs are correctly assigned to the corresponding interfaces, MAC addresses inside of the pfSense matches to what is assigned in Proxmox to pfSense VM.

What solution did you use, the passthrough or a bridge on the node?

 

[UniverseX]

What solution did you use, the passthrough or a bridge on the node?

I've tried both, result was the same in both cases - pfSense sloweness or not responding at all and no connection to the external websites.

 

[UniverseX]

Reading this - docs.netgate.com/pfsense/en/latest/virtualization/virtio-driver-support.html - Hardware checksums and other NIC offloading features like TSO may also need to be disabled on the hypervisor system in addition to the pfSense VM.
Do I also need to do it somehow in Proxmox?

 

[UniverseX]

I've tried both, result was the same in both cases - pfSense sloweness or not responding at all and no connection to the external websites.

I think the WAN side is working fine, the issue is with LAN. At some point I was even able to run pfSense update process via terminal.

 

[Alwin]

Reading this - docs.netgate.com/pfsense/en/latest/virtualization/virtio-driver-support.html - Hardware checksums and other NIC offloading features like TSO may also need to be disabled on the hypervisor system in addition to the pfSense VM.
Do I also need to do it somehow in Proxmox?

Possibly. Use ethtool for this and see if it gets any better.

 

[UniverseX]

Possibly. Use ethtool for this and see if it gets any better.

Unfortunetaly this didn't help at all. Tried to install OPNsense and results are better - able to access OPNsense web interface without any delays, however no access to internet.

 

[nicholasc1790]

There is no need to passthrough.
It is much easier and simpler to just create a bridge on the port you want to use amd attach vm to that bridge. I bave been running my pfsence like this for almost a year now.
Why complicate things with passthrough.

Bridge? do you mean to create a linux bridge and attached it to the ethernet port and if you put the gateway to pfsense LAN. will this work without needed to use VLAN in pfsense

 

[jim.bond.9862]

Bridge? do you mean to create a linux bridge and attached it to the ethernet port and if you put the gateway to pfsense LAN. will this work without needed to use VLAN in pfsense

Yes it works. I am running pfsence vm on Lenovo p58 machine right now.
Idea was to be able to try out several firewalls for easy switching but pfsence was the simplest to setup and run for my needs. It is a simple home router replacement.
Machine has 3 nics.
One for Proxmox management
One for wan and one for lan.
Proxmox is set with 3 bridges.
Vbr0 vbr1 and vbr2

Vbr0 is Proxmox default set in management nic
Vbr1 and 2 is for pfsence.