Proxmox & pfsense configuration

[richjim]

I have an existing network that I am routing out my traffic to a public IP 1 using a physical router. I am trying to set up a Proxmox VM using Pfsense so I can move traffic through it using public IP 2. I would like to be able to configure my network devices and VMs to point to either the physical router or the pfsense by changing the gateway on the device depending on what public IP I would like them to use.

When I change the network settings in Prox for a VM to use vmbr2 and change the gateway on the VM to use pfsense (192.168.0.2) I can't get out to public IP 2.
Any assistance would be appreciated to get this accomplished.


I have a network with the following configuration
vmbr0 (eno1) Used to access Proxmox IP 192.168.0.5 GATE 192.168.0.1 which points to my PUBLIC 1 through a physical router (192.168.0.1)
vmbr1 (eno2) which I am using for pfsense WAN PUBLIC IP 2
vmbr2 (eno3) which I am using for pfsense LAN 192.168.0.2

 

[datdenkikniet]

When I change the network settings in Prox for a VM to use vmbr2 and change the gateway on the VM to use pfsense (192.168.0.2) I can't get out to public IP 2.

Can you clarify "can't get out to public IP 2"? Does L2 traffic propagate to the pfSense VM and is it an L3 issue (you might be able to verify this by checking the ARP tables of either machine), or can traffic from a VM on vmbr2 not reach the pfSense VM at all?

 

[richjim]

Can you clarify "can't get out to public IP 2"? Does L2 traffic propagate to the pfSense VM and is it an L3 issue (you might be able to verify this by checking the ARP tables of either machine), or can traffic from a VM on vmbr2 not reach the pfSense VM at all?

Thanks for your quick response. I actually can't reach the pfsense from a VM on vmbr2

 

[datdenkikniet]

Thanks for your quick response. I actually can't reach the pfsense from a VM on vmbr2
View attachment 20217

How do you obtain the IP address? Are you 100% certain that your subnet and gateway are correct?

Asking because the VM you are pinging from right now does appear to be able to reach the machine with IP address 192.168.0.253.

 

[richjim]

I did verify the pfsense LAN is configured for 192.168.02/24. Also, I just checked a physical machine and changed the gateway to 192.168.0.2 which worked! All its traffic went through pfsense > Public IP2.

The issue now is only with the VM's. When the VM's are using vmbr0 (192.168.0.1 gate) they can access the pfsense but traffic goes through public IP1. When I change the gateway to vmbr2 (192.68.0.2 gateway) I can't ping or access the pfsense.

 

[datdenkikniet]

I did verify the pfsense LAN is configured for 192.168.02/24. Also, I just checked a physical machine and changed the gateway to 192.168.0.2 which worked! All its traffic went through pfsense > Public IP2.

The issue now is only with the VM's. When the VM's are using vmbr0 (192.168.0.1 gate) they can access the pfsense but traffic goes through public IP1. When I change the gateway to vmbr2 (192.68.0.2 gateway) I can't ping or access the pfsense.

I think a more detailed description (in terms of networks and segments) of what you are trying to accomplish would help a lot here.

It seems like both your gateways are in the same network (192.168.0.1 and 192.168.0.2 are both in 192.168.0.0/24), but for some reason they are on separate L2 segments (vmbr0 for your normal LAN and vmbr2 for your pfSense LAN). Is there any specific reason that you don't want to connect your pfSense LAN to vmbr0 as well, or why you do not want the pfSense LAN to be a different network (i.e. 192.168.1.0/24)? The first option might solve the issue, and the second one probably won't, but it *might* help debug the issue a little further.

 

[richjim]

There was no reason I created the additional VMBR2 other than I just followed the Proxmox/Pfsense install guide which had me create the 2 bridges. I went ahead and did a re-installed pfsense using VMBR0 (LAN) and VMBR1 (WAN) for pfsense.

Current config:


I am looking to use 2 gateways.
Gateway 192.168.0.1 > physical router >Public IP1
Gateway 192.168.0.2 > pfsense> Public IP2

After the reinstall here is what is happening:
Physical devices using gateway 192.168.0.1 can ping/access pfsense (192.168.0.2) and traffic goes through Public IP1
Physical devices using gateway 192.168.0.2 can ping/access the pfsense and traffic goes through Public IP2
VM's using gateway 192.168.0.1 can ping/access pfsense and traffic goes through Public IP1
This is exactly what I want.

Here is where the issue is:
Any VM's using gateway 192.168.0.2 with VMBR0 can ping/access pfsense but no traffic is passed to Public IP2

I hope I gave enough information.

 

[datdenkikniet]

Aha! That does sound like it's more of a pfSense issue than a Proxmox issue, though, especially if the VMs can reach the pfSense machine. How do you connect the physical machines to the virtual bridge? By plugging them/the switch they are connected to into eno1?

 

[richjim]

I found the problem. I used this guide for the install https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox.html which had me use VirtIO for my network devices. I switched them to Intel E1000 which resolved the issue.

The install guide must be old because it was way off. Thank you for your help!

 

[datdenkikniet]

Though a little late to the party:

The problem is likely to also be solved by disabling "Hardware checksum offloading" (in "Avanced > Networking") in pfSense, since that seems to produce exactly this issue when VirtIO network devices are used. When using E1000 emulation the hardware checksum offloading does actually work, but the overhead is likely greater, as paravirtualized devices are easier to virtualize then hardware devices.

 

[whalewatcher]

Found the same problem, took me 2 days for narrowing down the problem and finding this post here. Great work. For disabling the hardware checksum offload you need to tick the checkbox in pfsense.

Tnx
Whalewatcher