PFSense/OPNsense w/linux bridge: Unable to procure WAN IP upon node reboot

[exar_kun]

Greetings!

Successfully installed pfsense as vm with linux bridges, one for LAN and one for WAN. Operating as expected with no running issues and full connection speeds.

Issue is when the node reboots, pfsense is unable to grab WAN ip from cable modem. If I only reboot pfsense, maintains WAN ip with no problem. I have also tested with opnsense with same behavior.

I have found other posts here and on reddit with others having experiencing the same issue. Some reported that they solved the problem using NIC passthrough.

While I could pass the NIC through as well, I'd really rather not due to losing migration, and more importantly losing the ability for snapshots.

Has anyone found/heard of a solution for this other than pcie passthrough?

Thanks in advance!

 

[nqnz]

Is the WAN bridge or the NIC associated with the bridge configured to be used anywhere else? I'm using pfsense inside a vm and haven't ever ran into this.
Does the proxmox box itself get a WAN ip?

 

[vesalius]

Post your /etc/network/interfaces from proxmox and the network interfaces you have given the *sense VMs in proxmox, so everyone can see what *sense gets as wan.

 

[exar_kun]

Thanks for the replies!

@nqnz - At first, I did have the WAN bridges configured on both the *sense virtual machines. Obviously I kept one powered down while testing the other. But just to be sure, I eventually deleted the bridge configuration from the pfsense vm just in case there was some sort of residual thing going on that was causing the issue. However, no change in behavior. I'm not aware how to check and see if the proxmox node itself is grabbing the WAN ip initially before the *sense vm. Any suggestions on how to check? "IP A" cmd from the pve node did not show my WAN IP associated anywhere.

@vesalius - Here is /etc/network/interfaces from the pve node:

auto lo

iface lo inet loopback



iface eno1 inet manual



iface eno2 inet manual



iface enp4s0f0 inet manual



iface enp4s0f1 inet manual



iface enp4s0f2 inet manual



iface enp4s0f3 inet manual



auto vmbr0

iface vmbr0 inet static

        address 192.168.5.10/24

        gateway 192.168.5.1

        bridge-ports eno1

        bridge-stp off

        bridge-fd 0



auto vmbr1

iface vmbr1 inet manual

        bridge-ports enp4s0f0

        bridge-stp off

        bridge-fd 0



auto vmbr2

iface vmbr2 inet manual

        bridge-ports enp4s0f1

        bridge-stp off

        bridge-fd 0

and here is the screenshots of the configs:

pve node



and here is the opnsense config - I've already deleted the pfsense network bridges so don't thing posting that would be useful



Thanks again guys for taking a look!

 

[vesalius]

Don't see anything wrong with your setup. Is the vmbr0 gateway 192.168.5.1 the cable modem or something else?

 

[exar_kun]

vmbr0 is used for my management network.

Anyone have a suggestion on a particular log to review?

 

[vesalius]

might be the modem and pfSense combo?

https://www.reddit.com/r/PFSENSE/co.../?utm_source=share&utm_medium=web2x&context=3

 

[exar_kun]

It appears that is exactly what my situation is!

Wow, I would have never suspected a known working modem in bridge mode. I do have an Arris TM1602 cable modem. Your link gave me new search path and I found many posts reflecting my situation.

For anyone else researching the same problem: The issue can occur with a few different models of Arris modems. Either pfsense or opnsense will have the issue. I found posts with users running either as vm or baremetal with the same issue. In addition to the rebooting issue, in my case if my isp changes the wan ip *sense will not update. I would have to power cycle the modem.

I've elected to enable nic/pci passthrough and the issue is resolved. My setup can now survive any combo of proxmox/opnsense reboot and wan ip appears as expected. Disappointed to lose migration/snapshots, however no other options for now.

I don't feel my isp will provide me with a different modem brand, however we will have another isp option in my area soon and I plan to switch. Will revisit this issue when I do.

Thank you very much @vesalius, I was becoming obsessed with this issue and now I can move on.

 

[exar_kun]

Additional update: Utilizing nic/pci passthrough I've resolved the reboot issue, however I still have an issue with *sense not updating to a new wan ip whenever my isp changes it. Most likely going to move to a retail router, as it is becoming inconvenient to continue to deal with this issue.

 

[Dryadwoods]

Additional update: Utilizing nic/pci passthrough I've resolved the reboot issue, however I still have an issue with *sense not updating to a new wan ip whenever my isp changes it. Most likely going to move to a retail router, as it is becoming inconvenient to continue to deal with this issue.

Hi, I was wondering if you have news about this subject.
I also have a Arris modem and it seems that I am having a similar experience.

 

[Nollimox]

Additional update: Utilizing nic/pci passthrough I've resolved the reboot issue, however I still have an issue with *sense not updating to a new wan ip whenever my isp changes it. Most likely going to move to a retail router, as it is becoming inconvenient to continue to deal with this issue.

You may need to increase the time in pfSense to wait until the handshake to complete...900seconds = 15mins...I think it's in general settings below DNS. I am not near my firewall at the moment to be specific.

Editing the following day...OP if you're still here and continue have the problem, you need to see the WAN Interface has shown in image and where to increase the DHCP handling to get an IP...when you check advance, the box appears to insert the increase tome to 900 seconds which translates to 15mins.

 

[Dryadwoods]

You may need to increase the time in pfSense to wait until the handshake to complete...900seconds = 15mins...I think it's in general settings below DNS. I am not near my firewall at the moment to be specific.

Editing the following day...OP if you're still here and continue have the problem, you need to see the WAN Interface has shown in image and where to increase the DHCP handling to get an IP...when you check advance, the box appears to insert the increase tome to 900 seconds which translates to 15mins.
View attachment 48079

View attachment 48080

Thanks for the detailed information and printscreens.
One final question: will this also "force" the pfSense to identify that the current public IP is still valid or not, and if it is not (not a fixed IP from ISP, and I do not know if then would change it on the fly), would this be detected and somehow ask for the new one?

 

[Nollimox]

Thanks for the detailed information and printscreens.
One final question: will this also "force" the pfSense to identify that the current public IP is still valid or not, and if it is not (not a fixed IP from ISP, and I do not know if then would change it on the fly), would this be detected and somehow ask for the new one?

Yes, this is also a problem with cable on bare metal pfSense when it had been given an IP and for some reason (usually power interruption) pfSense doesn't have IP. It usually gets the same IP it had since the lease would still be effective...just need a little time to sort out.