Proxmox 4.2 with pfSense 2.2.6

L

lubberlick

Active Member
May 20, 2016
12
1
43
45
I am trying to setup a small server with a NAS + pfSense for my home network.

pfSense is in a VM with vmbr0 and vmbr1 assigned as virtio devices.

vmbr0 is a bridge of eth0 that is connected to my cable modem
vmbr1 is a bridge of eth1 that is connected to my switch

vmbr2 is a bridge of eth2 that is connected to my switch and has the host assigned to it and the NAS

Everything is working as expected until I either reboot the proxmox or simulate a power failure on that machine. Afterwords pfSense fails to pull WAN until I reset my cable modem.

I can only speculate this has to do with the modem remembering the MAC of eth0 prior to the pfSense VM taking control of it? Or the modem getting mad about seeing a MAC it doesn't recognize momentarily.

I have tried matching vmbr0 to eth0 mac, as well as "spoofing" the mac in pfSense. Both options did not give desired results.

Any help is appreciated. Thank you.
 
In the spirit of helping future people who may one day google this thread with the same problem I am having. I have found a workaround that consistently resolves the problem.

A simple sh script executed after the host boots up to reset the interface seems to do the trick. This must be done on the host machine and not the VM.

Create this sh script following the directions indicated in the script. I put the directions there so I would know what I did in the future.

Code: 
#!/bin/sh
# Script for resetting eth* after boot completed
# Needed to fix cable modem locking onto MAC address
#
# Please configure eth* to be your WAN interface
# 
# Store in /usr/local/bin/comcrap.sh
# chmod +x /usr/local/bin/comcrap.sh
# add '/usr/local/bin/comcrap.sh' to '/etc/rc.local' above 'exit 0'
#
# test command 'cd /etc' then './rc.local'


# Bring down interface eth*
ifconfig eth1 down

# Sleep a few seconds to allow the modem to lose MAC lock
sleep 3

# Bring up interface eth*
ifconfig eth1 up

Example of /etc/rc.local

Code: 
#!/bin/sh -e
#
# Other stuff I chose not to put here
#
# By default this script does nothing.

/usr/local/bin/comcrap.sh

exit 0

I tried doing this inside the VM and on vmbr* inside the host. These options did not have the desired outcome.
 
  • Like
Reactions: Flu
Thank you I move my pfsense to proxmox and have this same issue. I will give your solution a try.
 
Try manually entering the commands after rebooting your proxmox. The idea is that during the reboot of the system the modem forgets the lan mac. When the system boots the real mac of the network card is locked onto by the modem before the virtual mac of the pfsense VM takes over. We simulate this again by turning the network card off long enough for the modem to forget the mac again and when we turn it back on the virtual mac is recognized because it is already initialized.

Test how long you need yo wait for your modem to forget the mac and adjust the above script accordingly.
ifconfig eth1 down
Wait a few seconds
ifconfig eth1 up
 
I tried it from the command line with no luck. I waited for 90 sec and it still didn't work. Are you using the virtio NICs on offense?
 
It is entirely possible the network card you are using does not support or does not fully support being turned off then back on via command line. If this is the case you could try other network cards to see if any of them will support this function. Otherwise my workaround won't work, because it is simulating that momentary turn off that occurs during a system reboot.

I had also contemplated using pcie pass through of the network card, however as my system does not support VT-d I could not test if that would work.
 
For clarity eth1 was the network card that was connected to my cable modem. It very well could be eth0 eth1 eth2.... Depending on your configuration.
 
Eth1 is definitely correct. I will see what I can do about trying another nic card. The way I get it to work when my power goes out is power cycle the modem then reboot the offense KVM.
 
Try just unplugging the Ethernet cable then plugging it back in then rebooting the pfsense. If this works then my workaround should work assuming you have a network card that can be turned off then on by command line.
 
This one also worked for me on my PVE with pfSense on HP MicroServer Gen8. Thank you! Also I have set mine on 18 seconds. Now it's 3 seconds before WAN detection. But this time depends on Hardware specs and boot time. So it can be too long for others. I am also on PVE 5

Anayway it would be very interesting to know the real root cause of this problem!
 
For anyone googling the issue, if you have an Arris/Motorola modem and are running into this issue, use a dumb switch between cablemodem and proxmox and set a laptop/pc to 192.168.100.2/24 and open a web browser and go to http://192.168.100.1 - proceed to the "addresses" tab. There you will see the learned mac-addresses. On Comcast, the max-cpe setting is 1 for residential and either 3 or 5 for commercial, I can't remember. Most Cable ISP's limit max-cpe to 1.

My best guess is that that the issue is the learned mac-address from the physical NIC. If this is the case, one solution for this would be to use a managed switch and make proxmox vmbr vlan-aware connected to a trunk port on said switch, and use vlan5 (for example) for the WAN and configure vlan5 as an untagged port on the switch going to the modem. In this scenario, the NIC mac address will not be present in vlan5 since it would only be present in the native vlan.

Another possibility is IPMI running inline on the NIC, or maybe PXE? IPMI is set to dhcp by default. Logging into the cablemodem will verify what mac-address it is locking into. Also, you can use Wireshark to sniff the traffic while the system is booting, which would also solve the mystery. If IPMI and/or PXE is present - it's easy enough to disable the rogue traffic.
 
Last edited:
  • Like
Reactions: crypted
The issue is definitely related to the modem locking into the mac address of the physical network card during reboot. Once the virtual mac takes over it refuses the connection. However it will allow a new mac address if the link is lost for a short time. This is what my workaround simulates and has been working without issue for me since my initial post.
 
Last edited:
Out of curiosity, try disabling PXE on NIC and verify you don't have IPMI enabled with dhcp (assuming it's built into the motherboard). I can't remember for sure, but max-cpe may be set by DHCP snooping on the modem (first mac-address to broadcast dhcp).
 
I am nowhere near that system at the moment, and I am sorry but I don't really want to poke at something not broken.

That said your theory does hold weight. If the modem really does only lock into the first DHCP request you may have found another solution to this problem.
 
You must log in or register to reply here.
Top Bottom