Proxmox 4.2 with pfSense 2.2.6

[crypted]

Hi,

@Sharan H
Sound very interesting. I am going to sniff it with wireshark in some days. My temporary solution is to put a switch inbetween modem and nic. First I have to get a valid DHCP lease with connected modem to NIC. After that I have to switch cabel from NIC to switch and switch to NIC. This has to happen a little bit faster. Now I can safely reboot the server.

@lubberlick I went for this solution because yours isn't stable enough for me. After PVE kernel update I had to reboot and your solution faild. Than I went for the switch solution I read somewehere. I don't know what happend exactly but it was enough for me to search for another solution. Anyway your way is a good way

Also important to know is that if I lose power on my switch, than I have to redo the procedure above to get it working again. At weekend I killed a fuse and everything in the room where also the server and switch is went powerless.

@Sharan H
PXE seams to me to be disabled at my HP ProLiant Microserver Gen8. I have check every BIOS setting. I hope I haven't forgot something in there.
IPMI, at my server it is iLo, is not disabled, but it has a dedicated NIC which is also not shared to my other two GbE NICs. NIC 1 is my WAN, NIC2 is LAN and iLO is NIC 3. So this should not be the mistake?!?

I have also logged into my Cable Modem. And those Mac adresse what I have seen in there weren't thos from my NICs or VMBR0. This is crazy.

To mention it. My Modem is a Touchstone Arris TM722B.

Also this is the second server I have this problem. Before I had a normal PC setup with PVE and pfSense. It seems for me to be a PVE Problem?

Take care!
crypted

 

[lubberlick]

We all need to do what works for us. I am glad to have other ideas to look at from this thread should my solution fail me in the future. Thank you everyone for feedback!

 

[crypted]

@Sharan H
I ordered this: https://is.gd/Q58Oz9 and will try to get it working with VLAN. I will post my results.

 

[ndew]

@crypted I've setup quite a few pfsense boxes recently with proxmox 4 and 5, and the one thing i never could get working was virtio network cards, i had to revert to using e1000 for the card type when adding the network devices, once I did this everything works with out issue. If this has already been suggested I apologize for duplicating the info I did not see it in the thread.

 

[crypted]

@ndew
Sounds interesting. I already tried E1000 but not in detail. I think I should give this another try. I have chosen virtio because of the performance but I think at GbE both drivers should be fine, but I don't know. Could you tell me if you had in general a problem with virtio or only concering at this topic with cabel modem locking mac adress?

 

[ndew]

@crypted
We had many problems in general with virtio network cards, wasn't specific to your case, but VPN would not bind to them correctly, and some custom routing would not work. I don't think BSD/pFsense has great support for them, but the e1000 work great, never had any issues with them at all.

 

[Sharan H]

I have a production Pfsense running on Proxmox4/Virtio. Only issue I had was had to disable checksum offloading for IPSEC to work. Had the exact same problem on a hardware based pfsense running Intel nics.

 

[crypted]

My best guess is that that the issue is the learned mac-address from the physical NIC. If this is the case, one solution for this would be to use a managed switch and make proxmox vmbr vlan-aware connected to a trunk port on said switch, and use vlan5 (for example) for the WAN and configure vlan5 as an untagged port on the switch going to the modem. In this scenario, the NIC mac address will not be present in vlan5 since it would only be present in the native vlan.

@Sharan H
Could you please be so kind and explain this VLAN solution a bit more for me? I tried it now for 3 hours and googled so much but I think a am right now too newbie like to get it done. VLAN is new for me.

Thank you.

 

[Sharan H]

I haven't used Netgear, but I glanced at the manual: WebManagedSwitches_UM_EN.pdf Starting at page 47

There is a video on youtube, look up "Netgear GS105E Vlan Configuration / Setup"


You need to configure a single port as a unique vlan for the cablemodem (let's use Vlan 5). You need to configure this under Vlan > 802.1q >Advanced

Port 1: Cablemodem (Untagged Vlan5)
Port 2: Proxmox (Untagged Vlan1, Tagged Vlan5)
Port 3-5 Default (Untagged Vlan1)

Make sure Port 1 membership includes only Vlan5 and set to Untagged
Make sure Port 2 membership includes both Vlan1 and Vlan5 and set Vlan5 to Tagged


Proxmox:
/etc/network/interfaces
auto vmbr0
iface vmbr0 inet static
address 192.168.1.11
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports eth0
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes


Pfsense Hardware options:
- Edit Network Device for WAN
- Set VLAN Tag to 5

 

[crypted]

I don't know how to thank you. First thank you for your really fast respone. I tried it after that yesterday but didn't get it working. Now, in the morning I found my mistake. I had to choose "Interface > Assign > WAN > Choose the new setup WAN with VLAN 5. After that it worked. Before also my problem was, that I have forgot to set the pfSense WAN into VLAN 5 and also this tagged/untagged thing was a bit confusing for me. In future I am going to learn more about this VLAN thing. Seams very interesting. Now it is working!!!

Again, thank you so much for you support and effort!

By the way. I also found another intersting solution from another guy: Have a look at this

He sets a different MAC Address on the real NIC so that pfSense can spoof the MAC adress from the real NIC.:

auto eth0
iface eth0 inet manual
pre-up ifconfig eth0 hw ether 00:11:22:33:44:55​

But this also didn't work for me, or I made a mistake somewhere while configuring it. What do you think about this solution?

Also I have had a look at the pfSense virtio NIC, there I can see, that it has a different MAC Address. This MAC Adress is the important thing for my Arris modem. At PVE boot the modem locks with real NIC MAC Adresse, of course. After that it isn't possible anymore to use the virtio MAC Adress. If i plug out and in my cable and go to pfsense Webinterface > Status > Interfaces > WAN Interface > click buttons "Release" after that "Renew", then is is working. Your managed switch solution bypasses the real NIC and with VLAN 5 it tunnels the virtio NIC. This trick is also from @lubberlick but without extra pressing those two buttons. Thank you also for this thread. Without your first investigation, I wouldn't have a solution yet!

I think this "pre-up ifconfig eth0 hw ether 00:11:22:33:44:55" thing can also a a fine solution, but at the moment I can't get it working. Anyway I am fine with my 28€ Netgear investment. Now I can play with VLAN and a working reboot fix for my server.

 

[Sharan H]

The pre-up command would absolutely work assuming that command actually changes the mac. I haven't tried it myself, but this would definitely be another workaround assuming the NIC accepts the command. I never had a compelling reason to change mac addresses on a linux machine.

If that command doesn't actually change the mac, maybe this one does?
ip link set eth0 address 00:11:22:33:44:55

Try running it from the proxmox host commandline to see if it works (without the pre-up).

As far as setting the vlan on pfsense, never occurred to me to set it there, which obviously works.

I think what you may have been missing from my instructions was not setting the VLAN Tag in the proxmox config in hardware options for the pfsense network hardware configuration on the wan port. This way, it wouldn't have required you to set vlan in pfsense host itself. Either way of course it works.

Glad to know you ultimately have a fix!