DHCP on pfSense VM not handing out addresses past the host

C

crowax

New Member
Jan 9, 2022
4
0
1
45
I have built a pfsense server as a VM on one of my ProxMox hosts. I have two hosts and plan to have HA pfsense on the second host. Most everything appears to be working like the NIC passthrough for WAN and the bridge for local traffic. Physical computers outside the ProxMox environment connect to the switch are able to go out of the network just fine through the firewall.

However when I turn on DHCP server on pfsense, it only hands out IP address to the other VMs on the host. The second ProxMox host and all physical servers do not get an IP address. I can manually add one and everything runs fine after that. I have two switches (mikrotik 10Gb and Cisco 1Gb PoE) which I have directly connected the ProxMox to just to verify there wasn't a switch issue blocking things. I also saw it could be possible that promiscuous mode needed to be enabled on the bridge so I did that without any luck.

Any other thoughts or ideas? Thanks in advance!
 
Could you post the output of ip a on the pfsense VM?
 
Hannes Laimer said:
Could you post the output of ip a on the pfsense VM?
Click to expand...
I couldn't get ip a to work. Maybe because pfsense if freebsd? I did a ipfonfig in case that gives similar information. vnet0 is the LAN side. The WAN side is currently unplugged because without the DHCP working I have to keep it on another router. But I hook it back up for testing.

Code: 
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether ca:cb:0e:76:c0:54
        inet6 fe80::c8cb:eff:fe76:c054%vtnet0 prefixlen 64 scopeid 0x1
        inet 192.168.86.2 netmask 0xffffff00 broadcast 192.168.86.255
        inet 192.168.86.7 netmask 0xffffff00 broadcast 192.168.86.255 vhid 1
        carp: MASTER vhid 1 advbase 1 advskew 0
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether ba:99:a5:81:d4:72
        inet6 fe80::b899:a5ff:fe81:d472%vtnet1 prefixlen 64 scopeid 0x2
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
 
You're right ifconfig not ip a. So, pfsense is handing out IPs only on vtnet0. On WAN you usually get an IP and not hand IPs out. You need another interface that connects to your physical network, not just the bridge the other VMs use. That interface would then hand out IPs to your physical network.
 
Last edited:
Hannes Laimer said:
You're right ifconfig not ip a. So, pfsense is handing out IPs only on vtnet0. On WAN you usually get an IP and not hand IPs out. You need another interface that connects to your physical network, not just the bridge the other VMs use. That interface would then hand out IPs to your physical network.
Click to expand...
Understood. So even though my physical machines can see the pfSense on the network through the bridge, it doesn't hand out DHCP requests through that? If I connect it to the physical, do I still need the bridge as well?

The reason I haven't gone that route is because I don't have extra NIC's and no room for a card, so I may need to upgrade my hosts to get that working.
 
Hello,

I have a mini pc with only 2 RJ45 ports, proxmox + pfsense installed on it.

First port : enp46s0 is the wan port
Second one : enp45s0 is the lan port

2 Linux bridges created:
vmbr0 bridged with enp46s0
vmbr1 bridged with enp45s0

My VMs receive ip configuration from DHCP when connected to vmbr1, whereas the devices connected on enp45s0 port doesn't receive anything from DHCP server. I have to enter config manually.

I found this topic. What's the solution ?
 
Last edited:
I have spent days and multiple full reinstalls on this. For the benefit of others, I'll add the details here.

I set up a new Proxmox on a new Intel Nuc with dual I225-LM nics and added pfSense. (spoiler alert - it is the nics that are the issue).
Hardware (https://ark.intel.com/content/www/us/en/ark/products/212518/intel-nuc-11-pro-kit-nuc11tnhv50l.html)

I created a Linux Bridge and set the port to one of the two nics.

Set up pfSense and turned on DHCP.

VMs linked to the bridge were receiving IPs from the DHCP but any physical PC/device on the LAN were not. However, if a physical device had a static IP, pfSense would route the traffic correctly. It is only DHCP offers that are being dropped.

Using WireShark on a laptop on the LAN, I could see multiple devices requesting DHCP but the server did not respond.
Using a Packet Capture on pfSense, I could see the requests being received and pfSense DHCP respond with the DHCP offer.

The offer was received by any VM on the bridge but never made it to the physical LAN through the port.

Things I tried:
- swapping nics
- moving management onto a third USB nic as a temporary solution
- turning on and off promiscuous mode
- using an OVS bridge
- trying to force broadcast on the DHCP offers even though the DHCP requests asked for unicast.


What worked:
- I set my LAN on the USB nic and linked it to the bridge. The 2 I225-LMs are for WAN and Management (which can be static or get a DHCP address from another server).
- VMs on the Bridged LAN now get DHCP offers and Physical PCs also get DHCP offers.

The issue:
vPro on these cards is the issue. Here is a quote from Intel testing on this page
https://forum.netgate.com/topic/175592/can-not-get-dhcp-leases-on-new-intel-i225-lm-based-machine/13
We have discovered LAN1 (Intel I225-LM with vPro Essentials) could not work as DHCP server properly.
When we switch to LAN2 (Intel I225-LM without vPro), the client PC could receive DHCPOFFER (assigned IP).
After some further testing, we found the symptom is related to vPro (AMT) function of I225-LM in general.
Currently, we are still co-working with Intel to solve this symptom.
If there is any update about the solution or conclusion, we will inform you as soon as possible.
Click to expand...

In the latest version of Proxmox, the icg driver is version 6.2.16 and Intel report that the latest version should be 6.3.11 (https://www.kernelconfig.io/config_igc) but compiling drivers is not something I am comfortable with right now.

USB Nic:
I think any USB nic will solve this until the icg drivers are updated. But this is the one I used.
www.amazon.com.au/dp/B07M967CRD

Hope that helps.

Matt.
 
  • Like
Reactions: cfalcken, Maximus3485, adrianh85 and 1 other person
Yes, I worked through that many times as I was trying to solve this. This particular issue - which I described in my post above - is caused by a fault in the NIC driver for the Intel 2.5GB nic and only affects the DHCP responses (offers) from the VM pfSense server to the physical lan if that nic is being used. I can not see in that documentation where this issue is addressed.
 
madmat777 said:
Yes, I worked through that many times as I was trying to solve this. This particular issue - which I described in my post above - is caused by a fault in the NIC driver for the Intel 2.5GB nic and only affects the DHCP responses (offers) from the VM pfSense server to the physical lan if that nic is being used. I can not see in that documentation where this issue is addressed.
Click to expand...
There might be a bug surrounding that issue...seen it discussed on pfSense forum. My hardware has that NIC as on-board, and I only used it for Proxmox...no issues.
 
Nollimox said:
There might be a bug surrounding that issue...seen it discussed on pfSense forum. My hardware has that NIC as on-board, and I only used it for Proxmox...no issues.
Click to expand...
If you're using pfSense in a VM on Proxmox and handing out IP addresses on the bridged physical LAN then this must be a strange hardware glitch. It took a lot to diagnose and it was only that post on the Netgate site that led me to try the LAN bridged to a USB nic which worked. Hopefully, this helps anyone else that hits this issue as there is not a lot of information on it but the OP was one that I found while searching.
 
@madmat777
Thank you m8!
I have lost two days trying to troubleshoot this issue with Proxmox 8.1.4, Mikrotik CHR and two Intel I226 nics. One LM and one V.
No matter what I tried, even with AMT off in BIOS, I226 LM does not work for DHCP. Leases were not bound no matter what.
Changed to Intel X710 nics and the problem went away.
 
I'm having a similar issue while running FreeBSD 14.0 on PVE 8.1.4 in a Q35 VM with OVMF (UEFI), and a VirtIO NIC attached to a Proxmox simple SNAT SDN running DHCPv4. When the VM is booted with an Xubuntu 22.04.4 live ISO it quickly gets an address via DHCPv4 and can access the internet. When booted with FreeBSD 14.0 the VM does not receive a DHCPv4 response, although the attempts can be seen in the CLI.
I think the issue might be attributed to FreeBSD drivers, which would also affect pfSense and OPNsense.
 
madmat777 said:
I have spent days and multiple full reinstalls on this. For the benefit of others, I'll add the details here.

I set up a new Proxmox on a new Intel Nuc with dual I225-LM nics and added pfSense. (spoiler alert - it is the nics that are the issue).
Hardware (https://ark.intel.com/content/www/us/en/ark/products/212518/intel-nuc-11-pro-kit-nuc11tnhv50l.html)

I created a Linux Bridge and set the port to one of the two nics.

Set up pfSense and turned on DHCP.

VMs linked to the bridge were receiving IPs from the DHCP but any physical PC/device on the LAN were not. However, if a physical device had a static IP, pfSense would route the traffic correctly. It is only DHCP offers that are being dropped.

Using WireShark on a laptop on the LAN, I could see multiple devices requesting DHCP but the server did not respond.
Using a Packet Capture on pfSense, I could see the requests being received and pfSense DHCP respond with the DHCP offer.

The offer was received by any VM on the bridge but never made it to the physical LAN through the port.

Things I tried:
- swapping nics
- moving management onto a third USB nic as a temporary solution
- turning on and off promiscuous mode
- using an OVS bridge
- trying to force broadcast on the DHCP offers even though the DHCP requests asked for unicast.


What worked:
- I set my LAN on the USB nic and linked it to the bridge. The 2 I225-LMs are for WAN and Management (which can be static or get a DHCP address from another server).
- VMs on the Bridged LAN now get DHCP offers and Physical PCs also get DHCP offers.

The issue:
vPro on these cards is the issue. Here is a quote from Intel testing on this page
https://forum.netgate.com/topic/175592/can-not-get-dhcp-leases-on-new-intel-i225-lm-based-machine/13


In the latest version of Proxmox, the icg driver is version 6.2.16 and Intel report that the latest version should be 6.3.11 (https://www.kernelconfig.io/config_igc) but compiling drivers is not something I am comfortable with right now.

USB Nic:
I think any USB nic will solve this until the icg drivers are updated. But this is the one I used.
www.amazon.com.au/dp/B07M967CRD

Hope that helps.

Matt.
Click to expand...
Thanks a lot, this was my issue. I'm using a miniforum MS-01 and it comes with one I226-V and one I226-LM. Switching the two interfaces helped me. The I226-V isn't affected and can be used for DHCP in the bridge mode. The USB Nic solution also worked.
 
  • Like
Reactions: cfalcken
Thanks madmat777 for your detailed analysis, and thanks maximus3485 for pointing to the minisforum MS-01. That was exactly my issue as well, and switching the interfaces was the end of several hours of troubleshooting. Would not have thought that this could be a hardware issue on just one of the two NICs ...
 
You must log in or register to reply here.
Top Bottom