pfSense on Proxmox using multiple physical interfaces.

[icmtf]

Hi!

Right... I need an adult to explain to me like I'm five years old what I'm doing wrong.


Current environment:
1. ISP router (network 192.168.5.0/24)
2. HP terminal server with 5 RJ-45:

1 Gbps Realtek (integrated NIC, enp2s0)​

1 Gbps Intel I350T4 (4-port-NIC, enp1s0f0..4) ​

3. Managed switch

ToDo environment:

• ISP router with network 192.168.5.0/24 has a DMZ device with IP 192.168.5.200 that should belong to enp2s0 (Intergrated Realtek on HP Terminal
• pfSense set up on Proxmox in a way that:
  
  • WAN link would be that particular Realtek NIC which would have this DMZ address set up as WAN IP
  • LAN link would be first RJ-45 port on I350T4 NIC with address 10.0.0.1/24
• Proxmox's IP would be 10.0.0.5/24 and accessible only from 10.0.0.0/24 (enp1s0f0)
• pfSense's IP would be naturally 10.0.0.1/24

pub IP ---> ISP --- 192.168.5.0/24 ---> [Proxmox(10.0.0.0.5 via Intel-p0), pfSense(192.168.5.200 via Realtek NIC, 10.0.0.1 via intel-p0)] --- 10.0.0.0/24 ----> Switch -> other devices

Problems

Scenario #1
I tried to set up following network on Proxmox:



pfSense hardware configured like this:



I'm ignoring the default 192.168.1.1/24 IP address assigned to LAN by pfSense - that's not relevant right now. The problem is that WAN link does not get the IP:



Additionally Proxmox dmesg throws this:

[ 2402.325304] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2412.309329] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2422.293374] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2432.281052] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2442.261542] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2452.245063] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2457.365260] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2463.255248] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).
[ 2473.237390] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).

So I manually set up WAN IP address on pfSense:



With upstream gateway address that points to ISP router which is 192.168.5.1/24
But dmesg continues to throw the same messages...



Questions

Can someone tell me what am I doing wrong?

Maybe what I'm trying to achieve is impossible?

 

[MarvinE]

Have you disabled Hardware Checksum Offloading ?

I think is necessary for pfSense on virtio.
https://docs.netgate.com/pfsense/en/latest/virtualization/virtio.html

 

[icmtf]

Have you disabled Hardware Checksum Offloading ?

I think is necessary for pfSense on virtio.
https://docs.netgate.com/pfsense/en/latest/virtualization/virtio.html

Yup, I did. I read a couple of guides and all of them were mentioning about disabling it.

 

[bobmc]

it should work as you expected, I have exactly this setup working at home and I have a couple of similar systems installed in satellite offices where I work.
The only difference would be that I usually use vmbr0 as the LAN (because that is setup by Proxmox on installation) and then I assign vmbr1 to another nic and use that for the WAN connection.

 

[vesalius]

How is the isp router assigning 192.168.5.200 to the enp2s0 (Intergrated Realtek on HP)? By MAC address? Couple of things to try,

1. Copy the MAC address the ISP router needs to assign the DMZ address to the WAN interface clone MAC address in pfSense and in vmbr0 Proxmox settings with the Realtek nic.
2. Take the Realtek nic out of the equation plug the isp router into the i350t4. Then use the appropriate enp1s0f1, 2 or 3 for vmbr0. Make sure to change the isp router dmz to this nic/mac as well and reboot the isp router.

 

[bobmc]

yeah, the mac-address of vmbr0 will be different to the mac-address of the realtek but you say it doesn't get a dhcp lease at all which suggests something else is involved. What does your network topology look like?