Proxmox + PFSense routing?

[lixaotec]

Dear folks,

I have a Proxmox with a PFSense VM and Im trying to reach PFSense through Proxmox shell.

The network is setup as follows:

ISP modem (with dhcp 192.168.100.1)

Proxmox connected to isp modem, static 192.168.100.10 (vmbr0)

PFsense a vm inside Proxmox, also gets an ip from isp modem (192.168.100.6) vmbr0

inside PFSense I created a NAT Port Forwarind to a linux VM, also inside proxmox (192.168.1.100) another iface, vmbr1 , and listen with netcat to test

When I use another machine that also connected to isp modem (192.168.100.12) I can do a netcat sucessfully.

However when I try from inside of proxmox shell to netcat that port, it does not works. Even traceroute does not works to PFSense ip. However it works if I tracerout to the outside machine.

What am I missing?

Thanks

 

[lixaotec]

ip route
default via 192.168.100.1 dev vmbr0 proto kernel onlink
192.168.100.0/24 dev vmbr0 proto kernel scope link src 192.168.100.10

 

[lixaotec]

root@*:~# ip route get 192.168.100.6
192.168.100.6 dev vmbr0 src 192.168.100.10 uid 0
cache
root@*:~# ip route get 192.168.100.12
192.168.100.12 dev vmbr0 src 192.168.100.10 uid 0
cache

 

[lixaotec]

traceroute 192.168.100.12
traceroute to 192.168.100.12 (192.168.100.12), 30 hops max, 60 byte packets
1 192.168.100.12 (192.168.100.12) 1.169 ms 1.123 ms 1.112 ms

traceroute 192.168.100.6 (runs but no route )*****

 

[bobmc]

Any device on the 192.168.100.0/24 network will be able to reach each other because they can establish connections via a broadcast request. But any requests destined for another network - e.g 192.168.1.100 will have to go through a routing device which in your case is the isp router - but the ISP router does not know how to reach 192.168.1.100 unless it has a specific instruction to direct them through 192.168.100.6 (the pFsense VM) which is as doing NAT for the other host.

traceroute to 192.168.100.6 probably fails because pFsense security is dropping the requests

you probably need an IP route setting up on your proxmox host to make the traffic flow work but that probably won't allow you to reach 192.168.1.100 from outside of your own network environment which is what I'm guessing you're try to achieve

 

[lixaotec]

Dear @bobmc, tks for replying

In fact its something on PFSense, i did a shell at PFSense and started a nc listening at some port, and frmo proxmox i could connect to it sucefully simply by using its isp router ip 192.168.100.6

So i guess its something inside PFSense.

The bad news for me is that I have tried everything I know, and a bunch of tips from internet, like nat reflection, assymetric nat.. none has solved it.

the pfsense is brand new installation, so no rules that could affect it.

i dont know much about routing, however I can see that connection is reaching pfsense now and it becomes SYS_SENT / CLOSED

What else could I try?

 

[lixaotec]

tcpdump at PFSense doenst show anything .. perhaps it mus be droping packages by some internal rule of pfsense.

 

[lixaotec]

 

[bobmc]

It would probably be easier to help if you could say what you are trying to achieve? what is your goal?

Also, how many physical network interfaces does your proxmox host have?

 

[lixaotec]

basically I have 2 physical nics, (configs bellow)

I wish to ssh into proxmox from outside, and tunneling a connection to a guest VM. that simple!

However the VMs are behing a PFSense, that is causing me this trouble

auto lo
iface lo inet loopback

iface enp5s0 inet manual
#PCI AsRock Gigabit

auto enx9cebe8c3766f
iface enx9cebe8c3766f inet manual
#USB Dell Gigabit

auto vmbr0
iface vmbr0 inet static
        address 192.168.100.10/24
        gateway 192.168.100.1
        netmask 255.255.255.0
        bridge-ports enp5s0 enx9cebe8c3766f
        bridge-stp off
        bridge-fd 0
        pre-up echo 2 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra

auto vmbr254
iface vmbr254 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#Host

auto vmbr50
iface vmbr50 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#Internet

auto vmbr1
iface vmbr1 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#Dev

 

[bobmc]

There's a very good 'how-to' get pfsense running under proxmox here - https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox.html

The key things to note are that conceptually the network flow is --> isp router --> pfsense --> proxmox and this is how I have my system setup at home and I can access my servers and data either using NAT or VPN from wherever I am and it works really well.

However, I have put my the running of my entire home network on Proxmox and the virtual hosts so if my proxmox server is down, no-one at home can access the internet which means I do have to plan any maintenance around the family otherwise I'm not very popular

My ISP router can be turned into a dumb modem - where it does not do wireless, dhcp, dns, or any firewall functions - it simply just passes everything in and out to the pfsense VM which does all of the functions apart from wireless - for which I have a couple of Unifi access points. This does simplify everything massively.

 

[lixaotec]

Thanks for the info @bobmc.

My isp modem unfortunately doesnt support that, and I ended up letting it as it is. My challenge is at with pfsense now, and my scenario requires me to access from some other machine at isp modem network.

What i did for now is to create another VM at vmbr0 and im 'jumping' from it to other networks.. until I can understand what is this pfsense selective behavior at nat port forward. (I´ve read something related to the fact that when there is the pfs managment interface on the way, it may cause that, ill try to dig more later) And I´ve also posted at pfsense forum ,.. lets see

Appreciate your effort in helping me.

Thanks

 

[RobFantini]

Hello lixaotec,
I am trying to get pfsense to work as a kvm , and having nat difficulties .
Did you end up getting pfsense to work as a KVM?

thanks.

 

[bobmc]

pfSense works fine in a KVM.

NAT depends on your network setup, and your goals. Can you describe them?

 

[RobFantini]

we have pf running on hardware [ for over 15 years ]

we are getting a new WAN connections and want to try pf as a kvm.

so I want to have our port forwards done same as on hardware. I have done that in the past.

this time instead of connecting WAN to a hardware port on one pve node , I am connecting it to a switch port at our cumulus linux switches. those are connected to by our 7 node pve cluster using lacp lag to a bridge on the switches. the bonds, vlans and wan ports all are part of the bridge.


I can connect to the pf from vlan-37 an vlan-10.

However can not get NAT to work.


I appreciate your help, let me know if I can send configuration parts to explain this a lot better then above.

 

[bobmc]

Things are certainly simpler if you can dedicate a WAN hardware port on a PVE node. Are you trying to implement a setup where pfsense can run on any of the nodes in the cluster?

 

[RobFantini]

yes high availability is important.

 

[bobmc]

I don't have any experience with HA Proxmox clusters so I can't really comment on that aspect but I do run serveral single node installations and they all work well.

Do you have pfsense installed and functional as a firewall/router for clients on your network, and is pfsense the final hop for traffic leaving your LAN?

 

[RobFantini]

pfsense LAN is in the same vlan as pve hosts .

perhaps it pfsense LAN should be in its own vlan ?

 

[bobmc]

Can pfsense reach the NAT hosts from the pve host VLAN?