Outgoing Proxmox IP address with PFSense and multiple Public IPs

[stormy27]

Hey all!

I've a PROXMOX server with many IP aliases on eth0 as eth0:0 , eth0:1, etc.
I installed a Pfsense on vmbr1 (considered as WAN with 10.0.0.x) and vmbr2 (considered as LAN with 192.168.9.x).

Everything is working fine for coming traffic from Internet: VM are accessible from outside, and responding with correct IP address.
Internet -> eth0 (or eth0:0, eth0:1, etc) to vmbr0 PROXMOX -> vmbr1 private Wan Pfsense (10.0.0.x) -> vmbr2 lan Pfsense (192.168.9.x) -> VM

BUT when a VM tries to go to outside world on Internet (ex: sending an email from a VM, etc.) then for this outgoing traffic ONLY the IP bare-metal server is used.
VM (ex:192.168.9.10) -> vmbr2 lan Pfsense (192.168.9.x) -> vmbr1 private Wan Pfsense (10.0.0.x) -> vmbr0 PROXMOX -> eth0 -> Internet (with eth0 IP ADDRESS)

I've no clue how to specify the outgoing IP address by selecting those from eth0:0, eth0:1, etc ????

I've searched in many way but didn't find the answer. Need to put entry in PROXMOX Firewall? What kind of entries? Another ideas??

Architecture is like this one: https://blog.zwindler.fr/2017/07/18/deploiement-de-proxmox-ve-5-sur-un-serveur-dedie-part-2/

Thanks for your feedback!

 

[imthiazaar]

Hi, Check my post and configuration.. it might help..

https://forum.proxmox.com/threads/network-setup-help.105689/

you are trying to achieve something similar..

 

[stormy27]

Hi!
I've checked your post but I didn't find similarity with your configuration.

My system is accepting entering IPs without any problem.

But the problem is related for OUTGOING IP that is systematic server main eth0 IP and not one that I may select (an eth0:0 or eth0:x alias).

I don't want my different VMs to send external data using only 1 IP for all my VMs!!! (I've 40 failover IPs, so one for mail messaging, one for another service, etc...)

Please help on this outgoing traffic please!

 

[imthiazaar]

At the end, gateway is the one that forwards to internet, which means all your private IP address sends its traffic to gateway from there its routed to internet via ISP's modem.

 

[The Merchant]

Hey all!

I've a PROXMOX server with many IP aliases on eth0 as eth0:0 , eth0:1, etc.
I installed a Pfsense on vmbr1 (considered as WAN with 10.0.0.x) and vmbr2 (considered as LAN with 192.168.9.x).

Everything is working fine for coming traffic from Internet: VM are accessible from outside, and responding with correct IP address.
Internet -> eth0 (or eth0:0, eth0:1, etc) to vmbr0 PROXMOX -> vmbr1 private Wan Pfsense (10.0.0.x) -> vmbr2 lan Pfsense (192.168.9.x) -> VM

BUT when a VM tries to go to outside world on Internet (ex: sending an email from a VM, etc.) then for this outgoing traffic ONLY the IP bare-metal server is used.
VM (ex:192.168.9.10) -> vmbr2 lan Pfsense (192.168.9.x) -> vmbr1 private Wan Pfsense (10.0.0.x) -> vmbr0 PROXMOX -> eth0 -> Internet (with eth0 IP ADDRESS)

I've no clue how to specify the outgoing IP address by selecting those from eth0:0, eth0:1, etc ????

I've searched in many way but didn't find the answer. Need to put entry in PROXMOX Firewall? What kind of entries? Another ideas??

Architecture is like this one: https://blog.zwindler.fr/2017/07/18/deploiement-de-proxmox-ve-5-sur-un-serveur-dedie-part-2/

Thanks for your feedback!

This is the exact situation i am in, and i have still not found a solution. I've been fighting with this for months, then gave up, and now im back lol.
If you managed to find a fix, please let me know.

 

[stormy27]

@boutzamat unfortunetaly not, searching again and again and it's becoming really frustrating....

Any chance to have someone having brillant ideas on this topic please???? Thx

 

[The Merchant]

@boutzamat unfortunetaly not, searching again and again and it's becoming really frustrating....

Any chance to have someone having brillant ideas on this topic please???? Thx

I managed to fix it in my case, hopefully you can use some of it.
My situation was kinda similar to yours. I have a main public IP and a public IP for mail, but the mail kept going out through the main ip.

In my case, i have 1 nic with 2 public IP's - no aliases (wouldn't work) so i had to use NAT to instruct my nic to forward traffic to specific vmbr's depending on incoming IP.

How i fixed it:
I have created vmbr0 for public ip 1 and vmbr1 for public ip 2. I NAT the incoming traffic on each ip to its respective vmbr.

auto enp1s0
iface enp1s0 inet static
address PUBLIC-IP-1
gateway PUBLIC-IP-1-GATEWAY
up ip addr PUBLIC-IP-2 dev enp1s0
down ip addr del PUBLIC-IP-2 dev enp1s0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A PREROUTING -i enp1s0 -d PUBLIC-IP-1 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 10.0.0.2
post-up iptables -t nat -A PREROUTING -i enp1s0 -d PUBLIC-IP-1 -p udp -j DNAT --to 10.0.0.2
post-up iptables -t nat -A PREROUTING -i enp1s0 -d PUBLIC-IP-2 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 10.0.1.2
post-up iptables -t nat -A PREROUTING -i enp1s0 -d PUBLIC-IP-2 -p udp -j DNAT --to 10.0.1.2

Note: You may want to ignore my dport (ignored ports) as i am not forwarding Proxmox GUI port or SSH so i can always access the hypervisor in case pfSense is down. They are restricted to my home and office IP anyway on server provider firewall level.
Don't leave them open as they might pose a security risk. But if you forward them to pfSense you might lock yourself out. Be mindfuld about this part.

The magic part:
At first, i used masquerade on my vmbr0 and vmbr1 to the enp1s0, but that ment all traffic automatically went out through my primary IP.
I then switched to SNAT and that was the trick. This allowed me to tell each vmbr what outgoing IP it should use, as you must pass a destination IP when you use SNAT. Bonus: SNAT apparently is even better than masquerade in terms of performance.
Doing this, suddenly it worked exactly as intended. Everything on vmbr0 used main ip and vmbr1 used secondary, just like i wanted.

auto vmbr0
iface vmbr0 inet static
address 10.0.0.1/29
bridge-ports none
bridge-stp off
bridge-fd 0
post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/29' -o enp1s0 -j SNAT --to-source PUBLIC-IP-1
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/29' -o enp1s0 -j SNAT --to-source PUBLIC-IP-1

auto vmbr1
iface vmbr1 inet static
address 10.0.1.1/29
bridge-ports none
bridge-stp off
bridge-fd 0
post-up iptables -t nat -A POSTROUTING -s '10.0.1.0/29' -o enp1s0 -j SNAT --to-source PUBLIC-IP-2
post-down iptables -t nat -D POSTROUTING -s '10.0.1.0/29' -o enp1s0 -j SNAT --to-source PUBLIC-IP-2

The pfSense part:
Naturally, vmbr0 was my WAN in pfSense, and i created a new interface (Interface->Assignments) and called it WAN1.
In this part, remember to also create a new gateway matching the gateway of vmbr1 (the vmbr1's own IP). Assign this new gateway to WAN1 in (Interface->Assignments->WAN1).




Now i have 2 WAN's with their own gateway. However, all outgoing connections still kept using the WAN for all my outgoing connections instead of WAN1 for my mail LAN - ofcourse, because pfSense doesn't know that you want to use another gatway for another LAN. Ideally i wanted all connections to go out out WAN and my mail LAN to go out through WAN1.

The trick:
In you mail LAN (or which ever LAN you want yo use another outgoing IP), in the "Allow" rule (Firewall->Rules->MAIL_LAN), scroll down to "Show advanced" and keep scrolling until you see "Gateway" - here you can select which gateway this specific LAN should use to access the internet.





Your quest should now be complete, and hopefully you are able to sleep better at nights - i was

Maybe this is not the 1:1 recipe for your situation, but hopefully you can use some of it and it might give you a better understanding of what needs to happen for it to work.
Proxmox itself is one thing. Once you get that working you should be set if you don't have any firewall between your connections. However, since we are using pfSense, we have to also configure pfSense to know what to do with which connections.

I hope this helps. You are welcome to DM me if you have any specific questions.

Best of luck

 

[stormy27]

Hi @boutzamat and all Proxmox fans !
Thanks a lot for your really interesting and descriptive content, it is definitely clear and may help lot of people on this difficul task!

But regarding my current problems there're some differences I'll explain to you, if you've ideas about how to solve them.
Your approach is great for a small number of Public IPs, but because I don't have only 2 IP, but 60 Public IPs to manage on my server I don't want to create 60 WAN network for proxmox hehe So I need to find another solution.

(1) public IP aliases on vmbr0 are working in my situation - I can use main IP or IP aliases as well, traffic is entering in PVE then PFSENSE... in that direction everything is fine. You'll find above a description of my current /etc/network/interfaces and also routes declared in reference#1.

(2) for entering traffic directed to my VMs, I'm using a Port Forward for 80/443 ports in order to throw all this traffic to a specific HAProxy VM. This one is then redirecting to specific VM as defined in HAProxy configuration depending of domain requested. It's OK on that side.
But here nevertheless I'm still not able to select a specific incoming traffic for a specific VM... maybe I'm wrong on the configuration itself.
Just see Reference (#2) for Port Forward and firewall rules for 80/443 incoming traffic

(3) I tried to declare a VIRTUAL IP for linking external Public address to PFSense VM Lan (192.168.9.x). So PFSense created a route for this entry as well. But I was unable to create firewall rules in order to tell PFSense what to do for incoming external traffic (ex: 92.xx.xx.4) directed to my internal Proxmox VM on LAN 192.168.9.x ... I tested NAT 1:1, Port Forward, etc... and I'm sure I'm wrong somewhere.
Using tcpdump I see the traffic up to PFSense them appearing as 10.0.0.2 ... and I don't know how to declare correctly the rules. That's why I put a global 80/443 port forward (as described above on topic 2) for an easy solution but that's not correct in my sense. See Reference #3.

I give you an overall picture in reference #4.
Hope you'll understand and you'll able to give me tips, because I'm lost:
- for using firewall rules: I'm not doing in a good manner and unable to apply PFSense rules as VM are declared with LAN address and even despite Virtual IP nothing is working.... I'm certainly stupid on this topic!
- for outgoing traffic from the VM itself: currently using the main vmbr0 IP and never the given IP alias on vmbr0 ... how to perform??

So, thanks guyz to help me, give me information, tips, etc... I'm ready to test everything!
Bye bye

PS: I've read many documentation, forum posts, etc.. and found information but never helping me to manage my current problem. Maybe I'm thinking in a wrong direction???!

PS2: also a link to the website I used to build this architecture, and here the given iptables file used: https://blog.zwindler.fr/2020/05/proxmox6_pfsense.txt



Reference (#1) /etc/network/interfaces
auto vmbr0
iface vmbr0 inet static
address 141.xx.xx.xx/24
gateway 141.xx.xx.254
bridge-ports eth0
bridge-stp off
bridge-fd 0
hwaddress d0:xx:xx:fb:c6:04
#BLOC1
up ip addr add 92.xx.xx.4/24 dev $IFACE label $IFACE:1
down ip addr del 92.xx..4/24 dev $IFACE label $IFACE:1
up ip addr add 92.xx.xx.5/24 dev $IFACE label $IFACE:2
down ip addr del 92.xx.xx.5/24 dev $IFACE label $IFACE:2
up ip addr add 92.xx.xx.6/24 dev $IFACE label $IFACE:3
down ip addr del 92.xx.xx.6/24 dev $IFACE label $IFACE:3
up ip addr add 92.xx.xx.7/24 dev $IFACE label $IFACE:4
down ip addr del 92.xx.xx.7/24 dev $IFACE label $IFACE:4
...


Then for Routes:
I created a route to throw all 192.168.9.0 packets through PFSense on 10.0.0.2 network.
ip route change 192.168.9.0/24 via 10.0.0.2 dev vmbr1

root@nsxxx19:/home/myssh# ip r
default via 141.xx.xx.254 dev vmbr0 proto kernel onlink
5.xx.xx.0/24 dev vmbr0 proto kernel scope link src 5.xx.xx.164
5.196.xx.0/24 dev vmbr0 proto kernel scope link src 5.196.xx.200
10.0.0.0/30 dev vmbr1 proto kernel scope link src 10.0.0.1
10.2.2.0/24 via 10.0.0.2 dev vmbr1
92.xx.xx.0/24 dev vmbr0 proto kernel scope link src 92.xx.xx.4
192.168.9.0/24 via 10.0.0.2 dev vmbr1
...



Reference (#2) Port Forward and Rules configuration to HAProxy on LAN 192.168.9.22






Reference (#3) Virtual IPs




Reference (#4) Overall picture of the architecture and IPs