[SOLVED] proxmox post forwarding doesn't work // kimsufi

[um3n]

Hey Guys,

I try to build an environment with proxmox, hosted by kimsufi, with one external ip.
In this environment I installed a firewall and I want to get all traffic through the firewall.

Therefor I configured the following network interfaces:

auto lo
iface lo inet loopback

iface enp4s0 inet manual
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/enp4s0/proxy_arp

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet dhcp
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.42.0.254/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up iptables -t nat -A POSTROUTING -s '10.42.0.0/24' -o vmbr0 -j MASQUERADE
        post-up iptables -t nat -A PREROUTING -p tcp -d ex.te.rn.al --dport 443 -j DNAT --to-destination 10.42.0.1
        post-up iptables -t nat -A PREROUTING -p tcp -d ex.te.rn.al --dport 4444 -j DNAT --to-destination 10.42.0.1
#external

auto vmbr2
iface vmbr2 inet static
        address 10.42.20.254/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#internal

I also tried it with the iptables like that:

        post-up iptables -t nat -A POSTROUTING -s '10.42.0.0/24' -o vmbr0 -j MASQUERADE
        post-up iptables -t nat -A PREROUTING -p tcp -d ex.te.rn.al -m multiport ! --dport 22,8006 -j DNAT --to-destination 10.42.0.1

The external ip is bound on the interface vmbr0.

root@virtulator:~# iptables-save
# Generated by iptables-save v1.8.2 on Sun Dec 20 19:47:40 2020
*raw
:PREROUTING ACCEPT [65088:106935283]
:OUTPUT ACCEPT [43387:9004817]
COMMIT
# Completed on Sun Dec 20 19:47:40 2020
# Generated by iptables-save v1.8.2 on Sun Dec 20 19:47:40 2020
*filter
:INPUT ACCEPT [52437:104761064]
:FORWARD ACCEPT [5474:392261]
:OUTPUT ACCEPT [43438:9007605]
COMMIT
# Completed on Sun Dec 20 19:47:40 2020
# Generated by iptables-save v1.8.2 on Sun Dec 20 19:47:40 2020
*nat
:PREROUTING ACCEPT [7981:478653]
:INPUT ACCEPT [1505:70370]
:OUTPUT ACCEPT [106:6973]
:POSTROUTING ACCEPT [106:6973]
-A PREROUTING -d ex.te.rn.al/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.42.0.1
-A PREROUTING -d ex.te.rn.al/32 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.42.0.10:22
-A POSTROUTING -s 10.42.0.0/24 -o vmbr0 -j MASQUERADE
COMMIT
# Completed on Sun Dec 20 19:47:40 2020

root@virtulator:~# pveversion
pve-manager/6.3-3/eee5f901 (running kernel: 5.4.78-2-pve)

From the firewall iam able to ping the external interface, but nothing behind that. Neither can I reach my firewall or services behind it or next to it.

I appreciate every help if you guys need any further information, just ask!

 

[ph0x]

I'm didn't really dive into it, but if neither of your two bridges bridges any ethernet port, how should you reach any address connected to that bridges?
At least the external bridge should be connected to an ethernet port, I think.

 

[um3n]

the external link is connected to enp4s0 and the ip address is bound to the vmbr0 bridge.
I can communicate to external services on the pve node but not from any virtual machine.

 

[ph0x]

I guess this is because the kernel sets the default route via enp4s0/vmbr0.
Could you provide the output of ip route please?
And which bridge do your VMs use?

 

[ph0x]

After reading the issue on a bigger screen, I suppose your VMs are connected to one of the bridges and you're trying to convince the Proxmox host to forward the traffic to vmbr0. Is this correct?
Where does the firewall sit? Are we speaking about a VM with a firewall or the host's packet filter or another node between the Proxmox host and the internet?

 

[um3n]

I guess this is because the kernel sets the default route via enp4s0/vmbr0.
Could you provide the output of ip route please?
And which bridge do your VMs use?

Of Course:

root@virtulator:~# ip route
default via 87.98.219.254 dev vmbr0
10.42.0.0/24 dev vmbr1 proto kernel scope link src 10.42.0.254
10.42.20.0/24 dev vmbr2 proto kernel scope link src 10.42.20.254
ex.te.rn.al/24 dev vmbr0 proto kernel scope link src ex.te.rn.al
root@virtulator:~#

After reading the issue on a bigger screen, I suppose your VMs are connected to one of the bridges and you're trying to convince the Proxmox host to forward the traffic to vmbr0. Is this correct?
Where does the firewall sit? Are we speaking about a VM with a firewall or the host's packet filter or another node between the Proxmox host and the internet?

yes - this is the problem. There are two vms at the moment. One with a sophos utm (asg) and a normal linux client.
I can send Communication in this internal LANs but can not communicate with external networks over proxmox.

 

[ph0x]

I don't know what this iptables voodoo is for but you hopefully know.

You should give the sophos VM three network adapters, one for each bridge.
And the communication then gets configured inside this sophos VM, with the adapter for vmbr0 being WAN and the other two being LAN1 and LAN2, for example.

Did you already do this and now have these problems?

 

[um3n]

nope - I build this environment just a few days ago, it never worked. But a Colleague uses this configuration.
The iptables should send every messages to the defined ports to the firewall.

 

[ph0x]

Okay, although that might work (I'm absolutely no iptables guru), I suggest the way that I described. Then you have a "normal" firewall appliance, that you can manage via web GUI.
If you're not having any need for a stateful firewall you could also use the builtin Proxmox firewall (basically the packet filter of the Proxmox host).

 

[um3n]

and how can I send traffic to the firewall?

 

[ph0x]

From my experience all the traffic from the bridges gets sent to the gateway (therefore there can only be one). So this should happen automatically.

 

[um3n]

From my experience all the traffic from the bridges gets sent to the gateway (therefore there can only be one). So this should happen automatically.

Okay - i'll give it a try! I will do it tomorrow.

 

[ph0x]

Although I might also have a misconception here. Usually I define a dedicated bridge for inter-VM traffic (if needed) and have the traffic for the internet leave through a VLAN over vmbr0.
Internet traffic sent through the internal bridge would probably not be routed by the host. That is where your iptables voodoo might come into play, don't know.

 

[um3n]

kimsufi is working with dhcp on the internal interface. Don't know why, but it works.
I found an image in the so called internet - that is simular to my setup, just with a pfsense.



He also works with the nat rules.

 

[ph0x]

Ah, now I get it. They are necessary to still be able to reach your host per ssh and web gui.
But apart from that, it is like I described. A firewall VM which acts as a router for all other VMs (and also your host if wanted). Default route should then be via vmbr1, I suppose.

 

[um3n]

Ah, now I get it. They are necessary to still be able to reach your host per ssh and web gui.
But apart from that, it is like I described. A firewall VM which acts as a router for all other VMs (and also your host if wanted). Default route should then be via vmbr1, I suppose.

default route on what? The firewall?
I just played a littlebit and its confusing.

I set a defaultroute on the firewall vm that points to the leg from the proxmox (faked wan network).
From there it should be forwarded to the internet, but nothing happens...

I cleaned the iptables for the test... it should have worked for outgoing traffic right?

If I try to ping google.de from a bridgeinterface on the proxmox it also dont work

so it must be routing right?

I just want something like this

 

[ph0x]

Default route in the Proxmox host, I thought. Otherwise its traffic would not go through the firewall but directly through vmbr0.
It's a bit of a brain f*ck if Proxmox's firewall is a VM within Proxmox itself. I haven't tried that so far but there should be a few tutorials on that, because it's quite common from what I've read so far.

 

[bobmc]

I'm sure iptables can do this but if you're not really sure how then I would suggest using pfSense to handle all your traffic routing - it's a lot easier to setup and maintain

Have a look here

https://webworxshop.com/virtualised-pfsense-on-proxmox-with-open-vswitch/

 

[ph0x]

Thanks for the link, I was too lazy to search for one.

 

[um3n]

Okay guys,

I did it, okay a colleague of mine did the trick respectivly give the right hint.

I changed the two echo's for the interfaceconfiguration. I put them into the vmbr0 config.

auto vmbr0
iface vmbr0 inet dhcp
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr0/proxy_arp

Then I tested it with a normal lxc container, directly attached. The NAT-Rule not really different then the one in my first post.

auto vmbr1
iface vmbr1 inet static
        address 10.42.0.254/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up iptables -t nat -A POSTROUTING -s '10.42.0.0/24' -o vmbr0 -j MASQUERADE
        post-up iptables -t nat -A PREROUTING -p tcp -d ex.te.rn.al --dport 443 -j DNAT --to-destination 10.42.0.100

In the meanwhile, I added my sophos utm into the setup and everything is working like charm.
Thanks for the help! I hope, it helps some other dudes around the internet