[SOLVED] Bridge not working - MASQUERADE - PFSENSE

Michael-stk · Dec 22, 2022

Hello everyone,

I've been trying to get proxmox working for 2 days now, watched 100 videos, read 100 different things online. many look alike but no matter what I do it just doesn't work.

My setup
Online VPS
Interface ens6
Public Ip of my VPS 85.123.123.100
Gateway of my VPS 85.123.123.1

What I want to create is the following

VPS >
Firewall PfSense >
Bridge? LAN 1 - VM 1 - VM 2 in a network of 192.168.1.2, 192.168.1.3 etc

I suspect that my hosting only allows 1 mac address on my server, so incoming and outgoing connections must go through my public ip and mac address

Hope someone can help me with this, I'm slowly going crazy

etc/network/interfaces

Code:

auto lo
iface lo inet loopback

auto ens6
iface ens6 inet manual

auto vmbr0
iface vmbr0 inet static
    address 85.123.123.100/32
    gateway 85.123.123.1
    bridge-ports ens6
    bridge-stp off
    bridge-fd 0
     post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up echo 1 > /proc/sys/net/ipv4/conf/ens6/proxy_arp

auto vmbr1
iface vmbr1 inet static
    address 192.168.1.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0

source-directory /etc/network/interfaces.d
source-directory /run/network/interfaces.d

Kind regards,
Michael

B.Otto · Dec 23, 2022

Hello,

you have configured ip-forwarding and proxy-arp but there is no NAT rule (masquerade). I would try something like that:

Code:

auto lo
iface lo inet loopback

auto ens6
iface ens6 inet manual

auto vmbr0
iface vmbr0 inet static
    address 85.123.123.100/32
    gateway 85.123.123.1
    bridge-ports ens6
    bridge-stp off
    bridge-fd 0


auto vmbr1
iface vmbr1 inet static
    address 192.168.1.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up   iptables -t nat -A POSTROUTING -s '192.168.1.0/24' -o vmbr0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.1.0/24' -o vmbr0 -j MASQUERADE
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up echo 1 > /proc/sys/net/ipv4/conf/ens6/proxy_arp

Then ifup vmbr1 or reboot the machine.

Also I can't remember whether you have to activate proxy_arp for the bridge vmbr0 or the interface ens6, might wanna try it with
post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr0/proxy_arp

Kind regards,
Benedikt

Michael-stk · Dec 26, 2022

Hi,

Sorry for my delay.

I've tested the above but can't connect at all.

I have now copied exactly the data from the manual for Masquerading, but it does not work with this either.

I have no idea what to do now.

My goal,

1 public ip, maximum 1 outgoing mac address.
Bridge wan to pfsense (Not set up at this time)
Bridge to all VMs

Onderstaande heb ik momenteel ingesteld in /etc/network/interfaces
(https://pve.proxmox.com/wiki/Networ...ith_tt_span_class_monospaced_iptables_span_tt)

Code:

auto lo
iface lo inet loopback

auto en6
#real IP address
iface en6 inet static
        address  85.XXX.XXX.222
        gateway  85.XXX.XXX.1

auto vmbr0
#private sub network
iface vmbr0 inet static
        address  10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        
        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

I have set up a debian 11 VM with the following settings in
/etc/network/interfaces I can't get a connection when I want to update but also pinging to 8.8.8.8 doesn't work

Code:

auto lo
iface lo inet loopback

allow-hotplug ens18
iface en18 inet static
        address 10.10.10.10/24
        gateway 10.10.10.1
        dns-nameservers 10.10.10.1

Please your advice

Michael-stk · Dec 26, 2022

Code:

root@vps:~# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
root@vps:~# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 10.10.10.0/24 -o eno1 -j MASQUERADE
root@vps:~# iptables -t raw -S
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
-A PREROUTING -i fwbr+ -j CT --zone 1

root@vps:~# pve-firewall status
Status: disabled/running

budder · Dec 26, 2022

Michael-stk said:

eno1 is not your network device. Given the config you posted it should be ens6.

Best regard

budder

Michael-stk · Dec 27, 2022

Thank you Budder for your response, I read over that. Thank you!

But unfortunately, not working

Code:

auto lo
iface lo inet loopback

auto ens6
iface ens6 inet static
    address 85.123.123.123/24
    gateway 85.123.123.1

auto vmbr0
iface vmbr0 inet static
    address 10.10.10.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
 
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o ens6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o ens6 -j MASQUERADE
        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

Code:

root@vps:~# systemctl status networking ; ip -c a
● networking.service - Network initialization
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2022-12-26 22:05:22 UTC; 16h ago
       Docs: man:interfaces(5)
             man:ifup(8)
             man:ifdown(8)
    Process: 437 ExecStart=/usr/share/ifupdown2/sbin/start-networking start (code=exited, status=0/SUCCESS)
   Main PID: 437 (code=exited, status=0/SUCCESS)
        CPU: 422ms

Dec 26 22:05:22 vps.domain.nl systemd[1]: Starting Network initialization...
Dec 26 22:05:22 vps.domain.nl networking[437]: networking: Configuring network interfaces
Dec 26 22:05:22 vps.domain.nl systemd[1]: Finished Network initialization.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:01:95:12:3c:03 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 85.123.123.123/24 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::1:95ff:fe19:3c08/64 scope link
       valid_lft forever preferred_lft forever
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5a:4c:71:52:84:20 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.1/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::584c:71ff:fe55:8420/64 scope link
       valid_lft forever preferred_lft forever
13: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether b6:46:84:28:27:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::b446:54ff:fe27:27b4/64 scope link
       valid_lft forever preferred_lft forever
26: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether f2:76:cb:be:2c:ea brd ff:ff:ff:ff:ff:ff
27: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:a9:25:9a:56:d5 brd ff:ff:ff:ff:ff:ff
28: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 9e:61:10:aa:35:53 brd ff:ff:ff:ff:ff:ff
29: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether b6:25:09:59:7c:18 brd ff:ff:ff:ff:ff:ff

Code:

root@vps:~# iptables -t raw -S
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
-A PREROUTING -i fwbr+ -j CT --zone 1
-A PREROUTING -i fwbr+ -j CT --zone 1
root@vps:~# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 10.10.10.0/24 -o ens6 -j MASQUERADE
-A POSTROUTING -s 10.10.10.0/24 -o ens6 -j MASQUERADE
root@vps:~# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

Setup debian 10 and 11
/ect/net.../interfaces

Code:

auto lo
iface lo inet loopback

allow-hotplug ens18
iface ens18 inet static
    address 10.10.10.100/24
    gateway 10.10.10.1
    dns-nameservers 8.8.8.8 (also tryed 10.10.10.1 as default)

Hope someone can help me with this, all manuals online say something different but nothing helps.

budder · Dec 27, 2022

Is vmbr0 set as the bridge for the container? Your config seems to be correct (unless I missed something).

I'll just post my config as it is in fact working (on a dedicated server) so you can compare it to yours (IPv4 NAT, routed setup, static IPv6 from a subnet). Routed setup should even work if your host is blocking virtual MAC-Addresses.

Code:

auto lo

iface lo inet loopback





auto eno1 #Change that to your network device

iface eno1 inet static

        address host_public_ipv4/27

        gateway host_public_gateway



#IPv6 Connectivity (setup for static ipv6 on vm's) might not be requird for your setup

iface eno1 inet6 static

        address host_public_ipv6/128

        gateway ipv6_gateway

        post-up sleep 5; /sbin/ip -6 route add aaaa:bbbb:cccc:ddff:ff:ff:ff:ff dev eno1

        post-up sleep 5; /sbin/ip -6 route add default via aaaa:bbbb:cccc:ddff:ff:ff:ff:ff

        pre-down /sbin/ip -6 route del default via aaaa:bbbb:cccc:ddff:ff:ff:ff:ff

        pre-down /sbin/ip -6 route del aaaa:bbbb:cccc:ddff:ff:ff:ff:ff dev eno1



#bridged network used for lxc/kvm

auto vmbr6

iface vmbr6 inet static

        address 10.0.0.254/24

        bridge-ports none

        bridge-stp off

        bridge-fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward

        post-up iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eno1 -j MASQUERADE

        post-down iptables -t nat -D POSTROUTING -s 10.0.0.0/24 -o eno1 -j MASQUERADE



iface vmbr6 inet6 static

        address one_ipv6_from_subnet (diffrent to host_public_ipv6)

Try a reboot if everything seems to be correct and create a new instance listening on your bridge, for my config vmbr6 (yours might differ depending on your config).

Best regards

budder

Michael-stk · Dec 30, 2022

Thank you Buddy for your response,

I reinstalled everything and with your settings it didn't seem to work at first, but after a long search it turned out to be my DNS.

I have now got everything working including pfsense.

Thank you so much for your help, one frustration less so for the new year

Search

Search

[SOLVED] Bridge not working - MASQUERADE - PFSENSE

Michael-stk

Member

B.Otto

Active Member

Michael-stk

Member

Michael-stk

Member

budder

New Member

Michael-stk

Member

budder

New Member

Michael-stk

Member

We value your privacy