Clients in subnet B cannot ping clients in subnet A

[cmonty14]

Hello!

I have setup a network with 2 ISP, 2 gateways, 2 subnets (see attached graphic) on Proxmox VE host.
ISP A is connected to Fritz!Box (router A), ISP B is connected to eth0 of Proxmox VE host (router B).

Subnet A 192.168.178.0/24 with gateway 192.168.178.1 (= router A) is connected to eth1 of Proxmox VE host.

Virtual Bridge vmbr0 is configuring a private network (Subnet B) 10.0.0.0/24 according to instructions of Proxmox Network Model.
Any client in subnet B is routed to 10.0.0.1 (= router B).

Virtual Bridge vmbr1 is configuring the additional default gateway to force any client in subnet 192.168.178.0/24 to be routed to 192.168.178.1.

The routing table is this:

route -n
Kernel-IP-Routentabelle
Ziel            Router          Genmask         Flags Metric Ref    Use Iface
0.0.0.0         217.8.50.xx     0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
192.168.178.0   0.0.0.0         255.255.255.0   U     0      0        0 vmbr1
217.8.50.xx     0.0.0.0         255.255.255.192 U     0      0        0 eth0

ip route show
default via 217.8.50.xx dev eth0 
10.0.0.0/24 dev vmbr0  proto kernel  scope link  src 10.0.0.1 
192.168.178.0/24 dev vmbr1  proto kernel  scope link  src 192.168.178.10 
217.8.50.xx/26 dev eth0  proto kernel  scope link  src 217.8.50.xx

The problem is, that currently any client in subnet A 192.168.178.0/24 cannot ping, ssh another client in subnet B 10.0.0.0/24 and vice versa.
I assume I need another route to fix this.

If this is confirmed, there are two questions:
1. How should the route be defined? From subnet A (192.168.178.0/24) to subnet B (10.0.0.0/24)? Or from subnet B to subnet A?
2. Where must the route be created? On router A (= Fritz!Box) or router B (= Proxmox VE host)?

When I'm connected to Proxmox VE host I can ping, ssh any client in subnet A and subnet B.

THX

 

[Nemesiz]

If I`m not misleading sub-net A sends data to gateway A who is outside network range and sub-net B do the same with gateway B. So you have to add rule to routing table in VM so send from sub-net A to sub-net B not through gateway but through host. So try to add something like this to VM :

VM in sub-net A

#ip route add 10.0.0.0/24 via 192.168.178.10 dev eth0

VM in sub-net B

#ip route add 192.168.178.0/24 via 10.0.0.1 dev eth0

p.s. maybe net.ipv4.ip_forward=1 need in host to be set.

 

[wbumiller]

The above routes should cover the VM part. On the host you need to enable forwarding like above plus make sure the bridges have forwarding enabled as well (net.ipv4.conf.vmbr0.forwarding=1 and net.ipv4.conf.vmbr1.forwarding=1), also if both bridges are connected to the internet you'll want to make sure only those two subnets are routed with some rules in the FORWARD filter chain. Something like:

iptables -P FORWARD DROP
iptables -A FORWARD -i vmbr0 -s 10.0.0.0 -o vmbr1 -d 192.168.178.0/24 -j ACCEPT
iptables -A FORWARD -o vmbr0 -d 10.0.0.0 -i vmbr1 -s 192.168.178.0/24 -j ACCEPT

 

[cmonty14]

The above routes should cover the VM part. On the host you need to enable forwarding like above plus make sure the bridges have forwarding enabled as well (net.ipv4.conf.vmbr0.forwarding=1 and net.ipv4.conf.vmbr1.forwarding=1), also if both bridges are connected to the internet you'll want to make sure only those two subnets are routed with some rules in the FORWARD filter chain. Something like:

iptables -P FORWARD DROP
iptables -A FORWARD -i vmbr0 -s 10.0.0.0 -o vmbr1 -d 192.168.178.0/24 -j ACCEPT
iptables -A FORWARD -o vmbr0 -d 10.0.0.0 -i vmbr1 -s 192.168.178.0/24 -j ACCEPT

Hm... could you please specify where forwarding for bridges (net.ipv4.conf.vmbr0.forwarding=1) should be configured?

I have defined some rules in /etc/network/interfaces

auto vmbr0
iface vmbr0 inet static
        address  10.0.0.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o eth0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o eth0 -j MASQUERADE

Would you recommend to set the forward rules in this config file accordingly?
And would you recommend to define the iptables rules in this config file, too?

THX

 

[wbumiller]

I'd recommend replacing that echo command in the interface section you pasted with the one to enable bridge forwarding (ie use /proc/sys/net/ipv4/conf/vmbr0/forwarding as file instead of /proc/sys/net/ipv4/ip_forward and put the latter into /etc/sysctl.conf (with the dotted syntax net.ipv4.ip_forward=1). Mostly because the bridge entry won't exist before the bridge is created (so using /etc/sysctl.conf there might not work at boot-time) and because net.ipv4.ip_forward is a general option not related to a particular interface.

And yes, I suppose the forwarding rules would make sense in the interface section for the same reason: they apply to the bridge.

 

[cmonty14]

OK.
I have modified /etc/sysctl.conf and /etc/network/interfaces accordingly.

# The loopback network interface
auto lo
iface lo inet loopback

## unitymedia business static IPv4
auto eth0
iface eth0 inet dhcp
        post-up echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

auto eth1
iface eth1 inet manual

## LAN by pc4-gigabyte (Proxmox VE)
auto vmbr0
iface vmbr0 inet static
        address  10.0.0.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr0/forwarding
        post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o eth0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o eth0 -j MASQUERADE
        post-up   iptables -P FORWARD DROP
        post-up   iptables -A FORWARD -i vmbr0 -s 10.0.0.0 -o vmbr1 -d 192.168.178.0/24 -j ACCEPT
        post-up   iptables -A FORWARD -o vmbr0 -d 10.0.0.0 -i vmbr1 -s 192.168.178.0/24 -j ACCEPT


## LAN net1-fb6490 mit zweiter Routingtabelle
auto vmbr1
iface vmbr1 inet static
        address  192.168.178.10
        netmask  255.255.255.0
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr1/forwarding
        post-up ip route add 192.168.178.0/24 dev vmbr1 src 192.168.178.10 table rt2
        post-up ip route add default via 192.168.178.1 dev vmbr1 table rt2
        post-up ip rule add from 192.168.178.10/32 table rt2
        post-up ip rule add to 192.168.178.10/32 table rt2

Question:
Do you think the post-up rule
post-up iptables -P FORWARD DROP
is correct?
Or should this be a post-down rule?

THX