[SOLVED] How to route host traffic over lxc?

[kenkin360]

.. sorry to re-post as the previous one get `Awaiting approval before being displayed publicly.` ; that might because of my revision

I got the situation that I'm unable to migrate my openwrt vm to lxc ..
I used the vm to route and manage the host's network via its virtual bridges but when I used the same configuration on lxc, it doesn't work.

/etc/hosts

127.0.0.1 localhost.localdomain localhost
192.168.132.4 pve

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

/etc/network/interfaces

iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.132.4/24
        gateway 192.168.132.1
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0

210.conf

arch: amd64
cores: 1
memory: 128
net0: name=eth0,bridge=vmbr1,hwaddr=CA:2B:9D:E6:52:08,type=veth
net1: name=eth1,bridge=vmbr0,hwaddr=FA:24:4E:32:4B:9B,type=veth
ostype: unmanaged
rootfs: datastore1:210/vm-210-disk-0.raw,size=204M
swap: 512

/etc/config/network of the OpenWrt container

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'wan'
    option ifname 'eth0'
    option proto 'dhcp'

config interface 'wan6'
    option proto 'dhcpv6'
    option ifname '@wan'
    option reqaddress 'try'
    option reqprefix 'auto'

config interface 'lan'
    option proto 'static'
    option ifname 'eth1'
    option type 'bridge'
    option netmask '255.255.255.0'
    option ipaddr '192.168.132.1'

My upstream router ip was 10.0.0.1
Ping from the container (192.168.132.1) to anywhere(including the host, upstream LAN and public) => works
Ping from the host (192.168.132.4) to 192.168.132.1 => works
Ping from the host (192.168.132.4) to external 10.0.0.1 => doesn't work
It was used to work on the vm, but seems not the same case on lxc.

Please shed some light on how to resolve this ..
Thank you

 

[kenkin360]

Is there more information or anything I should provide .. ? Any help please ..

 

[largem]

I have the exact the same issue.
Firstly, I noticed the iptables rule is empty in the openwrt lxc. (iptables -L returns empty). After login to the openwrt webUI, and applied the firewall configuration, iptables rules applied/shown up.
I don't know what I did in openwrt firewall settings, but the VM behind openwrt worked (internet accessible). After I restart the openwrt lxc, everything stopped working again. Since then, I still can not make it work.

 

[kenkin360]

I have the exact the same issue.
Firstly, I noticed the iptables rule is empty in the openwrt lxc. (iptables -L returns empty). After login to the openwrt webUI, and applied the firewall configuration, iptables rules applied/shown up.
I don't know what I did in openwrt firewall settings, but the VM behind openwrt worked (internet accessible). After I restart the openwrt lxc, everything stopped working again. Since then, I still can not make it work.

So I guess you've also tried lxc-net or `ifconfig vethXXXiX promisc` and everything as I did ..

 

[largem]

yes.

My network topology is below
upstream router: 192.168.1.1
proxmox host (single nic): 192.168.1.107
proxmox bridge network 0 (vmbr0): 192.168.1.107
proxmox bridge network 1 (vmbr1): 10.10.10.2
openwrt wan: on vmbr0, dhcp, 192.168.1.2
openwrt lan: on vmbr1, static ip, 10.10.10.1

Linux vm on vmbr1, dhcp, 10.10.10.173, gateway 10.10.10.1

from openwrt, I can ping 192.168.1.1
from linux vm, I can ping 192.168.1.107(proxmox vmbr0), 10.10.10.2(proxmox vmbr1), 10.10.10.1(openwrt lan), 192.168.1.2(openwrt wan),
but not 192.168.1.1

run tcpdump from proxmox host (dump traffic on vmbr0), when I ping 192.168.1.1 from Linux vm,
I see the icmp request from 10.10.10.173 -> 192.168.1.1, this seems not right, suppose to be 192.168.1.2 -> 192.168.1.1

 

[kenkin360]

yes.

My network topology is below
upstream router: 192.168.1.1
proxmox host (single nic): 192.168.1.107
proxmox bridge network 0 (vmbr0): 192.168.1.107
proxmox bridge network 1 (vmbr1): 10.10.10.2
openwrt wan: on vmbr0, dhcp, 192.168.1.2
openwrt lan: on vmbr1, static ip, 10.10.10.1

Linux vm on vmbr1, dhcp, 10.10.10.173, gateway 10.10.10.1

from openwrt, I can ping 192.168.1.1
from linux vm, I can ping 192.168.1.107(proxmox vmbr0), 10.10.10.2(proxmox vmbr1), 10.10.10.1(openwrt lan), 192.168.1.2(openwrt wan),
but not 192.168.1.1

run tcpdump from proxmox host (dump traffic on vmbr0), when I ping 192.168.1.1 from Linux vm,
I see the icmp request from 10.10.10.173 -> 192.168.1.1, this seems not right, suppose to be 192.168.1.2 -> 192.168.1.1

Do you really mean the vm or lxc .. ? I got no problem with using a normal vm. However, I can see some problem of your configuration if what you mean is really a normal vm. You can take my config as an example and change `enp3s0` to your physical nic's name and don't assign IP address on any physical nic. And then create a network device of `vmbr0` and use it as `lan` interface on openwrt. Your other vm(s) whatever they're of Linux or any other OS, just add an network device of `vmbr0` should suffice.
It used to work on normal vm, my guess is because the tap devices allow promiscuous mode and the Linux Bridge acts like a switch. The difficulty for me exists only on lxc.

 

[largem]

Do you really mean the vm or lxc .. ? I got no problem with using a normal vm. However, I can see some problem of your configuration if what you mean is really a normal vm. You can take my config as an example and change `enp3s0` to your physical nic's name and don't assign IP address on any physical nic. And then create a network device of `vmbr0` and use it as `lan` interface on openwrt. Your other vm(s) whatever they're of Linux or any other OS, just add an network device of `vmbr0` should suffice.
It used to work on normal vm, my guess is because the tap devices allow promiscuous mode and the Linux Bridge acts like a switch. The difficulty for me exists only on lxc.

The openwrt is lxc, testing machine behind openwrt is a Linux VM, I also have a testing machine is lxc.
I was using openwrt as VM, which did not have problem (just have CPU usage issue, which was the reason I am trying lxc).

Some mistyped information in my previous reply, I don't have IP on physical nic (Thanks for pointing out).
I have two Linux Bridge Networks defined in Proxmox, maybe just opposite to yours.

vmbr0 (public subnet), bind on physical nic, has the access to upstream (to internet)
vmbr1 (private subnet), for internal VMs
openwrt (lxc) uses both (of course), vmbr0 (wan), vmbr1 (lan).

I suspect the tap device too, but things puzzled me are I did make everything worked once (lxc scenario).

 

[kenkin360]

The openwrt is lxc, testing machine behind openwrt is a Linux VM, I also have a testing machine is lxc.
I was using openwrt as VM, which did not have problem (just have CPU usage issue, which was the reason I am trying lxc).

Some mistyped information in my previous reply, I don't have IP on physical nic (Thanks for pointing out).
I have two Linux Bridge Networks defined in Proxmox, maybe just opposite to yours.

vmbr0 (public subnet), bind on physical nic, has the access to upstream (to internet)
vmbr1 (private subnet), for internal VMs
openwrt (lxc) uses both (of course), vmbr0 (wan), vmbr1 (lan).

I suspect the tap device too, but things puzzled me are I did make everything worked once (lxc scenario).

It looks like that I asked an unsolvable question ..

 

[largem]

I only see a few people having this issue when I search the internet, and most of the post I read, people have good working lxc openwrt in proxmox. Also I made it work once (and only once).

The problem I see here is the test machine did send the package to openwrt lan port, but openwrt didn't change to its wan ip before sending out. I was using tcpdump on proxmox host to capture icmp packages, and compared between openwrt in lxc and openwrt in vm, you can clearly see the difference.

 

[kenkin360]

I only see a few people having this issue when I search the internet, and most of the post I read, people have good working lxc openwrt in proxmox. Also I made it work once (and only once).

The problem I see here is the test machine did send the package to openwrt lan port, but openwrt didn't change to its wan ip before sending out. I was using tcpdump on proxmox host to capture icmp packages, and compared between openwrt in lxc and openwrt in vm, you can clearly see the difference.

Most of the articles I've read don't share the same goal as what we are trying to do. They had the host traffic also routed directly to the upstream. I also thought I've achive our goal once but then I suspected it was because I didn't shutdown the vm one ...

 

[largem]

Finally, I think I made it work.
Add the following settings in /etc/config/firewall

config nat
        option name 'MASQUERADE'
        list proto 'all'
        option src 'lan'
        option target 'MASQUERADE'

There is still a small problem though, my firewall service does not persist the iptables rules, I have to restart the firewall service manually, some times I have to restart more than one time.

 

[kenkin360]

Finally, I think I made it work.
Add the following settings in /etc/config/firewall

config nat
        option name 'MASQUERADE'
        list proto 'all'
        option src 'lan'
        option target 'MASQUERADE'

There is still a small problem though, my firewall service does not persist the iptables rules, I have to restart the firewall service manually, some times I have to restart more than one time.

This doesn't work for me and I don't understand why this is even needed ..
For the problem you stated can be resolved by startup script `service firewall restart`; you might want to wait for the interfaces to bring up.

 

[largem]

This doesn't work for me and I don't understand why this is even needed ..
For the problem you stated can be resolved by startup script `service firewall restart`; you might want to wait for the interfaces to bring up.

Basically the rule is to replace the source ip from lan to the wan ip (MASQUERADE). I noticed such replacement was not happening when I tcpdump the traffic between test machine and upstream router. (ping). Or maybe you are facing other issue.

Also you could try manual iptables command,

iptables -t nat -A POSTROUTING -s "your lan ip CIDR" -o eth1 -j MASQERADE

eth1 is the wan interface in openwrt lxc.

Yes, I tried to put "service firewall restart" in /etc/rc.local, the doc said it will be executed at the end of boot process, but seems not helping, maybe it is too early.

 

[kenkin360]

Basically the rule is to replace the source ip from lan to the wan ip (MASQUERADE). I noticed such replacement was not happening when I tcpdump the traffic between test machine and upstream router. (ping). Or maybe you are facing other issue.

Also you could try manual iptables command,

iptables -t nat -A POSTROUTING -s "your lan ip CIDR" -o eth1 -j MASQERADE

eth1 is the wan interface in openwrt lxc.

Yes, I tried to put "service firewall restart" in /etc/rc.local, the doc said it will be executed at the end of boot process, but seems not helping, maybe it is too early.

Well I just solved it by simply restarting the firewall service until the default policies applied. I add the following statements to my startup script:

until $(iptables -t nat -L PREROUTING |grep -q 'zone_lan_prerouting')
do
    /etc/init.d/firewall restart
    sleep 1
done

So `/etc/rc.local` would look like this:

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

until $(iptables -t nat -L PREROUTING |grep -q 'zone_lan_prerouting')
do
    /etc/init.d/firewall restart
    sleep 1
done

exit 0

It seems the network changes are somehow not detected when it's a lxc and restarting of the service is therefore not triggered, no additional policies than the defaults are needed.