[SOLVED] How to route host traffic over lxc?

K

kenkin360

New Member
Oct 12, 2021
10
4
3
43
.. sorry to re-post as the previous one get `Awaiting approval before being displayed publicly.` ; that might because of my revision

I got the situation that I'm unable to migrate my openwrt vm to lxc ..
I used the vm to route and manage the host's network via its virtual bridges but when I used the same configuration on lxc, it doesn't work.

/etc/hosts

Code: 
127.0.0.1 localhost.localdomain localhost
192.168.132.4 pve

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

/etc/network/interfaces

Code: 
iface enp3s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.132.4/24
        gateway 192.168.132.1
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0

210.conf

Code: 
arch: amd64
cores: 1
memory: 128
net0: name=eth0,bridge=vmbr1,hwaddr=CA:2B:9D:E6:52:08,type=veth
net1: name=eth1,bridge=vmbr0,hwaddr=FA:24:4E:32:4B:9B,type=veth
ostype: unmanaged
rootfs: datastore1:210/vm-210-disk-0.raw,size=204M
swap: 512


/etc/config/network of the OpenWrt container

Code: 
config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'wan'
    option ifname 'eth0'
    option proto 'dhcp'

config interface 'wan6'
    option proto 'dhcpv6'
    option ifname '@wan'
    option reqaddress 'try'
    option reqprefix 'auto'

config interface 'lan'
    option proto 'static'
    option ifname 'eth1'
    option type 'bridge'
    option netmask '255.255.255.0'
    option ipaddr '192.168.132.1'

My upstream router ip was 10.0.0.1
Ping from the container (192.168.132.1) to anywhere(including the host, upstream LAN and public) => works
Ping from the host (192.168.132.4) to 192.168.132.1 => works
Ping from the host (192.168.132.4) to external 10.0.0.1 => doesn't work
It was used to work on the vm, but seems not the same case on lxc.

Please shed some light on how to resolve this ..
Thank you
 
Is there more information or anything I should provide .. ? Any help please ..
 
Last edited:
I have the exact the same issue.
Firstly, I noticed the iptables rule is empty in the openwrt lxc. (iptables -L returns empty). After login to the openwrt webUI, and applied the firewall configuration, iptables rules applied/shown up.
I don't know what I did in openwrt firewall settings, but the VM behind openwrt worked (internet accessible). After I restart the openwrt lxc, everything stopped working again. Since then, I still can not make it work.
 
Last edited:
  • Like
Reactions: kenkin360
largem said:
I have the exact the same issue.
Firstly, I noticed the iptables rule is empty in the openwrt lxc. (iptables -L returns empty). After login to the openwrt webUI, and applied the firewall configuration, iptables rules applied/shown up.
I don't know what I did in openwrt firewall settings, but the VM behind openwrt worked (internet accessible). After I restart the openwrt lxc, everything stopped working again. Since then, I still can not make it work.
Click to expand...
So I guess you've also tried lxc-net or `ifconfig vethXXXiX promisc` and everything as I did ..
 
yes.

My network topology is below
upstream router: 192.168.1.1
proxmox host (single nic): 192.168.1.107
proxmox bridge network 0 (vmbr0): 192.168.1.107
proxmox bridge network 1 (vmbr1): 10.10.10.2
openwrt wan: on vmbr0, dhcp, 192.168.1.2
openwrt lan: on vmbr1, static ip, 10.10.10.1

Linux vm on vmbr1, dhcp, 10.10.10.173, gateway 10.10.10.1

from openwrt, I can ping 192.168.1.1
from linux vm, I can ping 192.168.1.107(proxmox vmbr0), 10.10.10.2(proxmox vmbr1), 10.10.10.1(openwrt lan), 192.168.1.2(openwrt wan),
but not 192.168.1.1

run tcpdump from proxmox host (dump traffic on vmbr0), when I ping 192.168.1.1 from Linux vm,
I see the icmp request from 10.10.10.173 -> 192.168.1.1, this seems not right, suppose to be 192.168.1.2 -> 192.168.1.1
 
largem said:
yes.

My network topology is below
upstream router: 192.168.1.1
proxmox host (single nic): 192.168.1.107
proxmox bridge network 0 (vmbr0): 192.168.1.107
proxmox bridge network 1 (vmbr1): 10.10.10.2
openwrt wan: on vmbr0, dhcp, 192.168.1.2
openwrt lan: on vmbr1, static ip, 10.10.10.1

Linux vm on vmbr1, dhcp, 10.10.10.173, gateway 10.10.10.1

from openwrt, I can ping 192.168.1.1
from linux vm, I can ping 192.168.1.107(proxmox vmbr0), 10.10.10.2(proxmox vmbr1), 10.10.10.1(openwrt lan), 192.168.1.2(openwrt wan),
but not 192.168.1.1

run tcpdump from proxmox host (dump traffic on vmbr0), when I ping 192.168.1.1 from Linux vm,
I see the icmp request from 10.10.10.173 -> 192.168.1.1, this seems not right, suppose to be 192.168.1.2 -> 192.168.1.1
Click to expand...
Do you really mean the vm or lxc .. ? I got no problem with using a normal vm. However, I can see some problem of your configuration if what you mean is really a normal vm. You can take my config as an example and change `enp3s0` to your physical nic's name and don't assign IP address on any physical nic. And then create a network device of `vmbr0` and use it as `lan` interface on openwrt. Your other vm(s) whatever they're of Linux or any other OS, just add an network device of `vmbr0` should suffice.
It used to work on normal vm, my guess is because the tap devices allow promiscuous mode and the Linux Bridge acts like a switch. The difficulty for me exists only on lxc.
 
Last edited:
  • Like
Reactions: largem
kenkin360 said:
Do you really mean the vm or lxc .. ? I got no problem with using a normal vm. However, I can see some problem of your configuration if what you mean is really a normal vm. You can take my config as an example and change `enp3s0` to your physical nic's name and don't assign IP address on any physical nic. And then create a network device of `vmbr0` and use it as `lan` interface on openwrt. Your other vm(s) whatever they're of Linux or any other OS, just add an network device of `vmbr0` should suffice.
It used to work on normal vm, my guess is because the tap devices allow promiscuous mode and the Linux Bridge acts like a switch. The difficulty for me exists only on lxc.
Click to expand...
The openwrt is lxc, testing machine behind openwrt is a Linux VM, I also have a testing machine is lxc.
I was using openwrt as VM, which did not have problem (just have CPU usage issue, which was the reason I am trying lxc).

Some mistyped information in my previous reply, I don't have IP on physical nic (Thanks for pointing out).
I have two Linux Bridge Networks defined in Proxmox, maybe just opposite to yours.

vmbr0 (public subnet), bind on physical nic, has the access to upstream (to internet)
vmbr1 (private subnet), for internal VMs
openwrt (lxc) uses both (of course), vmbr0 (wan), vmbr1 (lan).

I suspect the tap device too, but things puzzled me are I did make everything worked once (lxc scenario).
 
Last edited:
  • Like
Reactions: kenkin360
largem said:
The openwrt is lxc, testing machine behind openwrt is a Linux VM, I also have a testing machine is lxc.
I was using openwrt as VM, which did not have problem (just have CPU usage issue, which was the reason I am trying lxc).

Some mistyped information in my previous reply, I don't have IP on physical nic (Thanks for pointing out).
I have two Linux Bridge Networks defined in Proxmox, maybe just opposite to yours.

vmbr0 (public subnet), bind on physical nic, has the access to upstream (to internet)
vmbr1 (private subnet), for internal VMs
openwrt (lxc) uses both (of course), vmbr0 (wan), vmbr1 (lan).

I suspect the tap device too, but things puzzled me are I did make everything worked once (lxc scenario).
Click to expand...
It looks like that I asked an unsolvable question ..
 
I only see a few people having this issue when I search the internet, and most of the post I read, people have good working lxc openwrt in proxmox. Also I made it work once (and only once).

The problem I see here is the test machine did send the package to openwrt lan port, but openwrt didn't change to its wan ip before sending out. I was using tcpdump on proxmox host to capture icmp packages, and compared between openwrt in lxc and openwrt in vm, you can clearly see the difference.
 
  • Like
Reactions: kenkin360
largem said:
I only see a few people having this issue when I search the internet, and most of the post I read, people have good working lxc openwrt in proxmox. Also I made it work once (and only once).

The problem I see here is the test machine did send the package to openwrt lan port, but openwrt didn't change to its wan ip before sending out. I was using tcpdump on proxmox host to capture icmp packages, and compared between openwrt in lxc and openwrt in vm, you can clearly see the difference.
Click to expand...
Most of the articles I've read don't share the same goal as what we are trying to do. They had the host traffic also routed directly to the upstream. I also thought I've achive our goal once but then I suspected it was because I didn't shutdown the vm one ...
 
Finally, I think I made it work.
Add the following settings in /etc/config/firewall

Code: 
config nat
        option name 'MASQUERADE'
        list proto 'all'
        option src 'lan'
        option target 'MASQUERADE'

There is still a small problem though, my firewall service does not persist the iptables rules, I have to restart the firewall service manually, some times I have to restart more than one time.
 
  • Like
Reactions: kenkin360
largem said:
Finally, I think I made it work.
Add the following settings in /etc/config/firewall

Code: 
config nat
        option name 'MASQUERADE'
        list proto 'all'
        option src 'lan'
        option target 'MASQUERADE'

There is still a small problem though, my firewall service does not persist the iptables rules, I have to restart the firewall service manually, some times I have to restart more than one time.
Click to expand...
This doesn't work for me and I don't understand why this is even needed ..
For the problem you stated can be resolved by startup script `service firewall restart`; you might want to wait for the interfaces to bring up.
 
kenkin360 said:
This doesn't work for me and I don't understand why this is even needed ..
For the problem you stated can be resolved by startup script `service firewall restart`; you might want to wait for the interfaces to bring up.
Click to expand...
Basically the rule is to replace the source ip from lan to the wan ip (MASQUERADE). I noticed such replacement was not happening when I tcpdump the traffic between test machine and upstream router. (ping). Or maybe you are facing other issue.

Also you could try manual iptables command,
Code: 
iptables -t nat -A POSTROUTING -s "your lan ip CIDR" -o eth1 -j MASQERADE
eth1 is the wan interface in openwrt lxc.

Yes, I tried to put "service firewall restart" in /etc/rc.local, the doc said it will be executed at the end of boot process, but seems not helping, maybe it is too early.
 
Last edited:
  • Like
Reactions: kenkin360
largem said:
Basically the rule is to replace the source ip from lan to the wan ip (MASQUERADE). I noticed such replacement was not happening when I tcpdump the traffic between test machine and upstream router. (ping). Or maybe you are facing other issue.

Also you could try manual iptables command,
Code: 
iptables -t nat -A POSTROUTING -s "your lan ip CIDR" -o eth1 -j MASQERADE
eth1 is the wan interface in openwrt lxc.

Yes, I tried to put "service firewall restart" in /etc/rc.local, the doc said it will be executed at the end of boot process, but seems not helping, maybe it is too early.
Click to expand...
Well I just solved it by simply restarting the firewall service until the default policies applied. I add the following statements to my startup script:

Code: 
until $(iptables -t nat -L PREROUTING |grep -q 'zone_lan_prerouting')
do
    /etc/init.d/firewall restart
    sleep 1
done

So `/etc/rc.local` would look like this:

Code: 
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

until $(iptables -t nat -L PREROUTING |grep -q 'zone_lan_prerouting')
do
    /etc/init.d/firewall restart
    sleep 1
done

exit 0

It seems the network changes are somehow not detected when it's a lxc and restarting of the service is therefore not triggered, no additional policies than the defaults are needed.
 
Last edited:
  • Like
Reactions: largem
You must log in or register to reply here.
Top Bottom