Missing br_netfilter module

Denis Rendler · Apr 28, 2019

Hello, everyone! I am a newcomer to linux kernel modules and I hope you can give me a helping hand. I am using Proxmox 5.4-4/97a96833, latest updates and kernel version 4.15.18-13-pve. I am running multiple LXC containers based on Ubuntu 18.04. Inside those I am trying to build a Docker swarm. Docker by itself runs without a problem, at least until now, but when I try to create a container in the swarm I get an error that states "please ensure that br_netfilter kernel module is loaded".

I've tried everything I could find on the internet, but I can't seem to make it work. Whenever I use lsmod the module doesn't show up in the list, modprobe does not give any input and modinfo says the module is not found. All of these commands were ran on the Proxmox host.

I also found a lot of posts mentioning a "lxc profile" command used to create an apparmor profile to allow this module, but I can't find an equivalent for it for Proxmox. When I try to install the LXC package is says that it needs to remove Proxmox packages, so that is a bust.

Any ideas on how I could fix this or pointing me into a direction would be much appreciated.

Addy90 · Jun 19, 2019

The problem you describe is a bug / missing feature in the kernel.
It has been discussed here: https://github.com/lxc/lxd/issues/5193
There was a patch provided towards the kernel 5.3, as you can see here: https://github.com/lxc/lxd/issues/5193#issuecomment-502857830
I hope Proxmox is able to merge this to its next kernel update, as this would dramatically expand the use case of LXC containers, for example with Kubernetes and Docker Swarm as you tried it.

Mike Lupe · Sep 5, 2019

I'm hoping for this too. K3S e.g. is so easy to install, just a few lines, but br_netfilter and overlay modules (?) not being able to load in the container kill the k3s-agent service.

Is there a workaround for 6.x?

edit: Seems to be coming in linux kernel 5.3: https://github.com/lxc/lxd/issues/5193#issuecomment-502857830

MadalinC · Feb 2, 2020

Any updates on this from the Proxmox crew?

We're at 5.3.13-2 and still doesn't seem like this has been patched. It's really awesome to have Swarm/K8s support in Proxmox, but sadly without this patch we're forced to run VMs with large overhead.

Addy90 · Feb 3, 2020

Yes, the bug was filled at the Proxmox Bugtracker here: https://bugzilla.proxmox.com/show_bug.cgi?id=2243
It is already patched since
pve-kernel-5.0.21-2-pve

Unfortunately, as it took some time until it was patched and I had to find a workaround, I was not able to test the patch yet.
More unfortunately, Docker Swarm is nearly dead already: https://boxboat.com/2019/12/10/migrate-docker-swarm-to-kubernetes/
This means that, personally, I will not try out docker swarm anymore.

BUT (and here is the big BUT): Kubernetes is said to have the same demands for the br-netfilter module that should be fixed with this patch.
I was not able to try out Kubernetes in LXC yet, but you are welcome to test if it works.

MadalinC · Feb 3, 2020

Addy90 said:
Yes, the bug was filled at the Proxmox Bugtracker here: https://bugzilla.proxmox.com/show_bug.cgi?id=2243
It is already patched since
pve-kernel-5.0.21-2-pve

Unfortunately, as it took some time until it was patched and I had to find a workaround, I was not able to test the patch yet.
More unfortunately, Docker Swarm is nearly dead already: https://boxboat.com/2019/12/10/migrate-docker-swarm-to-kubernetes/
This means that, personally, I will not try out docker swarm anymore.

BUT (and here is the big BUT): Kubernetes is said to have the same demands for the br-netfilter module that should be fixed with this patch.
I was not able to try out Kubernetes in LXC yet, but you are welcome to test if it works.

I am unsure it has been patched/merged into the current kernel as it's still not working properly with Swarm. I'm going to guess that K8s has the same issues since it uses the same dependency...

Addy90 · Feb 4, 2020

That is strange as it is said to be fixed.
Can you post more details, what exactly you did so that the staff can reproduce the issue?
Maybe you can post your results in the linked Bugzilla report I created, then there is a chance that it can be reinvestigated!
As I am still interested in Kubernetes in LXC containers, in some time, I will reach that problem again, too...

admiralakber · Jun 4, 2020

Addy90 said:
That is strange as it is said to be fixed.
Can you post more details, what exactly you did so that the staff can reproduce the issue?
Maybe you can post your results in the linked Bugzilla report I created, then there is a chance that it can be reinvestigated!
As I am still interested in Kubernetes in LXC containers, in some time, I will reach that problem again, too...

I can confirm this issue.

According to https://forum.proxmox.com/threads/docker-support-in-proxmox.27474/page-6#post-295237 it has to do with the kernel configuration? I am interested in this for the sake of running k3s inside a container and potentially making a k3os LXC template.
See: https://github.com/rancher/k3os/issues/34

pve-manager/6.2-4/9824574a (running kernel: 5.4.41-1-pve)
Linux 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15 May 2020 15:06:08 +0200) x86_64 GNU/Linux

Code:

root@pve:~# modinfo br_netfilter
modinfo: ERROR: Module br_netfilter not found.

root@pve:~# lsmod | grep br_netfilter
## returns nothing

root@pve:~# find /lib/modules/ -type f -iname '*br_netfilter.ko'
## returns nothing

root@pve:~# cat /etc/modules-load.d/k3s-lxc.conf
br_netfilter

root@pve:~# modprobe br_netfilter
## returns nothing

root@pve:~# grep 'BRIDGE_NETFILTER' /boot/config-$(uname -r)
CONFIG_BRIDGE_NETFILTER=y

pocok · Nov 17, 2020

Chiming in to follow this thread. I, too, as a Proxmox user and fan, is petitioning to get this fixed.

Marx · Dec 3, 2020

any progress?

Code:

Dec  3 22:18:18 node-silver modprobe[29558]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.73-1-pve/modules.dep.bin'
Dec  3 22:18:18 node-silver modprobe[29558]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.73-1-pve
Dec  3 22:18:18 node-silver modprobe[29564]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.73-1-pve/modules.dep.bin'
Dec  3 22:18:18 node-silver modprobe[29564]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.73-1-pve
Dec  3 22:18:19 node-silver k3s[29565]: time="2020-12-03T22:18:19.428373325Z" level=info msg="Starting k3s v1.19.4+k3s1 (2532c10f)"

Singman · Dec 5, 2020

Hi,
It's annoying. Similar problem with a LXC Debian 10 standard from template :

Bash:

root@backup:/# lsmod | grep -i iscsi
iscsi_tcp              24576  0
libiscsi_tcp           32768  1 iscsi_tcp
libiscsi               57344  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   110592  5 libiscsi_tcp,iscsi_tcp,ib_iser,libiscsi

root@backup:/# modprobe iscsi_tcp
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.73-1-pve/modules.dep.bin'
modprobe: FATAL: Module iscsi_tcp not found in directory /lib/modules/5.4.73-1-pve

root@backup:/# ls /lib/modules
ls: cannot access '/lib/modules': No such file or directory

But if I look on my host (pve), it's clear that all modules are there :

Bash:

root@pve:/# ls -CFL /lib/modules
4.15.18-10-pve/  4.15.18-23-pve/  5.3.18-3-pve/  5.4.60-1-pve/  5.4.65-1-pve/  5.4.73-1-pve/  5.4.78-1-pve/

eivamu · Mar 11, 2021

Hi there

I'm also facing this issue when trying to install K3S on LXC. I run Proxmox 6.3 with the following kernel:

Bash:

root@rack01:~# uname -r
5.4.78-2-pve

I have created an LXC container from the official Ubuntu 20.10 LXC image with features keyctl=1,nesting=1. I try to start the k3s service (systemctl start k3s) and it spits out the following error:

Code:

modprobe[1970]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.78-2-pve
modprobe[1971]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.78-2-pve
...
systemd[1]: k3s.service: Main process exited, code=exited, status=255/EXCEPTION
systemd[1]: k3s.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Lightweight Kubernetes.

I would really love for this to be fixed, or to find a workaround, as running LXC containers with k3s workers/agents seems like a very sensible usecase.

cmonty14 · May 1, 2021

eivamu said:
Hi there

I'm also facing this issue when trying to install K3S on LXC. I run Proxmox 6.3 with the following kernel:

Bash:

root@rack01:~# uname -r 5.4.78-2-pve

I have created an LXC container from the official Ubuntu 20.10 LXC image with features keyctl=1,nesting=1. I try to start the k3s service (systemctl start k3s) and it spits out the following error:

Code:

modprobe[1970]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.78-2-pve modprobe[1971]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.78-2-pve ... systemd[1]: k3s.service: Main process exited, code=exited, status=255/EXCEPTION systemd[1]: k3s.service: Failed with result 'exit-code'. systemd[1]: Failed to start Lightweight Kubernetes.

I would really love for this to be fixed, or to find a workaround, as running LXC containers with k3s workers/agents seems like a very sensible usecase.

Hi,
I'm in the same setup / installation of k3s.
Actually I can confirm the issue with kernel module br_netfilter, but module overlay is working:

Code:

$ modinfo overlay
filename:       /lib/modules/5.4.106-1-pve/kernel/fs/overlayfs/overlay.ko
alias:          fs-overlay
license:        GPL
description:    Overlay filesystem
author:         Miklos Szeredi <miklos@szeredi.hu>
srcversion:     ECD2473643261736203C6EA
depends:
retpoline:      Y
intree:         Y
name:           overlay
vermagic:       5.4.106-1-pve SMP mod_unload modversions
parm:           check_copy_up:Obsolete; does nothing
parm:           redirect_max:Maximum length of absolute redirect xattr value (ushort)                                                                         
parm:           redirect_dir:Default to on or off for the redirect_dir feature (bool)                                                                         
parm:           redirect_always_follow:Follow redirects even if redirect_dir feature is turned off (bool)                                                     
parm:           index:Default to on or off for the inodes index feature (bool)
parm:           nfs_export:Default to on or off for the NFS export feature (bool)                                                                             
parm:           xino_auto:Auto enable xino feature (bool)
parm:           metacopy:Default to on or off for the metadata only copy up feature (bool)

Can you please advise what you mean with "[...] LXC image with features keyctl=1,nesting=1"?
And what is your LXC configuration (in /etc/pve/lxc/<lxcname>.conf)?

biotecs · May 3, 2021

I can also confirm that overlay is loaded within the LXC container:

$ lsmod | grep overlay
overlay 114688 8

br_netfilter is missing.

But although overlay seems to be present, k3s throws an error because it tries to modprobe it:

Process: 603 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
Process: 608 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)

Trying to modprobe one of the modules does not work, but I think this is intential because of the structure with kernel sharing of LXC:

/sbin/modprobe overlay
modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve

The problem with not working K3S seems to be the br_netfilter because although k3s starts and runs within LXC on Proxmox there is no networking from or to the container.

cmonty14 · May 4, 2021

I Have switched to LXC running Ubuntu 20.04 and managed to setup a 4-node k3s cluster.
However you need to modify the LXC following this instruction.

malevolent · Jul 22, 2021

Same here on Proxmox 7, can't load br_netfilter in order to install kubeadm.

modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.11.22-2-pve

tabnul · Dec 16, 2021

I am wondering, anyone found a solution?
I have the same issue here, ubuntu 20.0.4 LXC on proxmox 6.4 latest. Followed the guide linked above.

K3S start: error on both overlay and br_netfilter
K3S seems to be working allright though, i have a 3 node cluster running and core pods are running.

In container AND on host, overlay seems to be loaded.
lsmod | grep overlay
overlay 114688 0

In container AND on host, br_netfilter > not loaded.
lsmod | grep overlay
<no output>

I tried to manually load it on the host, no succes, nothing gets loaded;
modprobe br_netfilter

lsmod | grep overlay
<no output>

IxsharpxI · Dec 20, 2021

cmonty14 said:
I Have switched to LXC running Ubuntu 20.04 and managed to setup a 4-node k3s cluster.
However you need to modify the LXC following this instruction.

I followed these instructions on my first master node and while it did allow k3s to start I still have the error about the modules. Is this recommended to proceed or are the modules necessary for a clean install.

Code:

* k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-12-20 19:28:25 UTC; 7s ago
       Docs: https://k3s.io
    Process: 3112 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 3114 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
    Process: 3115 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)
   Main PID: 3116 (k3s-server)
      Tasks: 94
     Memory: 1.1G
        CPU: 5.861s
     CGroup: /system.slice/k3s.service
             |- 937 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |- 975 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-1012 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-2145 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-2349 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-3116 /usr/local/bin/k3s server
             `-3129 containerd

tabnul · Dec 21, 2021

IxsharpxI said:

I followed these instructions on my first master node and while it did allow k3s to start I still have the error about the modules. Is this recommended to proceed or are the modules necessary for a clean install.

Code:

* k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-12-20 19:28:25 UTC; 7s ago
       Docs: https://k3s.io
    Process: 3112 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 3114 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
    Process: 3115 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)
   Main PID: 3116 (k3s-server)
      Tasks: 94
     Memory: 1.1G
        CPU: 5.861s
     CGroup: /system.slice/k3s.service
             |- 937 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |- 975 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-1012 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-2145 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-2349 /var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin/containerd-shim-runc->
             |-3116 /usr/local/bin/k3s server
             `-3129 containerd

K3S explicitly modprobes those modules and that fails.
However they are actually loaded and it uses hem regarless. Same behavior with microk8s.

So consider those messages 'informal'.

IxsharpxI · Dec 21, 2021

so as in informal you mean they are nothing to worry about and wont effect performance or features? So far I have added all of my nodes successfully. just haven't moved further until I knew if I was gonna have to delete everything.

thanks for your input.

Missing br_netfilter module

Member

Member

Member

New Member

Member

New Member

Member

New Member

Active Member

New Member

Member

New Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

Member

Member

Member

Member