Missing br_netfilter module

Denis Rendler

New Member
Apr 28, 2019
1
0
1
42
Hello, everyone! I am a newcomer to linux kernel modules and I hope you can give me a helping hand. I am using Proxmox 5.4-4/97a96833, latest updates and kernel version 4.15.18-13-pve. I am running multiple LXC containers based on Ubuntu 18.04. Inside those I am trying to build a Docker swarm. Docker by itself runs without a problem, at least until now, but when I try to create a container in the swarm I get an error that states "please ensure that br_netfilter kernel module is loaded".

I've tried everything I could find on the internet, but I can't seem to make it work. Whenever I use lsmod the module doesn't show up in the list, modprobe does not give any input and modinfo says the module is not found. All of these commands were ran on the Proxmox host.

I also found a lot of posts mentioning a "lxc profile" command used to create an apparmor profile to allow this module, but I can't find an equivalent for it for Proxmox. When I try to install the LXC package is says that it needs to remove Proxmox packages, so that is a bust.

Any ideas on how I could fix this or pointing me into a direction would be much appreciated.
 

Addy90

Member
Mar 12, 2018
11
2
8
31
  • Like
Reactions: Mike Lupe

MadalinC

New Member
Jan 28, 2020
14
0
1
Any updates on this from the Proxmox crew?

We're at 5.3.13-2 and still doesn't seem like this has been patched. It's really awesome to have Swarm/K8s support in Proxmox, but sadly without this patch we're forced to run VMs with large overhead. :confused:
 

Addy90

Member
Mar 12, 2018
11
2
8
31
Yes, the bug was filled at the Proxmox Bugtracker here: https://bugzilla.proxmox.com/show_bug.cgi?id=2243
It is already patched since
pve-kernel-5.0.21-2-pve

Unfortunately, as it took some time until it was patched and I had to find a workaround, I was not able to test the patch yet.
More unfortunately, Docker Swarm is nearly dead already: https://boxboat.com/2019/12/10/migrate-docker-swarm-to-kubernetes/
This means that, personally, I will not try out docker swarm anymore.

BUT (and here is the big BUT): Kubernetes is said to have the same demands for the br-netfilter module that should be fixed with this patch.
I was not able to try out Kubernetes in LXC yet, but you are welcome to test if it works.
 

MadalinC

New Member
Jan 28, 2020
14
0
1
Yes, the bug was filled at the Proxmox Bugtracker here: https://bugzilla.proxmox.com/show_bug.cgi?id=2243
It is already patched since
pve-kernel-5.0.21-2-pve

Unfortunately, as it took some time until it was patched and I had to find a workaround, I was not able to test the patch yet.
More unfortunately, Docker Swarm is nearly dead already: https://boxboat.com/2019/12/10/migrate-docker-swarm-to-kubernetes/
This means that, personally, I will not try out docker swarm anymore.

BUT (and here is the big BUT): Kubernetes is said to have the same demands for the br-netfilter module that should be fixed with this patch.
I was not able to try out Kubernetes in LXC yet, but you are welcome to test if it works.

I am unsure it has been patched/merged into the current kernel as it's still not working properly with Swarm. I'm going to guess that K8s has the same issues since it uses the same dependency...
 

Addy90

Member
Mar 12, 2018
11
2
8
31
That is strange as it is said to be fixed.
Can you post more details, what exactly you did so that the staff can reproduce the issue?
Maybe you can post your results in the linked Bugzilla report I created, then there is a chance that it can be reinvestigated!
As I am still interested in Kubernetes in LXC containers, in some time, I will reach that problem again, too...
 

admiralakber

New Member
Jun 4, 2020
1
0
1
32
That is strange as it is said to be fixed.
Can you post more details, what exactly you did so that the staff can reproduce the issue?
Maybe you can post your results in the linked Bugzilla report I created, then there is a chance that it can be reinvestigated!
As I am still interested in Kubernetes in LXC containers, in some time, I will reach that problem again, too...

I can confirm this issue.

According to https://forum.proxmox.com/threads/docker-support-in-proxmox.27474/page-6#post-295237 it has to do with the kernel configuration? I am interested in this for the sake of running k3s inside a container and potentially making a k3os LXC template.
See: https://github.com/rancher/k3os/issues/34

pve-manager/6.2-4/9824574a (running kernel: 5.4.41-1-pve)
Linux 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15 May 2020 15:06:08 +0200) x86_64 GNU/Linux

Code:
root@pve:~# modinfo br_netfilter
modinfo: ERROR: Module br_netfilter not found.

root@pve:~# lsmod | grep br_netfilter
## returns nothing

root@pve:~# find /lib/modules/ -type f -iname '*br_netfilter.ko'
## returns nothing

root@pve:~# cat /etc/modules-load.d/k3s-lxc.conf
br_netfilter

root@pve:~# modprobe br_netfilter
## returns nothing

root@pve:~# grep 'BRIDGE_NETFILTER' /boot/config-$(uname -r)
CONFIG_BRIDGE_NETFILTER=y
 

pocok

Member
Dec 6, 2019
31
10
8
Chiming in to follow this thread. I, too, as a Proxmox user and fan, is petitioning to get this fixed.
 
  • Like
Reactions: kobemtl

Marx

New Member
Oct 16, 2020
18
1
3
49
any progress?

Code:
Dec  3 22:18:18 node-silver modprobe[29558]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.73-1-pve/modules.dep.bin'
Dec  3 22:18:18 node-silver modprobe[29558]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.73-1-pve
Dec  3 22:18:18 node-silver modprobe[29564]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.73-1-pve/modules.dep.bin'
Dec  3 22:18:18 node-silver modprobe[29564]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.73-1-pve
Dec  3 22:18:19 node-silver k3s[29565]: time="2020-12-03T22:18:19.428373325Z" level=info msg="Starting k3s v1.19.4+k3s1 (2532c10f)"
 

Singman

New Member
Sep 13, 2019
14
0
1
54
Hi,
It's annoying. Similar problem with a LXC Debian 10 standard from template :

Bash:
root@backup:/# lsmod | grep -i iscsi
iscsi_tcp              24576  0
libiscsi_tcp           32768  1 iscsi_tcp
libiscsi               57344  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   110592  5 libiscsi_tcp,iscsi_tcp,ib_iser,libiscsi

root@backup:/# modprobe iscsi_tcp
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.73-1-pve/modules.dep.bin'
modprobe: FATAL: Module iscsi_tcp not found in directory /lib/modules/5.4.73-1-pve

root@backup:/# ls /lib/modules
ls: cannot access '/lib/modules': No such file or directory

But if I look on my host (pve), it's clear that all modules are there :
Bash:
root@pve:/# ls -CFL /lib/modules
4.15.18-10-pve/  4.15.18-23-pve/  5.3.18-3-pve/  5.4.60-1-pve/  5.4.65-1-pve/  5.4.73-1-pve/  5.4.78-1-pve/
 
Last edited:

eivamu

New Member
Mar 11, 2021
1
0
1
Norway
Hi there :)

I'm also facing this issue when trying to install K3S on LXC. I run Proxmox 6.3 with the following kernel:

Bash:
root@rack01:~# uname -r
5.4.78-2-pve

I have created an LXC container from the official Ubuntu 20.10 LXC image with features keyctl=1,nesting=1. I try to start the k3s service (systemctl start k3s) and it spits out the following error:

Code:
modprobe[1970]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.78-2-pve
modprobe[1971]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.78-2-pve
...
systemd[1]: k3s.service: Main process exited, code=exited, status=255/EXCEPTION
systemd[1]: k3s.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Lightweight Kubernetes.

I would really love for this to be fixed, or to find a workaround, as running LXC containers with k3s workers/agents seems like a very sensible usecase.
 
Last edited:

cmonty14

Active Member
Mar 4, 2014
332
4
38
Hi there :)

I'm also facing this issue when trying to install K3S on LXC. I run Proxmox 6.3 with the following kernel:

Bash:
root@rack01:~# uname -r
5.4.78-2-pve

I have created an LXC container from the official Ubuntu 20.10 LXC image with features keyctl=1,nesting=1. I try to start the k3s service (systemctl start k3s) and it spits out the following error:

Code:
modprobe[1970]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.78-2-pve
modprobe[1971]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.78-2-pve
...
systemd[1]: k3s.service: Main process exited, code=exited, status=255/EXCEPTION
systemd[1]: k3s.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Lightweight Kubernetes.

I would really love for this to be fixed, or to find a workaround, as running LXC containers with k3s workers/agents seems like a very sensible usecase.
Hi,
I'm in the same setup / installation of k3s.
Actually I can confirm the issue with kernel module br_netfilter, but module overlay is working:
Code:
$ modinfo overlay
filename:       /lib/modules/5.4.106-1-pve/kernel/fs/overlayfs/overlay.ko
alias:          fs-overlay
license:        GPL
description:    Overlay filesystem
author:         Miklos Szeredi <miklos@szeredi.hu>
srcversion:     ECD2473643261736203C6EA
depends:
retpoline:      Y
intree:         Y
name:           overlay
vermagic:       5.4.106-1-pve SMP mod_unload modversions
parm:           check_copy_up:Obsolete; does nothing
parm:           redirect_max:Maximum length of absolute redirect xattr value (ushort)                                                                         
parm:           redirect_dir:Default to on or off for the redirect_dir feature (bool)                                                                         
parm:           redirect_always_follow:Follow redirects even if redirect_dir feature is turned off (bool)                                                     
parm:           index:Default to on or off for the inodes index feature (bool)
parm:           nfs_export:Default to on or off for the NFS export feature (bool)                                                                             
parm:           xino_auto:Auto enable xino feature (bool)
parm:           metacopy:Default to on or off for the metadata only copy up feature (bool)

Can you please advise what you mean with "[...] LXC image with features keyctl=1,nesting=1"?
And what is your LXC configuration (in /etc/pve/lxc/<lxcname>.conf)?
 

biotecs

New Member
May 3, 2021
1
0
1
40
I can also confirm that overlay is loaded within the LXC container:
$ lsmod | grep overlay
overlay 114688 8

br_netfilter is missing.

But although overlay seems to be present, k3s throws an error because it tries to modprobe it:

Process: 603 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
Process: 608 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)

Trying to modprobe one of the modules does not work, but I think this is intential because of the structure with kernel sharing of LXC:
/sbin/modprobe overlay
modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve

The problem with not working K3S seems to be the br_netfilter because although k3s starts and runs within LXC on Proxmox there is no networking from or to the container.
 

cmonty14

Active Member
Mar 4, 2014
332
4
38
I Have switched to LXC running Ubuntu 20.04 and managed to setup a 4-node k3s cluster.
However you need to modify the LXC following this instruction.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!