k3s on lxc - modprob: FATAL: Module overlay not found in directory /lib/modules/5.13.19-1-pve

arkan

New Member
Nov 29, 2021
13
2
1
44
I wrote the following as a reply in a thread https://forum.proxmox.com/threads/i...ing-kubernetes-on-their-lxc-containers.97158/
and I'm not sure if someone got the chance to look at it, so I decided to create a new post.

Hi, I installed Proxmox 7.1-4, and I'm interested in installing k3s on lxc. Did somebody succeeded to install k3s on lxc ? If yes, do you what instructions, tutorials did you follow/apply ?

I followed this link https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 but I did not have success

1.png


In the container:

modprob: FATAL: Module br_netfilter not found in directory /lib/modules/5.13.19-1-pve
modprob: FATAL: Module overlay not found in directory /lib/modules/5.13.19-1-pve

2.png


3.png


But on my host the overlay module is loaded:

4.png

Any idea?
 

arkan

New Member
Nov 29, 2021
13
2
1
44
By the way, if I use VM instead of lxc with the following settings on the host:

1638206694703.png


k3s will be installed successfully, and I can get the node:

1638206752400.png

and for get all:

1638206791721.png


It works for VM but for lxc, it does not work.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,125
1,585
164
seems like the k3s scripts unconditionally try to load the module, which will fail in the container. using a module that was previously loaded on the host should work (well, depending on what exactly is done with the module, it might require enabling features or adapting the apparmor policies)
 

arkan

New Member
Nov 29, 2021
13
2
1
44
Hi Fabian,

Thank you for your reply. Really I don't know how to make it works, so I decided to use vm instead of lxc.

Thank you,
Arkan
 
Last edited:
Jan 22, 2019
27
5
8
42
Hi Fabian,

Thank you for your reply. Really I don't know how to make it work, so I decided to use vm instead of lxc.

Thank you,
Arkan
I havent read the links above, however, make sure you have the overlay added to your module.conf, here is mine, I run k8s and k3s in LXC

Code:
root@manifold:~# cat /etc/modules-load.d/modules.conf
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

nvidia
nvidia_uvm
#for Docker
overlay
nf_nat
br_netfilter
xt_conntrack
aufs
vhci-hcd

Also make sure your lxc config has the following
lxc.cgroup.devices.allow: a
lxc.cap.drop:

and features should have nesting turned on. This is a privileged container.
 

arkan

New Member
Nov 29, 2021
13
2
1
44
Hi aghadjip,
I made the container to be privileg, and I added the mentioned modules (except nvidia, nvidia_uvm, and also I tried to add br_netfilter but it does not show in lsmod, aufs module I don't have it)

Modules in container:
1638365973145.png

1638366002169.png

And cat /etc/pve/lxc/101.conf:

1638366086651.png

But there is no master ready:

1638366158608.png


1638366186772.png


Also the modules are added in the container to be loaded at start:

1638366244182.png
 
Jan 22, 2019
27
5
8
42
Sorry, i see you are running lsmod on the containers,
the main server (pve1 in your case) is what will need its /etc/modules-load.d/modules.conf edited. Can you confirm that this is the case?

Also maybe its the screen shots, but there should be an _ (underscore) between nf_nat br_netfilter etc etc.
 

arkan

New Member
Nov 29, 2021
13
2
1
44
Hi aghadjip,

About the underscorre, my console is hiding the underscore, but it is actually written.

This is on the host:
1638385897275.png

also

1638387612093.png


But I think there is a problem:
Both machines (fresh proxmox installation 7.1-4) they don't have br_netfilter

I used here windows terminal in order to display the '_', because as you can see from the previous screenshots it is not displayed on the proxmox console.

1638387821112.png


I've searched for it but I can not find it:
1638388030941.png


1638388064847.png
 

H4R0

Well-Known Member
Apr 5, 2020
616
138
48
I have k3s running in LXC, but if you use ZFS it's not supporting overlay2 needed by k3s.

Instead use fuse-overlayfs. Any other storage should also work out of the box.
 

arkan

New Member
Nov 29, 2021
13
2
1
44
I have k3s running in LXC, but if you use ZFS it's not supporting overlay2 needed by k3s.

Instead use fuse-overlayfs. Any other storage should also work out of the box.
H4R0, thank you for the reply. The problem it is not getting the control plane to work. I've done this on vm with zfs and k3s was installed successfully and I can also see the master node, but on lxc it is not working.
I did not set any storage class, it is a simple k3s curl installation. I want first to see the master node ready.
 

H4R0

Well-Known Member
Apr 5, 2020
616
138
48
Here are my notes this should get you going.

Code:
## k3s in lxc on proxmox
# proxmox
# enable modules
cat << 'EOF' >> /etc/modules
rbd
fuse
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
overlay
br_netfilter
iptable_nat
EOF

# enable forwarding / optimization
cat << 'EOF' > /etc/sysctl.d/k3s.conf
kernel.dmesg_restrict=0

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1

kernel.pid_max=100000
user.max_user_namespaces=15000
vm.compact_memory=1
vm.overcommit_memory=1
EOF

# create privileged lxc (uncheck unprivileged)
# adjust lxc id in script => 107

cat << 'EOF' >> /etc/pve/lxc/107.conf
lxc.apparmor.profile = unconfined
lxc.cgroup2.devices.allow: c 1:11 rwm
lxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file
swap: 0
EOF

apt install fuse-overlayfs
reboot

# start lxc and enter it
pct start 107
pct enter 107

cat << 'EOF' >> /root/.bashrc
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
EOF

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--snapshotter fuse-overlayfs" sh -
 
Last edited:

arkan

New Member
Nov 29, 2021
13
2
1
44
Thank you H4R0,

I've replicated your commands, except my container was on 102 instead of 107, but unfortunately, it's the same issue as I mentioned above.

I'm curious, what version of proxmox are you running?

Thank you,
Arkan
 

H4R0

Well-Known Member
Apr 5, 2020
616
138
48
Thank you H4R0,

I've replicated your commands, except my container was on 102 instead of 107, but unfortunately, it's the same issue as I mentioned above.

I'm curious, what version of proxmox are you running?

Thank you,
Arkan

You are right, I had this working with 6.x something, with latest 7.1-7 it's not.

For me the problem is that k3s has no access to /dev/kmsg (operation not permitted) and quits.

The workaround isn't working anymore and disabling apparmor also has no effect.
 

arkan

New Member
Nov 29, 2021
13
2
1
44
Thank you H4R0,

After this beautiful journey, at least we know for now that because of the new version we can not install k3s on lxc on proxmox 7.1-4 (for me) and 7.1-7 (for you).

I think I will stick with VM, because it works if I install it on VM. And also to avoid changing the configuration (like adding a lot of modules, or installing other stuff, ...etc) on the PVE host.

I want to mention I had on my 2 rack servers Openstack installed (in the past), and the services of Openstack were installed using lxc, but if from the user interface (Horizon) I add VMs and I had Kubernetes (k8s) with Rancher installed.
And I can see in comparison with Proxmox, that Proxmox is very easy to install and also I can use VMs (like I used them in Openstack) for installing Kubernetes (this time k3s which is much lighter than k8s). For this reason I gave Proxmox a try and I've destroyed all my Openstack installation.

Thanks,
Arkan
 
  • Like
Reactions: Tmanok

H4R0

Well-Known Member
Apr 5, 2020
616
138
48
Thank you H4R0,

After this beautiful journey, at least we know for now that because of the new version we can not install k3s on lxc on proxmox 7.1-4 (for me) and 7.1-7 (for you).

I think I will stick with VM, because it works if I install it on VM. And also to avoid changing the configuration (like adding a lot of modules, or installing other stuff, ...etc) on the PVE host.

I want to mention I had on my 2 rack servers Openstack installed (in the past), and the services of Openstack were installed using lxc, but if from the user interface (Horizon) I add VMs and I had Kubernetes (k8s) with Rancher installed.
And I can see in comparison with Proxmox, that Proxmox is very easy to install and also I can use VMs (like I used them in Openstack) for installing Kubernetes (this time k3s which is much lighter than k8s). For this reason I gave Proxmox a try and I've destroyed all my Openstack installation.

Thanks,
Arkan

Well I'm also running k3s in vm's since lxc just doesn't work or makes problems.

Funny since I will probably migrate to OpenStack. Proxmox is a great replacement for esxi running old school vm's but this is in the past. They don't focus on the future at all, things like cloud, IaC, kubernetes or even support for arm nothing in that direction. Using LXC with apparmor is utterly broken it's just a pain in the ass to lose so much time on things like that which should work out of the box.
 

H4R0

Well-Known Member
Apr 5, 2020
616
138
48
It works with apparmor disabled and /dev/kmsg granted via cgroup2, both not needed with proxmox 6.x

Note that this removes all protections and breakout from lxc to pve is possible.

Since k3s isolates pods this is no problem though, unless you install malicious charts.

I updated my last post with the configuration.


1638714858864.png
 
Last edited:

arkan

New Member
Nov 29, 2021
13
2
1
44
As I wrote, Openstack run its services (like Horizon, Octavia, Cinder, ...etc) in lxc, but when you run your machines, it will use KVM.
Also It was a pain to intall control server + compute (It took me one month trying, with the help from the community), using Ansible, setting the networking, and settings CEPH (separate ansible installation).

Compared this with Proxmox installation. Proxmox installation is a piece of cake (very easy).
 

tabnul

New Member
Jan 3, 2021
16
1
3
39
H4R0, thank you for the reply. The problem it is not getting the control plane to work. I've done this on vm with zfs and k3s was installed successfully and I can also see the master node, but on lxc it is not working.
I did not set any storage class, it is a simple k3s curl installation. I want first to see the master node ready.

H4R0, thank you for the reply. The problem it is not getting the control plane to work. I've done this on vm with zfs and k3s was installed successfully and I can also see the master node, but on lxc it is not working.
I did not set any storage class, it is a simple k3s curl installation. I want first to see the master node ready.

When using fuse-overlay with ZFS, backupping or replicating the container results in a hard freeze of the container.
I solved that by using the 'native' snapshotter. But it seems the native snapshotter is not recommended for production workloads -- slowly i am leaning more and more towards vm's again for k3s ;)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!