k3s on lxc - modprob: FATAL: Module overlay not found in directory /lib/modules/5.13.19-1-pve

[arkan]

I wrote the following as a reply in a thread https://forum.proxmox.com/threads/i...ing-kubernetes-on-their-lxc-containers.97158/
and I'm not sure if someone got the chance to look at it, so I decided to create a new post.

Hi, I installed Proxmox 7.1-4, and I'm interested in installing k3s on lxc. Did somebody succeeded to install k3s on lxc ? If yes, do you what instructions, tutorials did you follow/apply ?

I followed this link https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 but I did not have success




In the container:

modprob: FATAL: Module br_netfilter not found in directory /lib/modules/5.13.19-1-pve
modprob: FATAL: Module overlay not found in directory /lib/modules/5.13.19-1-pve







But on my host the overlay module is loaded:



Any idea?

 

[arkan]

By the way, if I use VM instead of lxc with the following settings on the host:




k3s will be installed successfully, and I can get the node:



and for get all:




It works for VM but for lxc, it does not work.

 

[fabian]

seems like the k3s scripts unconditionally try to load the module, which will fail in the container. using a module that was previously loaded on the host should work (well, depending on what exactly is done with the module, it might require enabling features or adapting the apparmor policies)

 

[arkan]

Hi Fabian,

Thank you for your reply. Really I don't know how to make it works, so I decided to use vm instead of lxc.

Thank you,
Arkan

 

[aghadjip]

Hi Fabian,

Thank you for your reply. Really I don't know how to make it work, so I decided to use vm instead of lxc.

Thank you,
Arkan

I havent read the links above, however, make sure you have the overlay added to your module.conf, here is mine, I run k8s and k3s in LXC

root@manifold:~# cat /etc/modules-load.d/modules.conf
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

nvidia
nvidia_uvm
#for Docker
overlay
nf_nat
br_netfilter
xt_conntrack
aufs
vhci-hcd

Also make sure your lxc config has the following
lxc.cgroup.devices.allow: a
lxc.cap.drop:

and features should have nesting turned on. This is a privileged container.

 

[arkan]

Hi aghadjip,
I made the container to be privileg, and I added the mentioned modules (except nvidia, nvidia_uvm, and also I tried to add br_netfilter but it does not show in lsmod, aufs module I don't have it)

Modules in container:




And cat /etc/pve/lxc/101.conf:



But there is no master ready:







Also the modules are added in the container to be loaded at start:

 

[aghadjip]

Sorry, i see you are running lsmod on the containers,
the main server (pve1 in your case) is what will need its /etc/modules-load.d/modules.conf edited. Can you confirm that this is the case?

Also maybe its the screen shots, but there should be an _ (underscore) between nf_nat br_netfilter etc etc.

 

[aghadjip]

also check this as a resource to see if any steps are missed

https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185

 

[arkan]

Hi aghadjip,

About the underscorre, my console is hiding the underscore, but it is actually written.

This is on the host:


also




But I think there is a problem:
Both machines (fresh proxmox installation 7.1-4) they don't have br_netfilter

I used here windows terminal in order to display the '_', because as you can see from the previous screenshots it is not displayed on the proxmox console.




I've searched for it but I can not find it:

 

[H4R0]

I have k3s running in LXC, but if you use ZFS it's not supporting overlay2 needed by k3s.

Instead use fuse-overlayfs. Any other storage should also work out of the box.

 

[arkan]

I have k3s running in LXC, but if you use ZFS it's not supporting overlay2 needed by k3s.

Instead use fuse-overlayfs. Any other storage should also work out of the box.

H4R0, thank you for the reply. The problem it is not getting the control plane to work. I've done this on vm with zfs and k3s was installed successfully and I can also see the master node, but on lxc it is not working.
I did not set any storage class, it is a simple k3s curl installation. I want first to see the master node ready.

 

[arkan]

I found on this https://forum.proxmox.com/threads/docker-support-in-proxmox.27474/page-6#post-295237
that br_netfilter is set and I verified it on my machine and it already there:




So br_netfilter is set on the host. And it's set in cat /etc/modules in the guest

 

[H4R0]

Here are my notes this should get you going.

## k3s in lxc on proxmox
# proxmox
# enable modules
cat << 'EOF' >> /etc/modules
rbd
fuse
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
overlay
br_netfilter
iptable_nat
EOF

# enable forwarding / optimization
cat << 'EOF' > /etc/sysctl.d/k3s.conf
kernel.dmesg_restrict=0

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1

kernel.pid_max=100000
user.max_user_namespaces=15000
vm.compact_memory=1
vm.overcommit_memory=1
EOF

# create privileged lxc (uncheck unprivileged)
# adjust lxc id in script => 107

cat << 'EOF' >> /etc/pve/lxc/107.conf
lxc.apparmor.profile = unconfined
lxc.cgroup2.devices.allow: c 1:11 rwm
lxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file
swap: 0
EOF

apt install fuse-overlayfs
reboot

# start lxc and enter it
pct start 107
pct enter 107

cat << 'EOF' >> /root/.bashrc
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
EOF

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--snapshotter fuse-overlayfs" sh -

 

[arkan]

Thank you H4R0,

I've replicated your commands, except my container was on 102 instead of 107, but unfortunately, it's the same issue as I mentioned above.

I'm curious, what version of proxmox are you running?

Thank you,
Arkan

 

[H4R0]

Thank you H4R0,

I've replicated your commands, except my container was on 102 instead of 107, but unfortunately, it's the same issue as I mentioned above.

I'm curious, what version of proxmox are you running?

Thank you,
Arkan

You are right, I had this working with 6.x something, with latest 7.1-7 it's not.

For me the problem is that k3s has no access to /dev/kmsg (operation not permitted) and quits.

The workaround isn't working anymore and disabling apparmor also has no effect.

 

[arkan]

Thank you H4R0,

After this beautiful journey, at least we know for now that because of the new version we can not install k3s on lxc on proxmox 7.1-4 (for me) and 7.1-7 (for you).

I think I will stick with VM, because it works if I install it on VM. And also to avoid changing the configuration (like adding a lot of modules, or installing other stuff, ...etc) on the PVE host.

I want to mention I had on my 2 rack servers Openstack installed (in the past), and the services of Openstack were installed using lxc, but if from the user interface (Horizon) I add VMs and I had Kubernetes (k8s) with Rancher installed.
And I can see in comparison with Proxmox, that Proxmox is very easy to install and also I can use VMs (like I used them in Openstack) for installing Kubernetes (this time k3s which is much lighter than k8s). For this reason I gave Proxmox a try and I've destroyed all my Openstack installation.

Thanks,
Arkan

 

[H4R0]

Thank you H4R0,

After this beautiful journey, at least we know for now that because of the new version we can not install k3s on lxc on proxmox 7.1-4 (for me) and 7.1-7 (for you).

I think I will stick with VM, because it works if I install it on VM. And also to avoid changing the configuration (like adding a lot of modules, or installing other stuff, ...etc) on the PVE host.

I want to mention I had on my 2 rack servers Openstack installed (in the past), and the services of Openstack were installed using lxc, but if from the user interface (Horizon) I add VMs and I had Kubernetes (k8s) with Rancher installed.
And I can see in comparison with Proxmox, that Proxmox is very easy to install and also I can use VMs (like I used them in Openstack) for installing Kubernetes (this time k3s which is much lighter than k8s). For this reason I gave Proxmox a try and I've destroyed all my Openstack installation.

Thanks,
Arkan

Well I'm also running k3s in vm's since lxc just doesn't work or makes problems.

Funny since I will probably migrate to OpenStack. Proxmox is a great replacement for esxi running old school vm's but this is in the past. They don't focus on the future at all, things like cloud, IaC, kubernetes or even support for arm nothing in that direction. Using LXC with apparmor is utterly broken it's just a pain in the ass to lose so much time on things like that which should work out of the box.

 

[H4R0]

It works with apparmor disabled and /dev/kmsg granted via cgroup2, both not needed with proxmox 6.x

Note that this removes all protections and breakout from lxc to pve is possible.

Since k3s isolates pods this is no problem though, unless you install malicious charts.

I updated my last post with the configuration.

 

[arkan]

As I wrote, Openstack run its services (like Horizon, Octavia, Cinder, ...etc) in lxc, but when you run your machines, it will use KVM.
Also It was a pain to intall control server + compute (It took me one month trying, with the help from the community), using Ansible, setting the networking, and settings CEPH (separate ansible installation).

Compared this with Proxmox installation. Proxmox installation is a piece of cake (very easy).

 

[tabnul]

H4R0, thank you for the reply. The problem it is not getting the control plane to work. I've done this on vm with zfs and k3s was installed successfully and I can also see the master node, but on lxc it is not working.
I did not set any storage class, it is a simple k3s curl installation. I want first to see the master node ready.

H4R0, thank you for the reply. The problem it is not getting the control plane to work. I've done this on vm with zfs and k3s was installed successfully and I can also see the master node, but on lxc it is not working.
I did not set any storage class, it is a simple k3s curl installation. I want first to see the master node ready.

When using fuse-overlay with ZFS, backupping or replicating the container results in a hard freeze of the container.
I solved that by using the 'native' snapshotter. But it seems the native snapshotter is not recommended for production workloads -- slowly i am leaning more and more towards vm's again for k3s