Starting k3s.service fails with error: level=fatal msg="failed to find cpu cgroup (v2)"

cmonty14

Well-Known Member
Mar 4, 2014
343
5
58
Hello,
I have configured a container (LXC) and deployed k3s.
However starting the relevant service fails.

Checking the output of journal -xe there are different errors:
Code:
Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689640997+02:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"                 
Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689668457+02:00" level=info msg="Run: k3s kubectl"                                           
Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689730179+02:00" level=fatal msg="failed to find cpu cgroup (v2)"                             
Mai 01 20:45:48 k3s-master1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE                                                       
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Mai 01 20:45:48 k3s-master1 systemd[1]: k3s.service: Failed with result 'exit-code'.                                                                           
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Mai 01 20:45:48 k3s-master1 systemd[1]: Failed to start Lightweight Kubernetes.
░░ Subject: A start job for unit k3s.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit k3s.service has finished with a failure.
░░
░░ The job identifier is 2663 and the job result is failed.
[...]
Mai 01 20:45:53 k3s-master1 modprobe[1923]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.106-1-pve
Mai 01 20:45:53 k3s-master1 modprobe[1924]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve
Mai 01 20:45:53 k3s-master1 k3s[1925]: time="2021-05-01T20:45:53.980447935+02:00" level=info msg="Starting k3s v1.21.0+k3s1 (2705431d)"
Mai 01 20:45:53 k3s-master1 k3s[1925]: time="2021-05-01T20:45:53.983114206+02:00" level=info msg="Managed etcd cluster bootstrap already complete and initializ

Based on this I want to focus on the following 3 erros:
Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689730179+02:00" level=fatal msg="failed to find cpu cgroup (v2)" Mai 01 20:45:53 k3s-master1 modprobe[1923]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.106-1-pve Mai 01 20:45:53 k3s-master1 modprobe[1924]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve

Checking on the host, the kernel module overlay is available:
Code:
root@pc4-pve:~
$ grep 'OVERLAY' /boot/config-$(uname -r)                                                                                                                     
CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
CONFIG_OVERLAY_FS_XINO_AUTO=y
# CONFIG_OVERLAY_FS_METACOPY is not set

root@pc4-pve:~
$ lsmod | grep overlay
overlay               114688  0

Checking the kernel module br_netfilter it is "somehow" loaded, but not listed in lsmod:
Code:
root@pc4-pve:~
$ grep 'BRIDGE_NETFILTER' /boot/config-$(uname -r)
CONFIG_BRIDGE_NETFILTER=y

root@pc4-pve:~
$ lsmod | grep net
netlink_diag           16384  0
inet_diag              24576  1 tcp_diag
nf_conntrack_netlink    45056  0
nf_conntrack          139264  7 xt_conntrack,nf_nat,xt_nat,openvswitch,nf_conntrack_netlink,nf_conncount,xt_MASQUERADE
nfnetlink_log          20480  1
nfnetlink              16384  6 nft_compat,nf_conntrack_netlink,nf_tables,ip_set,nfnetlink_log
vhost_net              32768  3
vhost                  49152  1 vhost_net
tap                    24576  1 vhost_net

So, this results in following questions:
  1. Why is LXC journalctl reporting error modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve if the module overlay is loaded on host?
  2. Why is kernel module br_netfilter no shown in output of lsmod?
  3. Why is LXC journalctl reporting error modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.106-1-pve?
  4. Why is LXC journalctl reporting error level=fatal msg="failed to find cpu cgroup (v2)"?
I'm optimistic that once these errors are solved everybody would be happy to user k3s in Proxmox LXC.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!