Starting k3s.service fails with error: level=fatal msg="failed to find cpu cgroup (v2)"

[cmonty14]

Hello,
I have configured a container (LXC) and deployed k3s.
However starting the relevant service fails.

Checking the output of journal -xe there are different errors:

Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689640997+02:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"                 
Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689668457+02:00" level=info msg="Run: k3s kubectl"                                           
Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689730179+02:00" level=fatal msg="failed to find cpu cgroup (v2)"                             
Mai 01 20:45:48 k3s-master1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE                                                       
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Mai 01 20:45:48 k3s-master1 systemd[1]: k3s.service: Failed with result 'exit-code'.                                                                           
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Mai 01 20:45:48 k3s-master1 systemd[1]: Failed to start Lightweight Kubernetes.
░░ Subject: A start job for unit k3s.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit k3s.service has finished with a failure.
░░
░░ The job identifier is 2663 and the job result is failed.
[...]
Mai 01 20:45:53 k3s-master1 modprobe[1923]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.106-1-pve
Mai 01 20:45:53 k3s-master1 modprobe[1924]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve
Mai 01 20:45:53 k3s-master1 k3s[1925]: time="2021-05-01T20:45:53.980447935+02:00" level=info msg="Starting k3s v1.21.0+k3s1 (2705431d)"
Mai 01 20:45:53 k3s-master1 k3s[1925]: time="2021-05-01T20:45:53.983114206+02:00" level=info msg="Managed etcd cluster bootstrap already complete and initializ

Based on this I want to focus on the following 3 erros:
Mai 01 20:45:48 k3s-master1 k3s[1910]: time="2021-05-01T20:45:48.689730179+02:00" level=fatal msg="failed to find cpu cgroup (v2)" Mai 01 20:45:53 k3s-master1 modprobe[1923]: modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.106-1-pve Mai 01 20:45:53 k3s-master1 modprobe[1924]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve

Checking on the host, the kernel module overlay is available:

root@pc4-pve:~
$ grep 'OVERLAY' /boot/config-$(uname -r)                                                                                                                     
CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
CONFIG_OVERLAY_FS_XINO_AUTO=y
# CONFIG_OVERLAY_FS_METACOPY is not set

root@pc4-pve:~
$ lsmod | grep overlay
overlay               114688  0

Checking the kernel module br_netfilter it is "somehow" loaded, but not listed in lsmod:

root@pc4-pve:~
$ grep 'BRIDGE_NETFILTER' /boot/config-$(uname -r)
CONFIG_BRIDGE_NETFILTER=y

root@pc4-pve:~
$ lsmod | grep net
netlink_diag           16384  0
inet_diag              24576  1 tcp_diag
nf_conntrack_netlink    45056  0
nf_conntrack          139264  7 xt_conntrack,nf_nat,xt_nat,openvswitch,nf_conntrack_netlink,nf_conncount,xt_MASQUERADE
nfnetlink_log          20480  1
nfnetlink              16384  6 nft_compat,nf_conntrack_netlink,nf_tables,ip_set,nfnetlink_log
vhost_net              32768  3
vhost                  49152  1 vhost_net
tap                    24576  1 vhost_net

So, this results in following questions:

1. Why is LXC journalctl reporting error modprobe: FATAL: Module overlay not found in directory /lib/modules/5.4.106-1-pve if the module overlay is loaded on host?
2. Why is kernel module br_netfilter no shown in output of lsmod?
3. Why is LXC journalctl reporting error modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.106-1-pve?
4. Why is LXC journalctl reporting error level=fatal msg="failed to find cpu cgroup (v2)"?

I'm optimistic that once these errors are solved everybody would be happy to user k3s in Proxmox LXC.

 

[tabnul]

Same issue (modprobe errors, not the cpu errors) here however k3s is running actually, 3 node cluster with core services up and running.
Seems that the errors can be ignored if you actually have those modules loaded.

by the way i followed this guide, ignoring the first command.
https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185