Hi,
At some point podman stopped running inside my Alpine LXC containers. When starting an instance, I am getting an error:
Importantly, I have nesting enabled on the container and unified cgroups2 on Proxmox hypervisor:
as well as in the container itself:
Podman inside container is configured to run as root. The cgroups available to container are:
And they are visible from within the container:
(notice the nobody user, though).
Container config is:
What am I missing here? This used to work before switching to cgroups v2 and I can't find any hint. I tried bunch of AI suggestions, nothing works.
Can someone please hopefully provide a comprehensive answer here and, perhaps, could Proxmox please maintain a wiki page for this? This issue comes up a lot and it would be useful to know what is the current approach to things, considering the not infrequent changes around containers/cgroups. I am sure the community would appreciate it tremendously!
At some point podman stopped running inside my Alpine LXC containers. When starting an instance, I am getting an error:
Code:
podman run hello-world Hello
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)
Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
Copying blob 81df7ff16254 done |
Copying config 5dd467fce5 done |
Writing manifest to image destination
WARN[0006] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: enabling controller cpuset: write /sys/fs/cgroup/libpod_parent/cgroup.subtree_control: no such file or directory
Error: crun: executable file `Hello` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
syncImportantly, I have nesting enabled on the container and unified cgroups2 on Proxmox hypervisor:
Code:
root@proxmox:/etc# cat /etc/mtab | grep cgroup
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0as well as in the container itself:
Code:
cat /etc/mtab | grep cgroup
none /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0Podman inside container is configured to run as root. The cgroups available to container are:
Code:
root@proxmox:/etc# ls -l /sys/fs/cgroup/lxc/115
total 0
-r--r--r-- 1 root root 0 Apr 22 18:33 cgroup.controllers
-r--r--r-- 1 root root 0 Apr 22 18:33 cgroup.events
-rw-r--r-- 1 root root 0 Apr 22 18:33 cgroup.freeze
--w------- 1 root root 0 Apr 22 18:33 cgroup.kill
-rw-r--r-- 1 root root 0 Apr 22 18:33 cgroup.max.depth
-rw-r--r-- 1 root root 0 Apr 22 18:33 cgroup.max.descendants
-rw-r--r-- 1 root root 0 Apr 22 18:33 cgroup.pressure
-rw-r--r-- 1 root root 0 Apr 22 18:33 cgroup.procs
-r--r--r-- 1 root root 0 Apr 22 18:33 cgroup.stat
-rw-r--r-- 1 root root 0 Apr 22 18:30 cgroup.subtree_control
-rw-r--r-- 1 root root 0 Apr 22 18:33 cgroup.threads
-rw-r--r-- 1 root root 0 Apr 22 18:33 cgroup.type
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.idle
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.max
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.max.burst
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.pressure
-rw-r--r-- 1 root root 0 Apr 22 18:30 cpuset.cpus
-r--r--r-- 1 root root 0 Apr 22 18:33 cpuset.cpus.effective
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpuset.cpus.exclusive
-r--r--r-- 1 root root 0 Apr 22 18:33 cpuset.cpus.exclusive.effective
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpuset.cpus.partition
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpuset.mems
-r--r--r-- 1 root root 0 Apr 22 18:33 cpuset.mems.effective
-r--r--r-- 1 root root 0 Apr 22 18:30 cpu.stat
-r--r--r-- 1 root root 0 Apr 22 18:33 cpu.stat.local
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.uclamp.max
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.uclamp.min
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.weight
-rw-r--r-- 1 root root 0 Apr 22 18:33 cpu.weight.nice
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.1GB.current
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.1GB.events
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.1GB.events.local
-rw-r--r-- 1 root root 0 Apr 22 18:33 hugetlb.1GB.max
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.1GB.numa_stat
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.1GB.rsvd.current
-rw-r--r-- 1 root root 0 Apr 22 18:33 hugetlb.1GB.rsvd.max
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.2MB.current
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.2MB.events
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.2MB.events.local
-rw-r--r-- 1 root root 0 Apr 22 18:33 hugetlb.2MB.max
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.2MB.numa_stat
-r--r--r-- 1 root root 0 Apr 22 18:33 hugetlb.2MB.rsvd.current
-rw-r--r-- 1 root root 0 Apr 22 18:33 hugetlb.2MB.rsvd.max
-rw-r--r-- 1 root root 0 Apr 22 18:33 io.max
-rw-r--r-- 1 root root 0 Apr 22 18:33 io.pressure
-rw-r--r-- 1 root root 0 Apr 22 18:33 io.prio.class
-r--r--r-- 1 root root 0 Apr 22 18:30 io.stat
-rw-r--r-- 1 root root 0 Apr 22 18:33 io.weight
-r--r--r-- 1 root root 0 Apr 22 18:30 memory.current
-r--r--r-- 1 root root 0 Apr 22 18:33 memory.events
-r--r--r-- 1 root root 0 Apr 22 18:33 memory.events.local
-rw-r--r-- 1 root root 0 Apr 22 18:30 memory.high
-rw-r--r-- 1 root root 0 Apr 22 18:33 memory.low
-rw-r--r-- 1 root root 0 Apr 22 18:30 memory.max
-rw-r--r-- 1 root root 0 Apr 22 18:33 memory.min
-r--r--r-- 1 root root 0 Apr 22 18:33 memory.numa_stat
-rw-r--r-- 1 root root 0 Apr 22 18:33 memory.oom.group
-r--r--r-- 1 root root 0 Apr 22 18:33 memory.peak
-rw-r--r-- 1 root root 0 Apr 22 18:33 memory.pressure
--w------- 1 root root 0 Apr 22 18:33 memory.reclaim
-r--r--r-- 1 root root 0 Apr 22 18:30 memory.stat
-r--r--r-- 1 root root 0 Apr 22 18:30 memory.swap.current
-r--r--r-- 1 root root 0 Apr 22 18:33 memory.swap.events
-rw-r--r-- 1 root root 0 Apr 22 18:33 memory.swap.high
-rw-r--r-- 1 root root 0 Apr 22 18:30 memory.swap.max
-r--r--r-- 1 root root 0 Apr 22 18:33 memory.swap.peak
-r--r--r-- 1 root root 0 Apr 22 18:33 memory.zswap.current
-rw-r--r-- 1 root root 0 Apr 22 18:33 memory.zswap.max
-rw-r--r-- 1 root root 0 Apr 22 18:33 memory.zswap.writeback
-r--r--r-- 1 root root 0 Apr 22 18:33 misc.current
-r--r--r-- 1 root root 0 Apr 22 18:33 misc.events
-rw-r--r-- 1 root root 0 Apr 22 18:33 misc.max
drwxrwxr-x 7 root 100000 0 Apr 22 18:31 ns
-r--r--r-- 1 root root 0 Apr 22 18:33 pids.current
-r--r--r-- 1 root root 0 Apr 22 18:33 pids.events
-rw-r--r-- 1 root root 0 Apr 22 18:33 pids.max
-r--r--r-- 1 root root 0 Apr 22 18:33 pids.peak
-r--r--r-- 1 root root 0 Apr 22 18:33 rdma.current
-rw-r--r-- 1 root root 0 Apr 22 18:33 rdma.maxAnd they are visible from within the container:
Code:
ls -al /sys/fs/cgroup
total 0
drwxrwxr-x 7 nobody root 0 Apr 22 16:31 .
drwxr-xr-x 10 nobody nobody 0 Apr 22 16:30 ..
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.controllers
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.events
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.freeze
--w------- 1 nobody nobody 0 Apr 22 16:30 cgroup.kill
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.max.depth
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.max.descendants
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.pressure
-rw-rw-r-- 1 nobody root 0 Apr 22 16:30 cgroup.procs
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.stat
-rw-rw-r-- 1 nobody root 0 Apr 22 16:31 cgroup.subtree_control
-rw-rw-r-- 1 nobody root 0 Apr 22 16:30 cgroup.threads
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cgroup.type
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.idle
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.max
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.max.burst
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.pressure
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.stat
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.stat.local
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.uclamp.max
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.uclamp.min
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.weight
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpu.weight.nice
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpuset.cpus
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cpuset.cpus.effective
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpuset.cpus.exclusive
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cpuset.cpus.exclusive.effective
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpuset.cpus.partition
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 cpuset.mems
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 cpuset.mems.effective
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.1GB.current
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.1GB.events
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.1GB.events.local
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.1GB.max
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.1GB.numa_stat
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.1GB.rsvd.current
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.1GB.rsvd.max
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.2MB.current
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.2MB.events
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.2MB.events.local
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.2MB.max
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.2MB.numa_stat
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.2MB.rsvd.current
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 hugetlb.2MB.rsvd.max
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 io.max
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 io.pressure
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 io.prio.class
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 io.stat
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 io.weight
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.current
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.events
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.events.local
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.high
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.low
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.max
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.min
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.numa_stat
-rw-rw-r-- 1 nobody root 0 Apr 22 16:30 memory.oom.group
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.peak
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.pressure
-rw-rw-r-- 1 nobody root 0 Apr 22 16:30 memory.reclaim
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.stat
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.swap.current
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.swap.events
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.swap.high
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.swap.max
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.swap.peak
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.zswap.current
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.zswap.max
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 memory.zswap.writeback
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 misc.current
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 misc.events
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 misc.max
drwxr-xr-x 2 root root 0 Apr 22 16:30 openrc.crond
drwxr-xr-x 2 root root 0 Apr 22 16:30 openrc.dropbear
drwxr-xr-x 2 root root 0 Apr 22 16:30 openrc.networking
drwxr-xr-x 2 root root 0 Apr 22 16:30 openrc.podman
drwxr-xr-x 2 root root 0 Apr 22 16:30 openrc.syslog
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 pids.current
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 pids.events
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 pids.max
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 pids.peak
-r--r--r-- 1 nobody nobody 0 Apr 22 16:30 rdma.current
-rw-r--r-- 1 nobody nobody 0 Apr 22 16:30 rdma.max(notice the nobody user, though).
Container config is:
Code:
arch: amd64
cores: 4
features: fuse=1,keyctl=1,nesting=1
hookscript: local:snippets/bridgefix.sh
hostname: ctr-rssbridge
memory: 2048
net0: name=eth0,bridge=vmbr0,hwaddr=DE:8C:9C:93:B5:20,ip=dhcp,type=veth
onboot: 1
ostype: alpine
rootfs: local-nvme-zfs:subvol-115-disk-0,size=8G
swap: 512
tty: 1
unprivileged: 1What am I missing here? This used to work before switching to cgroups v2 and I can't find any hint. I tried bunch of AI suggestions, nothing works.
Can someone please hopefully provide a comprehensive answer here and, perhaps, could Proxmox please maintain a wiki page for this? This issue comes up a lot and it would be useful to know what is the current approach to things, considering the not infrequent changes around containers/cgroups. I am sure the community would appreciate it tremendously!
Last edited: