Podman in rootless mode on LXC container

M

marceliszpak

New Member
Jul 27, 2023
3
0
1
I have a problem with starting podman as a non-root user on LXC. I have tried it on Debian and Fedora based LXC without success. Installation gone fine, on root account works fine but every podman command from non-root account ends with:

cockpit@Test:~$ podman info
ERRO[0000] running `/usr/bin/newuidmap 4852 0 1000 1 1 165536 65536`: newuidmap: write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

on root account

cockpit@Test:~$ sudo podman info
[sudo] password for cockpit:
host:
arch: amd64
buildahVersion: 1.28.2
cgroupControllers:
- cpu
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon_2.1.6+ds1-1_amd64
path: /usr/bin/conmon
version: 'conmon version 2.1.6, commit: unknown'
cpuUtilization:
idlePercent: 98.59
systemPercent: 0.41
userPercent: 1
cpus: 2
distribution:
codename: bookworm
distribution: debian
version: "12"
eventLogger: journald
hostname: Test
idMappings:
gidmap: null
uidmap: null
kernel: 6.5.11-8-pve
linkmode: dynamic
logDriver: journald
memFree: 8258920448
memTotal: 8589934592
networkBackend: netavark
ociRuntime:
name: crun
package: crun_1.8.1-1+deb12u1_amd64
path: /usr/bin/crun
version: |-
crun version 1.8.1
commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
exists: true
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns_1.2.0-1_amd64
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 0
swapTotal: 0
uptime: 0h 10m 36.00s
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/local/bin/overlayzfsmount
Package: Unknown
Version: 'mount from util-linux 2.38.1 (libmount 2.38.1: selinux, smack, btrfs,
verity, namespaces, assert, debug)'
overlay.mountopt: nodev
graphRoot: /var/lib/containers/storage
graphRootAllocated: 32212254720
graphRootUsed: 1242038272
graphStatus:
Backing Filesystem: zfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 0
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 4.3.1
Built: 0
BuiltTime: Thu Jan 1 00:00:00 1970
GitCommit: ""
GoVersion: go1.19.8
Os: linux
OsArch: linux/amd64
Version: 4.3.1

User has its subordinates defined

cockpit@Test:~$ grep cockpit /etc/subuid /etc/subgid
/etc/subuid:cockpit:165536:65536
/etc/subgid:cockpit:165536:65536

The same installation on VM works flawlessly, but on LXC I can't force it to work.
 
when I look into lxc configuration file I see

root@proxmox:~# cat /etc/pve/lxc/106.conf
arch: amd64
cores: 2
features: nesting=1
hostname: Test
memory: 8192
net0: name=eth0,bridge=vmbr2,firewall=1,hwaddr=BC:24:11:49:21:6B,ip=dhcp,type=veth
ostype: debian
rootfs: local-zfs:subvol-106-disk-0,size=30G
swap: 512
unprivileged: 1
root@HomeServer:~#

Doesn't it lack lxc.idmap definition?
 
Last edited:
I am encountering the same problem if the lxc is unprivileged.

But if the container is privileged, then rootless podman seems to work.
 
Last edited:
I haven't tried, because it seems a little senseless in my opinion. In such a solution podman by itself and each container run on LXC root level can have root privileges on the host. I have found few topics where running unprivileged podman in unprivileged lxc container is not recommended or even discouraged because of nested isolation which doesn't work. For sure Proxmox doesn't support it.

So I gave up and run podman in light debian vm with Cockpit. Woks fine but consumes ca 0,5 GB RAM more.
 
Last edited:
I think it depends on your use case. For me, everything of value would already be inside the LXC. If it gets compromised, it doesn't make much difference that technically there's a host over it that's probably fine. To me the value of an LXC is easy backups and snapshots I can revert when testing changes. If it's either/or I'd rather have security handled at the Podman level. I haven't come across anything that suggests running a privileged LXC is WORSE than running packages like podman natively on the host, but it's not the easiest topic to research.
 
I got it working yesterday. Unprivileged Alpine LXC running Podman as a non-root user. It requires nesting to be enabled, haven't had the time to look into this yet.
(PVE: Command on Proxmox host, LXC: Command on LXC)

Give Proxmox root access to more sub UIDs/GUIDs:
Bash: 
PVE> vi /etc/subuid
root:100000:200000   # <usr>:<start_uid>:<count>
PVE> vi /etc/subgid
root:100000:200000

Map UIDs/GUIDs of container <VMID> to host UIDs/GUIDs:
Bash: 
PVE> vi /etc/pve/lxc/<VMID>.conf
# <container_uid> <host_uid> <count>
lxc.idmap: u 0 100000 165536 # uids 0..165536 (container) -> 100000..265536 (host)
lxc.idmap: g 0 100000 165536 # gids

Test if container UIDs/GUIDs are mapped:
Bash: 
LXC> cat /proc/self/uid_map
0 100000 165536

Give unprivileged users in LXC that will use Podman access to sub UIDs/GUIDs:
Bash: 
LXC> vi /etc/subuid
username:100000:65536
LXC> vi /etc/subgid
username:100000:65536

Allow LXC access to tun kernel module (required by slirp4netns):
Bash: 
PVE> vi /etc/pve/lxc/<VMID>.conf
lxc.cgroup2.devices.allow: c 10:200 rwm  # cgroup2 for PVE >= 7.0
lxc.mount.entry: /dev/net dev/net none bind,create=dir

Install Podman:
Bash: 
LXC> apk add podman py3-pip shadow shadow-subids
LXC> pip3 install podman-compose --break-system-packages
LXC> rc-update add cgroups

Login as non-root user and try to run Podma:
Bash: 
LXC> podman run --rm hello-world

Alpine seems to require the shadow and shadow-subids packages to make /usr/bin/newuidmap work.
I don't know yet if everything is working, but running podman-compose with a Gotify config I had lying around worked without flaw.

I hope this helps :)
 
  • Like
Reactions: yushi, fireon, peterbs and 1 other person
A few hard earned tips for any lost souls ending up here:

Just to repeat, in any LXC running podman go to Options > Features and double check that Nesting is enabled (plus Keyctl if it's unprivileged). Failing to do this caused me great suffering.

You can get the latest Podman (better networking/ZFS) directly from Debian by using APT pinning. Just add the testing repo to
/etc/apt/sources.list
Code: 
deb https://ftp.debian.org/debian testing contrib main non-free non-free-firmware


Then make /etc/apt/preferences.d/pref and add:
Code: 
Package: *
Pin: release a=testing
Pin-Priority: 50


Then run:
apt update
apt-cache policy
Which should show the Testing/Trixie repos and have them marked with a 50. Now you have the repo, but it won't be used except when you manually install a package from it. So apt install -t trixie podman will install/update Podman 4.9+ and its dependencies (including pasta) from the testing repo.



Also a lot of people get stuck on weird errors when doing rootless because using sudo/su to go between accounts can mess with some variables podman relies on. Luckily if you install the systemd-container package you can run
Bash: 
machinectl shell --uid <UsernameHere>
for a clean way to launch user sessions from within your root session, and back out again with Ctrl+ ]]]



When allowing TUN networking for the LXC to pass to rootless podman, I like to include both cgroups at the bottom of
/etc/pve/lxc/<ContainerIDHere>.conf just in case:
Code: 
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
If you've already made snapshots of that LXC and these lines aren't working, you may also need to add them to the base group above the snapshots.



Getting podman containers to run at boot is kind of weird. Unlike docker it's daemonless so nothing runs by default. The old way of running containers automatically is to generate a separate Systemd service for each container. They've marked that option as depreciated on newer versions (to some minor uproar) although they've said there's no ~current~ plans to remove it entirely. The offered replacement is Quadlets, which are a bit like docker compose configs where the whole thing is written to go directly into Systemd, but they're podman specific and have completely different syntax. I like to avoid both and just make a single Systemd service that runs at boot and starts all containers with --restart=always set, making things about as simple as docker.

First linger needs to be enabled for the user running podman, so services can be run automatically as that user without needing to be logged in:
Bash: 
loginctl enable-linger <RootlessUsernameHere>

Then as that user create ~/.config/systemd/user/podman-restart.service with the contents:
Code: 
[Unit]
Description=Podman Start All Containers With Restart Policy Set To Always
Documentation=man:podman-start(1)
StartLimitIntervalSec=0
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=true
Environment=LOGGING="--log-level=info"
ExecStart=podman $LOGGING start --all --filter restart-policy=always
ExecStop=/bin/sh -c 'podman $LOGGING stop $(podman container ls --filter restart-policy=always -q)'

[Install]
WantedBy=default.target

Then start it with:
Bash: 
systemctl --user daemon-reload
systemctl --user start podman-restart
systemctl --user enable podman-restart
 
Last edited:
  • Like
Reactions: Jahara and webtroter
It's also possible to run containers from compose files, which don't support every podman specific feature (like pods) but can be pretty helpful for things only posted in that form like SWAG proxy. Dockge is a great webui for that. Uses a docker compatibility socket to manage containers but doesn't need to stay running once everything is set up, and works great under rootless.

First you install the podman-docker package. I use the testing repo since that's what I'm using for podman:
apt install -t trixie podman-docker

Then set up the socket and add it's variable to bash:
Bash: 
systemctl --user enable podman.socket
systemctl --user start podman.socket
systemctl --user status podman.socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
echo 'export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock' >> ~/.bash_profile


Then just install Dockge normally, but before you run its compose.yaml change the volume to
/var/run/user/<UserIDHere>/podman/podman.sock:/var/run/docker.sock

Haven't tried it with Yacht but they also manage podman via socket so ymmv.



If you need to run privileged TCP ports in rootless (i.e. a reverse proxy listening on 443) but don't want to give rootless users access to those ports, the podman documentation points out redir is a rad little alternative.

You just enable linger if you haven't already, then apt install redir, and put the following in a new file like
/etc/systemd/system/redir443.service
Code: 
[Unit]
Description=Redirect tcp port 443 to 1443 with redir

[Service]
ExecStart=/bin/redir -sn :443 127.0.0.1:1443

[Install]
WantedBy=multi-user.target
to redirect traffic coming in on host port 443 so a podman service can pick it up on 1443. Rinse and repeat for any other ports you want, then enable:

Bash: 
systemctl daemon-reload
systemctl enable --now redir443.service

BTW if you just want to test it till next reboot, you can skip all that and run
redir :443 127.0.0.1:1443
 
Last edited:
Mich4 said:
I got it working yesterday. Unprivileged Alpine LXC running Podman as a non-root user. It requires nesting to be enabled, haven't had the time to look into this yet.
(PVE: Command on Proxmox host, LXC: Command on LXC)

Give Proxmox root access to more sub UIDs/GUIDs:
Bash: 
PVE> vi /etc/subuid
root:100000:200000   # <usr>:<start_uid>:<count>
PVE> vi /etc/subgid
root:100000:200000

Map UIDs/GUIDs of container <VMID> to host UIDs/GUIDs:
Bash: 
PVE> vi /etc/pve/lxc/<VMID>.conf
# <container_uid> <host_uid> <count>
lxc.idmap: u 0 100000 165536 # uids 0..165536 (container) -> 100000..265536 (host)
lxc.idmap: g 0 100000 165536 # gids

Test if container UIDs/GUIDs are mapped:
Bash: 
LXC> cat /proc/self/uid_map
0 100000 165536

Give unprivileged users in LXC that will use Podman access to sub UIDs/GUIDs:
Bash: 
LXC> vi /etc/subuid
username:100000:65536
LXC> vi /etc/subgid
username:100000:65536

Allow LXC access to tun kernel module (required by slirp4netns):
Bash: 
PVE> vi /etc/pve/lxc/<VMID>.conf
lxc.cgroup2.devices.allow: c 10:200 rwm  # cgroup2 for PVE >= 7.0
lxc.mount.entry: /dev/net dev/net none bind,create=dir

Install Podman:
Bash: 
LXC> apk add podman py3-pip shadow shadow-subids
LXC> pip3 install podman-compose --break-system-packages
LXC> rc-update add cgroups

Login as non-root user and try to run Podma:
Bash: 
LXC> podman run --rm hello-world

Alpine seems to require the shadow and shadow-subids packages to make /usr/bin/newuidmap work.
I don't know yet if everything is working, but running podman-compose with a Gotify config I had lying around worked without flaw.

I hope this helps :)
Click to expand...
I followed your guide and can successfully run podman in rootless mode.

But now it has another problem which is whenever the LXC is rebooted and it shows Error: current system boot ID differs from cached boot ID; an unhandled reboot has occurred. Please delete directories "/tmp/storage-run-1001/containers" and "/tmp/storage-run-1001/libpod/tmp" and re-run Podman when I tried to run podman. I have to delete the directories as told in order to run podman again after boot.

Do you know how to solve this issue?
 
yushi said:
I followed your guide and can successfully run podman in rootless mode.

But now it has another problem which is whenever the LXC is rebooted and it shows Error: current system boot ID differs from cached boot ID; an unhandled reboot has occurred. Please delete directories "/tmp/storage-run-1001/containers" and "/tmp/storage-run-1001/libpod/tmp" and re-run Podman when I tried to run podman. I have to delete the directories as told in order to run podman again after boot.

Do you know how to solve this issue?
Click to expand...
The reason is that Alpine does not empty the /tmp folder when rebooted. It is just a normal folder and does not use a tmpfs.
I added the following line to my fstab file which mounts tmp as real tmpfs:

/tmp /tmp tmpfs rw,noexec,nodev,nosuid,size=2G 0 0

Issue solved.
 
To be honest it never worked for me, I had to build Podman from source and install it directly on the Proxmox VE Host :( .

It might be related to Podman requiring Access to the Hailo PCIe Accelerator (for Frigate), thus the Permissions & UID/GID Mappings are even weirder compared to "just" Bind Mounts Folders.
 
Johannes S said:
Why didn't you used a VM for that?
Click to expand...
That's what I usually do indeed.

However I cannot do that for some Hosts because of these **** IOMMU Groups: I CANNOT pass only the Hailo PCIe Accelerator to the VM, but instead it would pass the entire Chipset PCIe Devices (NIC, SATA controller, etc), thus crashing the entire Host System (since SATA Controller would effectively "disappear" from the Host).

IIRC I also tried pcie_acs_override=downstream,multifunction but it won't do anything in my Case at least. It's also very unsafe IMHO.
 
I wanted to give this another Try today on a new System ... Failing (again) :(.

Mich4 said:
I got it working yesterday. Unprivileged Alpine LXC running Podman as a non-root user. It requires nesting to be enabled, haven't had the time to look into this yet.
(PVE: Command on Proxmox host, LXC: Command on LXC)

Give Proxmox root access to more sub UIDs/GUIDs:
Bash: 
PVE> vi /etc/subuid
root:100000:200000   # <usr>:<start_uid>:<count>
PVE> vi /etc/subgid
root:100000:200000

Map UIDs/GUIDs of container <VMID> to host UIDs/GUIDs:
Bash: 
PVE> vi /etc/pve/lxc/<VMID>.conf
# <container_uid> <host_uid> <count>
lxc.idmap: u 0 100000 165536 # uids 0..165536 (container) -> 100000..265536 (host)
lxc.idmap: g 0 100000 165536 # gids

Test if container UIDs/GUIDs are mapped:
Bash: 
LXC> cat /proc/self/uid_map
0 100000 165536

Give unprivileged users in LXC that will use Podman access to sub UIDs/GUIDs:
Bash: 
LXC> vi /etc/subuid
username:100000:65536
LXC> vi /etc/subgid
username:100000:65536

Allow LXC access to tun kernel module (required by slirp4netns):
Bash: 
PVE> vi /etc/pve/lxc/<VMID>.conf
lxc.cgroup2.devices.allow: c 10:200 rwm  # cgroup2 for PVE >= 7.0
lxc.mount.entry: /dev/net dev/net none bind,create=dir

Install Podman:
Bash: 
LXC> apk add podman py3-pip shadow shadow-subids
LXC> pip3 install podman-compose --break-system-packages
LXC> rc-update add cgroups

Login as non-root user and try to run Podma:
Bash: 
LXC> podman run --rm hello-world

Alpine seems to require the shadow and shadow-subids packages to make /usr/bin/newuidmap work.
I don't know yet if everything is working, but running podman-compose with a Gotify config I had lying around worked without flaw.

I hope this helps :)
Click to expand...
I followed your instructions to the Letter (for a Fedora LXC Container) but I'm always stuck with
Code: 
ERRO[0000] running `/usr/bin/newuidmap 569 0 1000 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1

I also tried to add root to the LXC /etc/subuid and /etc/subgid, but that doesn't seem to make a Difference :( .

On the Host:
Code: 
root@pve:~# cat /etc/subuid
root:100000:200000

root@pve:~# cat /etc/subgid
root:100000:200000

root@pve:~# cat /etc/pve/lxc/101.conf
# <container_uid> <host_uid> <count>
# (524288 + 65536 = 589824)
# uids 0 ... 165536 (container) -> 100000 ... 265536 (host)
# gids 0 ... 165536 (container) -> 100000 ... 265536 (host)
arch: amd64
cmode: console
cores: 6
dev0: /dev/hailo0,mode=0666
dev1: /dev/hailo1,mode=0666
dev2: /dev/dri/card0,mode=0660
dev3: /dev/dri/renderD128,mode=0660
features: keyctl=1,nesting=1
hostname: podmanserver-container
memory: 16384
mp0: /tools,mp=/tools,mountoptions=discard;noatime,ro=1
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:D4:F4:7C,ip=dhcp,ip6=dhcp,type=veth
onboot: 1
ostype: fedora
rootfs: local-zfs:subvol-101-disk-0,mountoptions=discard;noatime,size=64G
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 165536
lxc.idmap: g 0 100000 165536
lxc.cgroup2.devices.allow: c 10:200 rwm

root@pve:~# cat /proc/self/uid_map
         0          0 4294967295

On the Guest:
Code: 
podman@podmanserver-container:~$ cat /etc/subuid
root:0:65536
podman:100000:65536

podman@podmanserver-container:~$ cat /etc/subgid
root:0:65536
podman:100000:65536

podman@podmanserver-container:~$ cat /proc/self/uid_map
         0     100000     165536

podman@podmanserver-container:~$ podman info
ERRO[0000] running `/usr/bin/newuidmap 569 0 1000 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1

podman@podmanserver-container:~$ stat -fc %T /sys/fs/cgroup/
cgroup2fs

podman@podmanserver-container:~$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc dmem

Clearly something is missing :( ...

EDIT 1: Quick & Dirty Troubleshooting running sleep 5000 inside the Container, just to check which User it maps on the Host.

If I run that Command as User podman (uid = 1000 & gid = 1000 in the Container), then:
Code: 
root@pve:~# ps aux | grep sleep
101000     42666  0.0  0.0 230760  1208 pts/3    S+   11:36   0:00 sleep 5000

If I run that Command as root (uid = 0 & gid = 0 in the Container), then:
Code: 
root@pve:~# ps aux | grep sleep
100000     42962  0.0  0.0 230760  1216 pts/3    S+   11:37   0:00 sleep 5000

Indeed it seems like newuidmap for Fedora is missing the setuid Flag:
Code: 
root@pve:~# stat /rpool/data/subvol-101-disk-0/usr/sbin/newuidmap
  File: /rpool/data/subvol-101-disk-0/usr/sbin/newuidmap
  Size: 35288         Blocks: 42         IO Block: 35328  regular file
Device: 0,60    Inode: 4044        Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (100000/ UNKNOWN)   Gid: (100000/ UNKNOWN)
Access: 2025-06-07 20:59:49.384017977 +0200
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-06-07 20:59:49.384017977 +0200
 Birth: 2025-06-07 20:59:49.384017977 +0200

On the Proxmox VE Host it's:
Code: 
root@pve:~# stat /usr/bin/newuidmap
  File: /usr/bin/newuidmap
  Size: 59488         Blocks: 74         IO Block: 59904  regular file
Device: 0,28    Inode: 178928      Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-08-24 10:43:13.557750045 +0200
Modify: 2025-04-19 12:20:28.000000000 +0200
Change: 2025-08-23 22:23:49.904935335 +0200
 Birth: 2025-08-23 22:23:49.892935335 +0200

On a Fedora Installation inside a VM (thus 100% under the Control of the OS, unlike LXC):
Code: 
root@podmanserver16:~# stat /usr/sbin/newuidmap
  File: /usr/sbin/newuidmap
  Size: 35288         Blocks: 80         IO Block: 4096   regular file
Device: 8,34    Inode: 1728411     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:bin_t:s0
Access: 2025-03-20 01:00:00.000000000 +0100
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-06-23 07:41:11.662779982 +0200
 Birth: 2025-06-23 07:41:11.197000000 +0200

So on Fedora /usr/sbin/newuidmap never has the setuid Flag set !

For Reference on Debian VM:
Code: 
root@debianVM:~# stat /usr/bin/newuidmap
  File: /usr/bin/newuidmap
  Size: 59488         Blocks: 120        IO Block: 4096   regular file
Device: 8,1    Inode: 1055433     Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-08-10 16:01:44.000000000 +0200
Modify: 2025-04-19 12:20:28.000000000 +0200
Change: 2025-08-10 16:03:21.434687837 +0200
 Birth: 2025-08-10 16:03:21.422688063 +0200

And on Ubuntu VM:
Code: 
root@ubuntuVM:~#stat /usr/bin/newuidmap
  File: /usr/bin/newuidmap
  Size: 41864         Blocks: 42         IO Block: 41984  regular file
Device: 0,30    Inode: 666524      Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-08-18 17:52:18.232035189 +0200
Modify: 2024-05-30 16:52:35.000000000 +0200
Change: 2024-10-09 06:49:42.822074357 +0200
 Birth: 2024-10-09 06:49:42.815074357 +0200

So basically Fedora does Things differently :rolleyes: .
 
Last edited:
Let's try to change that:
Code: 
root@pve:~# stat /rpool/data/subvol-101-disk-0/usr/sbin/newuidmap
  File: /rpool/data/subvol-101-disk-0/usr/sbin/newuidmap
  Size: 35288         Blocks: 42         IO Block: 35328  regular file
Device: 0,60    Inode: 4044        Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (100000/ UNKNOWN)   Gid: (100000/ UNKNOWN)
Access: 2025-06-07 20:59:49.384017977 +0200
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-06-07 20:59:49.384017977 +0200
 Birth: 2025-06-07 20:59:49.384017977 +0200

root@pve:~# chmod 4755 /rpool/data/subvol-101-disk-0/usr/sbin/newuidmap

root@pve:~# stat /rpool/data/subvol-101-disk-0/usr/sbin/newuidmap
  File: /rpool/data/subvol-101-disk-0/usr/sbin/newuidmap
  Size: 35288         Blocks: 42         IO Block: 35328  regular file
Device: 0,60    Inode: 4044        Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (100000/ UNKNOWN)   Gid: (100000/ UNKNOWN)
Access: 2025-06-07 20:59:49.384017977 +0200
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-08-24 11:44:54.396035040 +0200
 Birth: 2025-06-07 20:59:49.384017977 +0200

Does it work now ?
Code: 
podman@podmanserver-container:~$ podman info
ERRO[0000] running `/usr/bin/newgidmap 701 0 1000 1 1 100000 65536`: newgidmap: write to gid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newgidmap": should have setgid or have filecaps setgid: exit status 1

Let me guess ... now it's newgidmap that's different between Proxmox VE/Debian/Ubuntu compared to Fedora ?
Code: 
root@pve:~# stat /usr/bin/newgidmap
  File: /usr/bin/newgidmap
  Size: 63584         Blocks: 74         IO Block: 64000  regular file
Device: 0,28    Inode: 178926      Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-08-24 10:43:13.559750046 +0200
Modify: 2025-04-19 12:20:28.000000000 +0200
Change: 2025-08-23 22:23:49.904935335 +0200
 Birth: 2025-08-23 22:23:49.888935335 +0200


root@debianVM:~# stat /usr/bin/newgidmap
  File: /usr/bin/newgidmap
  Size: 63584         Blocks: 128        IO Block: 4096   regular file
Device: 8,1    Inode: 1055425     Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-08-10 16:01:44.000000000 +0200
Modify: 2025-04-19 12:20:28.000000000 +0200
Change: 2025-08-10 16:03:21.434687837 +0200
 Birth: 2025-08-10 16:03:21.422688063 +0200

root@ubuntuVM:~# stat /usr/bin/newgidmap
  File: /usr/bin/newgidmap
  Size: 41864         Blocks: 42         IO Block: 41984  regular file
Device: 0,30    Inode: 666522      Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-08-18 17:52:18.235157443 +0200
Modify: 2024-05-30 16:52:35.000000000 +0200
Change: 2024-10-09 06:49:42.821074357 +0200
 Birth: 2024-10-09 06:49:42.814074357 +0200

podman@fedoraLXC:~$ stat /usr/bin/newgidmap
  File: /usr/bin/newgidmap
  Size: 39384         Blocks: 50         IO Block: 39424  regular file
Device: 0,60    Inode: 4040        Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-06-07 20:59:49.383017977 +0200
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-06-07 20:59:49.383017977 +0200
 Birth: 2025-06-07 20:59:49.383017977 +0200

OK ... let's try to fix it:
Code: 
root@pve:~# stat /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap
  File: /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap
  Size: 39384         Blocks: 50         IO Block: 39424  regular file
Device: 0,60    Inode: 4040        Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (100000/ UNKNOWN)   Gid: (100000/ UNKNOWN)
Access: 2025-06-07 20:59:49.383017977 +0200
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-06-07 20:59:49.383017977 +0200
 Birth: 2025-06-07 20:59:49.383017977 +0200

# Trying to set the setgid Bit (like Podman Output would let us believe) results in 2755 Permissions
root@pve:~# chmod g+s /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap

root@pve:~# stat /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap
  File: /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap
  Size: 39384         Blocks: 50         IO Block: 39424  regular file
Device: 0,60    Inode: 4040        Links: 1
Access: (2755/-rwxr-sr-x)  Uid: (100000/ UNKNOWN)   Gid: (100000/ UNKNOWN)
Access: 2025-06-07 20:59:49.383017977 +0200
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-08-24 11:49:50.339057830 +0200
 Birth: 2025-06-07 20:59:49.383017977 +0200

# Let's set it to 4755 (setuid) like Proxmox VE/Debian/Ubuntu do
root@pve:~# chmod 4755 /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap

root@pve:~# stat /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap
  File: /rpool/data/subvol-101-disk-0/usr/sbin/newgidmap
  Size: 39384         Blocks: 50         IO Block: 39424  regular file
Device: 0,60    Inode: 4040        Links: 1
Access: (4755/-rwsr-xr-x)  Uid: (100000/ UNKNOWN)   Gid: (100000/ UNKNOWN)
Access: 2025-06-07 20:59:49.383017977 +0200
Modify: 2025-03-20 01:00:00.000000000 +0100
Change: 2025-08-24 11:50:16.166059819 +0200
 Birth: 2025-06-07 20:59:49.383017977 +0200

Does it work now ?

Code: 
podman@podmanserver-container:~$ podman info
host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 98.58
    systemPercent: 0.54
    userPercent: 0.88
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "42"
  eventLogger: journald
  freeLocks: 2048
  hostname: podmanserver-container
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.14.8-2-pve
  linkmode: dynamic
  logDriver: journald
  memFree: 17055006720
  memTotal: 17179869184
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.0-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.0
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.x86_64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 25m 40.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - ghcr.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-3.fc42.x86_64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev
  graphRoot: /home/podman/containers/storage
  graphRootAllocated: 68719476736
  graphRootUsed: 1307705344
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /home/podman/containers/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000
  transientStore: false
  volumePath: /home/podman/containers/volumes
version:
  APIVersion: 5.5.2
  BuildOrigin: Fedora Project
  Built: 1750723200
  BuiltTime: Tue Jun 24 02:00:00 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.2

Success :).

Now let's see if I can run a Podman Container though ...

Code: 
podman@podmanserver-container:~$ podman run --rm hello-world
✔ docker.io/library/hello-world:latest
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 17eec7bbc9d7 done   |
Copying config 1b44b5a3e0 done   |
Writing manifest to image destination
ERRO[0004] Preparing container 1175050e571180254177a9ad71afcdc55f198ed287d0c8d7587ef8c39b8d9e8e: pasta failed with exit code 1:
Failed to open() /dev/net/tun: No such file or directory
Failed to set up tap device in namespace
Error: mounting storage for container 1175050e571180254177a9ad71afcdc55f198ed287d0c8d7587ef8c39b8d9e8e: creating overlay mount to /home/podman/containers/storage/overlay/d701ea78a34812b0782a3dbd0e2cc221999bff2f3a6bddc56f8161ff01627dd6/merged, mount_data="lowerdir=/home/podman/containers/storage/overlay/l/S62ZKUO5MUKFJSY46Y74LLCGWE,upperdir=/home/podman/containers/storage/overlay/d701ea78a34812b0782a3dbd0e2cc221999bff2f3a6bddc56f8161ff01627dd6/diff,workdir=/home/podman/containers/storage/overlay/d701ea78a34812b0782a3dbd0e2cc221999bff2f3a6bddc56f8161ff01627dd6/work,nodev,volatile": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1

Well, there are a couple more Problems, but for now at least newuidmap and newgidmap work.

I guess I'll need to setup a Cron Job / Systemd Timer to make sure to set 4755 to both (from the Proxmox VE Host) should the Container update them at some Point ...
 
Last edited:
For the last couple of Errors, I added this to /etc/pve/lxc/101.conf:
Code: 
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir

Also enabled FUSE in the Container Options:
1756029961756.png

(I previously also tried to enable Create Device Nodes, although that does NOT seem to be required and it's NOT sufficient anyways instead of the /dev/net Bind Mount)

And finally I can run the Container :) :
Code: 
podman@podmanserver-container:~$ podman run --rm hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Now I'm sure I'll have some further Issues with the Hailo Device and/or the iGPU, but at least for now the Basics seem to work :).
 
I also came up with a Script & a Systemd Timer+Service which can be installed directly INSIDE the LXC Container and run Hourly to fix if those Permissions change due to System Updates:

https://github.com/luckylinux/podman-tools/blob/main/setup_podman_lxc_fixes.sh

EDIT 1: Fedora relies on Capabilities (getcap / setcap), as opposed to Debian/Ubuntu that rely on setuid directly (Permission 4755).

Code: 
root@FedoraVM:~# getcap /usr/bin/newuidmap
/usr/bin/newuidmap cap_setuid=ep

root@FedoraVM:~# getcap /usr/bin/newgidmap
/usr/bin/newgidmap cap_setgid=ep

EDIT 2: I am not sure why, but it seems that the LXC Container "Template" apparently has the Capability turned Off by default.

You can fix using
Code: 
[root@Fedora-lxc ~]# setcap cap_setuid=ep /usr/bin/newuidmap
[root@Fedora-lxc ~]# setcap cap_setgid=ep /usr/bin/newgidmap

Or just using:
Code: 
dnf reinstall shadow-utils

EDIT 3: Unsure why this occurred, there is probably a BUG / Regression somewhere in Fedora or LXC Build System :rolleyes:

EDIT 4: Bug Reports
- https://github.com/lxc/distrobuilder/issues/939
- https://bugzilla.redhat.com/show_bug.cgi?id=2390800
 
Last edited:
You must log in or register to reply here.
Top Bottom