LXC fail to start due to "newuidmap failed to write mapping" but they worked before

[Replicant]

I migrated my containers from an old host to a new one and they wouldn't even restore, complaining about uid errors:

lxc 20240303081721.668 ERROR    conf - ../src/lxc/conf.c:lxc_map_ids:3701 - newuidmap failed to write mapping "newuidmap: uid range [1000-1001) -> [1000-1001) not allowed": newuidmap 27750 0 100000 1000 1000 1000 1 1001 101001 64535
Failed to write id mapping for child process
lxc 20240303081721.668 ERROR    utils - ../src/lxc/utils.c:lxc_drop_groups:1365 - Operation not permitted - Failed to drop supplimentary groups
lxc 20240303081721.668 ERROR    utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1340 - Invalid argument - Failed to switch to gid 0
TASK ERROR: unable to restore CT 9110101 - command 'lxc-usernsexec -m u:0:100000:1000 -m g:0:100000:1000 -m u:1000:1000:1 -m g:1000:1000:1 -m u:1001:101001:64535 -m g:1001:101001:64535 -- tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/9110101/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 1

So I found a tip on the internet to bypass this issue by extracting the backup, removing the idmapping in the `lxc .conf`, recompressing, then restoring. That worked. Then, I manually added the idmapping back into the `lxc.conf`, following the tool `proxmox-lxc-idmapper` (exact same config as before removing it):

# Add to /etc/pve/lxc/<container_id>.conf:
lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535


# Add to /etc/subuid:
root:1000:1


# Add to /etc/subgid:
root:1000:1

Now the containers fail to start, same "newuidmap" issue as when restoring has come back, I just bypassed it blocking the restore by mucking around the backup:

lxc-start: 999010202: ../src/lxc/conf.c: lxc_map_ids: 3701 newuidmap failed to write mapping "newuidmap: uid range [1000-1001) -> [1000-1001) not allowed": newuidmap 48610 0 100000 1000 1000 1000 1 1001 101001 64535
lxc-start: 999010202: ../src/lxc/start.c: lxc_spawn: 1788 Failed to set up id mapping.
lxc-start: 999010202: ../src/lxc/start.c: __lxc_start: 2107 Failed to spawn container "999010202"
lxc-start: 999010202: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: 999010202: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

The contents of my`/etc/subuid` and `/etc/subgid`

root:100000:65536
root:1000:1

I have no idea what the issue is because my containers worked fine before. My idmapping for unpriveleged containers is very simple, map 1000 in the container to 1000 (my user) on the host, leave everything else untouched. That's it. Please help I'm pulling my hair out.

 

[Replicant]

Terrible news, for those of you unfortunate enough to have wasted your valuable time/attention on my thread:

one of my /etc/subgid files had:

root 1000:1 rather than root:1000:1

I highly suspect this is the issue, I really have no energy today to test anymore, and I am seriously rethinking my life.