PVE 9.0 + ZFS + unprivileged LXC: root cannot write to files owned by non-root user ("Permission denied")

[onlime]

Hi,

after upgrading host nodes to Proxmox VE 9.0, I am seeing a reproducible permission issue inside unprivileged LXC containers backed by ZFS subvolumes.

Inside the container, even root cannot append or write to certain files if they are owned by a non-root user, even though permissions and ACLs appear correct and the filesystem is mounted read-write.

This problem did not occur before PVE 9.0.


Environment

• Proxmox VE: 9.0 (latest updates installed)
• Kernel: 6.14.11-4-pve
• ZFS: 2.3.4
• LXC Container: Debian 12.12, unprivileged, storage = ZFS subvol, nesting enabled
• The issue occurs on multiple nodes

Container configuration (pct config CTID) uses default unprivileged ID-mapping; no special ACL or mount options.

Reproduction

Inside an unprivileged container (Debian 12):

1. File owned by user

# as root inside CT:
$ echo "# foobar" >> /home/web23/.bash_profile
-bash: /home/web23/.bash_profile: Permission denied

2. Permissions and ACLs look fine

$ ls -l /home/web23/.bash_profile
-rw-r--r-- 1 web23 web23 1698 Oct 24 17:31 /home/web23/.bash_profile

$ lsattr /home/web23/.bash_profile
---------------------- /home/web23/.bash_profile

$ getfacl /home/web23/.bash_profile
# file: home/web23/.bash_profile
# owner: web23
# group: web23
user::rw-
group::r--
other::r--

Directory:

$ getfacl /home/web23
# file: home/web23
# owner: root
# group: web23
# flags: --t
user::rwx
group::rwx
other::---

No immutable flags:

$ lsattr -d /home/web23
---------------------- /home/web23

3. Creating a new file works

If I replace the file:

$ cat .bash_profile > .bash_profile.tmp
$ mv .bash_profile.tmp .bash_profile
$ echo "# foobar" >> .bash_profile    # works

Because now the file is owned by root (UID 0 CT, UID 0 on host).

4. After restoring original ownership → fails again

$ chown web23:web23 .bash_profile
$ echo "# foobar" >> .bash_profile
-bash: .bash_profile: Permission denied

So the issue only appears when the file is owned by a *mapped user* (non-root) inside an unprivileged LXC.


Mount and storage info

Inside the container:

$ mount
dpool/zfsdisks/subvol-242-disk-1 on / type zfs (rw,relatime,xattr,posixacl,casesensitive)

On the host, dataset is not readonly:

$ zfs get readonly,acltype dpool/zfsdisks/subvol-242-disk-1
NAME                              PROPERTY  VALUE     SOURCE
dpool/zfsdisks/subvol-242-disk-1  readonly  off       default
dpool/zfsdisks/subvol-242-disk-1  acltype   posix     local

Plenty of free space, no ZFS corruption, no pool errors (`zpool status` clean).

pct config:

$ pct config 242
arch: amd64
cpulimit: 4
cpuunits: 256
features: nesting=1
hostname: s02.example.com
memory: 12288
net0: name=eth0,bridge=vmbr0,gw=x.x.x.x,hwaddr=...,ip=x.x.x.x/25,type=veth
onboot: 1
ostype: debian
rootfs: zfsvols:subvol-242-disk-1,acl=1
swap: 1024

Summary of the problem

After the upgrade to Proxmox VE 9.0, on hosts using ZFS 2.3.x, an unprivileged container cannot write to files owned by a non-root user, even when:

• permissions are correct
• ACLs are correct
• ZFS dataset is mounted rw
• no immutable flags
• root inside CT should be allowed to write

Replacing the file (owned by root) works.
Changing ownership back to the actual user immediately breaks write access again.

This strongly suggests a regression in how ZFS + unprivileged LXC ID-mapping interacts with POSIX ACL enforcement after the PVE 9.0 upgrades, possibly AppArmor related.


Expected behavior

Root inside an unprivileged LXC should be able to write to files with `0664` or `0644` permissions, regardless of mapped UID/GID, as it worked before PVE 9 on the same host or on any other host with similar setup.


Actual behavior

Write operations fail with:

Permission denied

Only when the file is owned by a mapped non-root user and the dataset is ZFS.
(I cannot test ext4 because all hosts run ZFS.)


Request

Could Proxmox developers confirm whether this is:

• a regression introduced with PVE 9.0,
• a side effect of kernel 6.14 + ZFS 2.3 + unprivileged LXC, or
• an intentional change in ID-mapped behavior?

Please let me know what additional information I could provide to help diagnose this.

Thanks!