nfs issue, shows different directory ownership/mode on client

[chalex]

I have what I think is a very simple setup, but I must be missing something very simple.

On the proxmox server, I have a zpool and a zfs dataset and I export it via NFS.

On the client (happens to be CentOS 7.x) I mount the nfs share via nfsv3 and it shows different ownership and permissions than on the server! What am I missing? Is it something about nfs v3 vs v4? Is it something about apparmor?

Here is the server side:

root@regulated01:~# zfs set acltype=posixacl tank
root@regulated01:~# cat /etc/exports | tail -n 1
/tank/regulated 10.0.0.0/8(rw,async,root_squash,no_subtree_check)
root@regulated01:~# exportfs -rv
exporting 10.0.0.0/8:/tank/regulated
root@regulated01:~# ls -alhd /tank/
drwxr-xr-x 3 root root 3 Feb 1 16:10 /tank/
root@regulated01:~# ls -alhd /tank/regulated/
drwxr-xr-x 3 root root 3 Feb 1 16:15 /tank/regulated/
root@regulated01:~# ls -alhd /tank/regulated/ukbb/
drwxr-x--- 2 13040 12711 2 Feb 1 16:15 /tank/regulated/ukbb/

Here is the client side:

[root@sci-vm-020 ~]# mount -t nfs -o vers=3 regulated01:/tank/regulated /mnt/test/
[root@sci-vm-020 ~]# ls -alh /mnt/test/ukbb/
total 1.0K
drwxr-xr-x 2 root root 2 Feb 1 16:15 .
drwxr-xr-x 3 root root 3 Feb 1 16:15 ..

/proc/mounts shows

regulated01:/tank/regulated /mnt/test nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.11.10.95,mountvers=3,mountport=33792,mountproto=udp,local_lock=none,addr=10.11.10.95 0 0


How is it possible that it shows root:root and 0755 on the client side and other ownership and permissions on the server side?


I also tried mounting via nfsv4, same result:

[root@sci-vm-020 ~]# mount -t nfs regulated01:/tank/regulated /mnt/test/

regulated01:/tank/regulated /mnt/test nfs4 rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.11.14.89,local_lock=none,addr=10.11.10.95 0 0


[root@sci-vm-020 ~]# ls -alh /mnt/test/
total 1.0K
drwxr-xr-x 3 root root 3 Feb 1 16:15 .
drwxr-xr-x. 5 root root 53 Feb 4 10:58 ..
drwxr-xr-x 2 root root 2 Feb 1 16:15 ukbb

 

[chalex]

If I remove the option
RPCMOUNTDOPTS="--manage-gids"
from /etc/defaults/nfs-kernel-server on the server side, then the mount operation just hangs indefinitely and I can't seem to get any error from either client or server side.

 

[chalex]

OK, I think I isolated the issue to a difference between a zfs dataset/filesystem and a regular directory.

e.g.

root@regulated01:~# systemctl stop nfs-kernel-server
root@regulated01:~# zfs destroy tank/regulated/ukbb
root@regulated01:~# mkdir /tank/regulated/ukbb/
root@regulated01:~# chown ceph:ceph /tank/regulated/ukbb ; chmod o-rwx /tank/regulated/ukbb
#the ceph is not related to ceph; just an existing username
root@regulated01:~# systemctl restart nfs-kernel-server
root@regulated01:~# ls -alh /tank/
total 5.0K
drwxr-xr-x 3 root root 3 Feb 1 16:10 .
drwxr-xr-x 23 root root 4.0K Jan 28 14:41 ..
drwxr-xr-x 4 root root 4 Feb 4 12:41 regulated
root@regulated01:~# ls -alh /tank/regulated/
total 2.0K
drwxr-xr-x 4 root root 4 Feb 4 12:41 .
drwxr-xr-x 3 root root 3 Feb 1 16:10 ..
drwxr-x--- 2 ceph ceph 2 Feb 4 12:41 ukbb
drwxr-xr-x 2 www-data root 2 Feb 4 11:26 ukbb2
root@regulated01:~# ls -alh /tank/regulated/ukbb
total 1.0K
drwxr-x--- 2 ceph ceph 2 Feb 4 12:41 .
drwxr-xr-x 4 root root 4 Feb 4 12:41 ..


Then on the client it correctly shows the uid/gid of the directory:

[root@sci-vm-020 ~]# ls -alh /mnt/test/
total 1.5K
drwxr-xr-x 4 root root 4 Feb 4 12:41 .
drwxr-xr-x. 5 root root 53 Feb 4 10:58 ..
drwxr-x--- 2 64045 64045 2 Feb 4 12:41 ukbb
drwxr-xr-x 2 33 root 2 Feb 4 11:26 ukbb2

 

[chalex]

OK, I think maybe this is just how it's supposed to work and I didn't have the right mental model of the nfs server traversing filesystems.
https://github.com/zfsonlinux/zfs/issues/8376