NFS4 and Kerberos

[juraedv]

Hey folks,

we are running proxmox in newest version on two host-maschines as a cluster. we have several nfs-shares mounted over the web-gui and it works fine.
now i wanted to add a nfs4-share which uses kerberos-authentication. i placed the krb5.keytab i got from the storage-maintainer to /etc/ on both of my cluster-hosts and added the nfs-share via web-gui.
after that i manually edited /etc/pve/storage.cfg to this:

[FONT=Menlo]nfs: hrz_jurvms[/FONT][FONT=Menlo]    export /ibm/gpfs/share/home[/FONT]
[FONT=Menlo]    server fs.xxxxxx.de[/FONT]
[FONT=Menlo]    path /mnt/pve/hrz_jurvms[/FONT]
[FONT=Menlo]    options vers=4,sec=krb5[/FONT]
[FONT=Menlo]    content images,backup[/FONT]
[FONT=Menlo]    [/FONT][FONT=Menlo]maxfiles 8[/FONT]

(i added the "vers=4,sec=krb5" line)

It wouldn't mount. I am actually not sure, where i can find some detailed logs about this pve actions. So i continued trying to mount manually via console.

command:

[FONT=Menlo]mount.nfs4 fs.xxxxxx.de:/ibm/gpfs/share/home /mnt/hrz -o vers=4,sec=krb5 -vvv[/FONT]

result:

[FONT=Menlo]mount.nfs4: timeout set for Mon Dec  7 15:02:07 2015[/FONT]
[FONT=Menlo]mount.nfs4: trying text-based options 'vers=4,sec=krb5,addr=131.xxxxx.226,clientaddr=131.xxxxx.2'[/FONT]
[FONT=Menlo]mount.nfs4: mount(2): Invalid argument[/FONT]
[FONT=Menlo]mount.nfs4: an incorrect mount option was specified[/FONT]

- I have installed krb5-user via apt-get install.
- I tried different styles of the mount command, like "nfsvers=4" and so. Problem seems to be the "sec=krb5" options. This one just gives the described error.

Any ideas? Where to investigate further?

Thanks a lot

 

[mir]

Try these howto/wiki
- https://wiki.debian.org/NFS/Kerberos
- https://help.ubuntu.com/community/NFSv4Howto
- http://wiki.linux-nfs.org/wiki/index.php/Nfsv4_configuration

 

[juraedv]

Thanks mate, i've checked your submitted tutorials, but none of them solve my issue: Using the option

-o sec=krb5

mount.nfs4 throws this error:

[COLOR=#333333][FONT=Menlo]mount.nfs4: an incorrect mount option was specified[/FONT][/COLOR]

But this is exactly the line everybody seems to use. I don't get it...

 

[mir]

you have installed kerberos libraries and run kinit?

 

[juraedv]

you have installed kerberos libraries and run kinit?

I`ve installed krb5-user and already run

echo PASWWORT | kinit USER

which worked with no errors. No effect on the nfs error.

 

[mir]

And also libpam-krb5 and modprobe rpcsec_gss_krb5

What does ktutil display?

 

[juraedv]

[FONT=Menlo]root@remus:~# ktutil[/FONT]
[FONT=Menlo]ktutil:  read_kt /etc/krb5.keytab [/FONT]
[FONT=Menlo]ktutil:  list[/FONT]
[FONT=Menlo]slot KVNO Principal[/FONT]
[FONT=Menlo]---- ---- ---------------------------------------------------------------------[/FONT]
[FONT=Menlo]   1    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   2    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   3    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   4    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   5    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   6    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   7    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   8    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo] [/FONT]

I did as you said and installed libpam-krb5 and did the modprobe. This doesn't change the behavior of mount.nfs4. Still sees sec=krb5 as an invalid option.

 

[mir]

Is the file share exported with option sec=krb5?

 

[juraedv]

Partially Solved:

I could solve the mount issue and successfully mount via console after i edited /etc/defaults/nfs-common:

[FONT=Menlo]
[/FONT]
[FONT=Menlo]# If you do not set values for the NEED_ options, they will be attempted[/FONT]
[FONT=Menlo]# autodetected; this should be sufficient for most people. Valid alternatives[/FONT]
[FONT=Menlo]# for the NEED_ options are "yes" and "no".[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Do you want to start the statd daemon? It is not needed for NFSv4.[/FONT]
[FONT=Menlo]NEED_STATD=[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Options for rpc.statd.[/FONT]
[FONT=Menlo]#   Should rpc.statd listen on a specific port? This is especially useful[/FONT]
[FONT=Menlo]#   when you have a port-based firewall. To use a fixed port, set this[/FONT]
[FONT=Menlo]#   this variable to a statd argument like: "--port 4000 --outgoing-port 4001".[/FONT]
[FONT=Menlo]#   For more information, see rpc.statd(8) or http://wiki.debian.org/SecuringNFS[/FONT]
[FONT=Menlo]STATDOPTS=[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Do you want to start the idmapd daemon? It is only needed for NFSv4.[/FONT]
[FONT=Menlo]NEED_IDMAPD=yes[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Do you want to start the gssd daemon? It is required for Kerberos mounts.[/FONT]
[FONT=Menlo]NEED_GSSD=yes[/FONT]

(needed to add "yes" to the last two entries)


Only thing i couldn't manage by now is to get the nfs-mount into my two-hosts cluster via web-gui. I added the storage in the web-gui and manually edited "/etc/pve/storage.cfg" and added "vers=4,sec=krb5":

I also made sure, that kinit USER (with correct passwort) is executed on startup.

But web-gui doesnt mount / not showing content.

 

[juraedv]

Yes - see my reply to my own previous message. I could manage mounting the share manually. Now i have the issue with the cluster mounting it, so that i can use it "globally"

 

[mir]

Maybe the proxmox daemon pvedaemon does not recognize and/or honor the option sec? I am afraid you must ask the proxmox team to have this clarified.

 

[juraedv]

Well thanks mate, anyways!

Any suggestion, how to contact those guys?

 

[mir]

If you have a subscription you could open a support ticket otherwise hope one from the team replies to this thread.

 

[dietmar]

Maybe the proxmox daemon pvedaemon does not recognize and/or honor the option sec? I am afraid you must ask the proxmox team to have this clarified.

We simply pass the option to the mount command, so that should work.

 

[mir]

Could it then be a session and/or permission related problem? (it seems OP tests from the command line as user root)

 

[pascal]

Sorry to revive this old thread, but I stumbled across this while looking to a solution to my problem:
I could not access a nfs4 share which was restricted to a freeipa/ldap user/group (chmod 770). I copied a user keytab to my proxmox instance and do a kinit via cron. Getting a tgt for that user worked. but when I tried to access the mounted share, it would get a "permission denied".
I also noticed I did not get a nfs service ticket. If I set the permissions to 777 it worked.
When i switched to the freeipa user on the proxmox host using the same keytab it worked fine. Also I got a nfs service ticket.

After some searching I found the following option for rpc.gssd:

-n
By default, rpc.gssd treats accesses by the user with UID 0 specially, and uses "machine credentials" for all accesses by that user which require Kerberos authentication. With the -n option, "machine credentials" will not be used for accesses by UID 0. Instead, credentials must be obtained manually like all other users. Use of this option means that "root" must manually obtain Kerberos credentials before attempting to mount an nfs filesystem requiring Kerberos authentication.

So I modified "/lib/systemd/system/rpc-gssd.service" and added the "-n" parameter to the ExecStart Line.
I can now finally use a user keytab to access my share. Hope this helps somebody.