NFS4 and Kerberos

juraedv

Renowned Member
Dec 7, 2015
11
1
68
Hey folks,

we are running proxmox in newest version on two host-maschines as a cluster. we have several nfs-shares mounted over the web-gui and it works fine.
now i wanted to add a nfs4-share which uses kerberos-authentication. i placed the krb5.keytab i got from the storage-maintainer to /etc/ on both of my cluster-hosts and added the nfs-share via web-gui.
after that i manually edited /etc/pve/storage.cfg to this:

Code:
[FONT=Menlo]nfs: hrz_jurvms[/FONT][FONT=Menlo]    export /ibm/gpfs/share/home[/FONT]
[FONT=Menlo]    server fs.xxxxxx.de[/FONT]
[FONT=Menlo]    path /mnt/pve/hrz_jurvms[/FONT]
[FONT=Menlo]    options vers=4,sec=krb5[/FONT]
[FONT=Menlo]    content images,backup[/FONT]
[FONT=Menlo]    [/FONT][FONT=Menlo]maxfiles 8[/FONT]


(i added the "vers=4,sec=krb5" line)

It wouldn't mount. I am actually not sure, where i can find some detailed logs about this pve actions. So i continued trying to mount manually via console.

command:
Code:
[FONT=Menlo]mount.nfs4 fs.xxxxxx.de:/ibm/gpfs/share/home /mnt/hrz -o vers=4,sec=krb5 -vvv[/FONT]

result:
Code:
[FONT=Menlo]mount.nfs4: timeout set for Mon Dec  7 15:02:07 2015[/FONT]
[FONT=Menlo]mount.nfs4: trying text-based options 'vers=4,sec=krb5,addr=131.xxxxx.226,clientaddr=131.xxxxx.2'[/FONT]
[FONT=Menlo]mount.nfs4: mount(2): Invalid argument[/FONT]
[FONT=Menlo]mount.nfs4: an incorrect mount option was specified[/FONT]

- I have installed krb5-user via apt-get install.
- I tried different styles of the mount command, like "nfsvers=4" and so. Problem seems to be the "sec=krb5" options. This one just gives the described error.

Any ideas? Where to investigate further?

Thanks a lot
 
Thanks mate, i've checked your submitted tutorials, but none of them solve my issue: Using the option

Code:
-o sec=krb5

mount.nfs4 throws this error:

Code:
[COLOR=#333333][FONT=Menlo]mount.nfs4: an incorrect mount option was specified[/FONT][/COLOR]


But this is exactly the line everybody seems to use. I don't get it...
 
you have installed kerberos libraries and run kinit?

I`ve installed krb5-user and already run

Code:
echo PASWWORT | kinit USER

which worked with no errors. No effect on the nfs error.
 
Code:
[FONT=Menlo]root@remus:~# ktutil[/FONT]
[FONT=Menlo]ktutil:  read_kt /etc/krb5.keytab [/FONT]
[FONT=Menlo]ktutil:  list[/FONT]
[FONT=Menlo]slot KVNO Principal[/FONT]
[FONT=Menlo]---- ---- ---------------------------------------------------------------------[/FONT]
[FONT=Menlo]   1    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   2    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   3    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   4    3 nfs/romulus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   5    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   6    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   7    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo]   8    3   nfs/remus.xxxxxxx.de@xxxxxxxx.DE[/FONT]
[FONT=Menlo] [/FONT]


I did as you said and installed libpam-krb5 and did the modprobe. This doesn't change the behavior of mount.nfs4. Still sees sec=krb5 as an invalid option.
 
Partially Solved:

I could solve the mount issue and successfully mount via console after i edited /etc/defaults/nfs-common:
Code:
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# If you do not set values for the NEED_ options, they will be attempted[/FONT]
[FONT=Menlo]# autodetected; this should be sufficient for most people. Valid alternatives[/FONT]
[FONT=Menlo]# for the NEED_ options are "yes" and "no".[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Do you want to start the statd daemon? It is not needed for NFSv4.[/FONT]
[FONT=Menlo]NEED_STATD=[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Options for rpc.statd.[/FONT]
[FONT=Menlo]#   Should rpc.statd listen on a specific port? This is especially useful[/FONT]
[FONT=Menlo]#   when you have a port-based firewall. To use a fixed port, set this[/FONT]
[FONT=Menlo]#   this variable to a statd argument like: "--port 4000 --outgoing-port 4001".[/FONT]
[FONT=Menlo]#   For more information, see rpc.statd(8) or http://wiki.debian.org/SecuringNFS[/FONT]
[FONT=Menlo]STATDOPTS=[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Do you want to start the idmapd daemon? It is only needed for NFSv4.[/FONT]
[FONT=Menlo]NEED_IDMAPD=yes[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo]# Do you want to start the gssd daemon? It is required for Kerberos mounts.[/FONT]
[FONT=Menlo]NEED_GSSD=yes[/FONT]

(needed to add "yes" to the last two entries)


Only thing i couldn't manage by now is to get the nfs-mount into my two-hosts cluster via web-gui. I added the storage in the web-gui and manually edited "/etc/pve/storage.cfg" and added "vers=4,sec=krb5":

I also made sure, that kinit USER (with correct passwort) is executed on startup.

But web-gui doesnt mount / not showing content.
 
Yes - see my reply to my own previous message. I could manage mounting the share manually. Now i have the issue with the cluster mounting it, so that i can use it "globally"
 
Maybe the proxmox daemon pvedaemon does not recognize and/or honor the option sec? I am afraid you must ask the proxmox team to have this clarified.
 
Maybe the proxmox daemon pvedaemon does not recognize and/or honor the option sec? I am afraid you must ask the proxmox team to have this clarified.

We simply pass the option to the mount command, so that should work.
 
Sorry to revive this old thread, but I stumbled across this while looking to a solution to my problem:
I could not access a nfs4 share which was restricted to a freeipa/ldap user/group (chmod 770). I copied a user keytab to my proxmox instance and do a kinit via cron. Getting a tgt for that user worked. but when I tried to access the mounted share, it would get a "permission denied".
I also noticed I did not get a nfs service ticket. If I set the permissions to 777 it worked.
When i switched to the freeipa user on the proxmox host using the same keytab it worked fine. Also I got a nfs service ticket.

After some searching I found the following option for rpc.gssd:
-n
By default, rpc.gssd treats accesses by the user with UID 0 specially, and uses "machine credentials" for all accesses by that user which require Kerberos authentication. With the -n option, "machine credentials" will not be used for accesses by UID 0. Instead, credentials must be obtained manually like all other users. Use of this option means that "root" must manually obtain Kerberos credentials before attempting to mount an nfs filesystem requiring Kerberos authentication.
So I modified "/lib/systemd/system/rpc-gssd.service" and added the "-n" parameter to the ExecStart Line.
I can now finally use a user keytab to access my share. Hope this helps somebody.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!