NFS3 works on one server, not the second

[dlewis]

We can successfully mount a share on our first proxmox server but, when we try to mount it on the second, we get the following error:
pvestatd[4164]: WARNING: mount error: mount.nfs: requested NFS version or transport protocol is not supported

Here is the entry in /etc/pve/storage.cfg:

nfs: openvz-nfs
    path /mnt/pve/openvz-nfs
    server [IP ADDR]
    export /mnt/nfs1/nfs-openvz/OF-NFS-1
    options vers=3
    content images,iso,vztmpl,rootdir,backup
    maxfiles 1

If I comment out the options line, it mounts, but as version 4. Likewise if I mount it from the command line using the mount command.

I've checked the version of mount, nfs-common, and other similar. All the same between the two servers.

Any ideas?

 

[svendsen]

If you mount with verbose output, what do you get?

mount -v -t nfs -o vers=3,nvsvers=3 server://mnt/nfs1/nfs-openvz/OF-NFS-1 /mnt/pve/openvz-nfs

 

[dlewis]

# mount -v -t nfs -o vers=3,nvsvers=3 [SERVER]://mnt/nfs1/nfs-openvz/OF-NFS-1 /mnt/tmp
mount.nfs: timeout set for Sat Jun 28 09:24:08 2014
mount.nfs: trying text-based options 'vers=3,nvsvers=3,addr=[SERVER]'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying [SERVER] prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: requested NFS version or transport protocol is not supported

# service rpcbind status
rpcbind is running.

# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  48946  status
    100024    1   tcp  34850  status

Interestingly, when I run the same mount command on the working proxmox box, it fails too and I get the same output. Yet proxmox mounts it just fine with version 3:

root@[WORKING-PROXMOX]:~# mount -l
...
[SERVER]:/mnt/nfs1/nfs-openvz/OF-NFS-1 on /mnt/pve/openvz-nfs type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[SERVER],mountvers=3,mountport=42528,mountproto=udp,local_lock=none,addr=[SERVER])

 

[dlewis]

Fixed a typo ("nvsvers" vs "nfsvers") but no change - same responses.

It looks to me like it's getting through to prog 100003, (nfs), but can't contact 100005, which should be mountd. Am I reading that right? Is it trying to contact mountd running on the local client or on the server?

 

[dlewis]

Telling mount the port to use for mountd works:

mount -v -t nfs -o vers=3,nfsvers=3,mountport=44939,mountproto=udp [SERVER]://mnt/nfs1/nfs-openvz/OF-NFS-1 /mnt/tmp

Not really an option long term and it doesn't explain why mounting works just fine on node 1. But I would say that indicates the problem might be that mountd is not registered with rpcbind on the server, right?

On server:

# ps auxw | grep mountd       
root      7672  0.0  0.0  17348  1324 ?        Ss   09:20   0:00 rpc.mountd

# rpcinfo -p                                                                                                 
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100011    1   udp    625  rquotad
    100011    2   udp    625  rquotad
    100011    1   tcp    628  rquotad
    100011    2   tcp    628  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100021    1   udp  42594  nlockmgr
    100021    3   udp  42594  nlockmgr
    100021    4   udp  42594  nlockmgr
    100021    1   tcp  55313  nlockmgr
    100021    3   tcp  55313  nlockmgr
    100021    4   tcp  55313  nlockmgr

 

[mir]

Is there something in /etc/hosts.allow /etc/hosts.deny

 

[dlewis]

They are empty on all three machines.

 

[mir]

Different settings for local firewall on the servers in question?

Try nmap between machines: sudo nmap -T4 <IP>

 

[dlewis]

iptables is off on all three.

nmap shows 2049 and 111 available from both nodes. And I know I can hit at least one port that mountd is listening on - as the share mounts if I specify the port. (mount just can't get the port from rpcbind.)

I've learned a lot about NFS over the last few days but I'm still unsure - with the newer versions of NFS, mountd should still be registered with rpcbind, right?

 

[mir]

NFSv3: port 111 and 2049 are in play
NFSv4: Only port 2049 is in play

Is the Domain configured identically on all servers in /etc/idmapd.conf

 

[dlewis]

idmapd.conf

On server:

[General]
Verbosity = 0
Pipefs-Directory = /var/lib/rpc_pipefs
Domain = localdomain

[Mapping]
Nobody-User = nobody
Nobody-Group = nobody

On clients:

[General]
Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs

[Mapping]
Nobody-User = nobody
Nobody-Group = nogroup