One node unable to access NFS share

[cdukes]

Everything has been working until I made some changes on the network, but now I don't know what I messed up

If I remove the NFS share for all nodes and re-add it, all nodes except one (name is pve5) map it fine.
On pve 5, I see this in the logs:

Apr 8 15:25:02 pve5 pvestatd[3411]: unable to activate storage 'cartman' - directory '/mnt/pve/cartman' does not exist or is unreachable

But if I mount the share manually using the command line, it mounts fine. I can ping the NFS server, ssh to it, etc. I've also verified that it's the right server (mac address) and can also get a `showmount -e cartman`.

Any idea why this might be happening?

 

[Stoiko Ivanov]

* please post your `/etc/pve/storage.cfg`
* post `/etc/fstab` from the affected node and one where it works
* does `/mnt/pve/cartman` exist on all nodes? (`ls -latr /mnt/pve`)

 

[cdukes]

* please post your `/etc/pve/storage.cfg`
* post `/etc/fstab` from the affected node and one where it works
* does `/mnt/pve/cartman` exist on all nodes? (`ls -latr /mnt/pve`)

/mnt/pve/cartman does exist on all nodes, but on pve5, I can't `ls` or `df` because it's hung. if I `umount -l` and then mount it from the shell using `mount -t nfs cartman:/volume1/proxmox /mnt/pve/cartman`, it mounts and works in the shell, but not in the gui.

Storage config:

root@pve5 [~]: # cat /etc/hosts | grep -i cartman
10.16.28.2 cartman
root@pve5 [~]: #

root@pve5 [~]: # cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content vztmpl
    shared 0

lvmthin: local-lvm
    thinpool data
    vgname pve
    content rootdir,images
    nodes pve0

nfs: tank0
    export /tank0/corp/proxmox
    path /mnt/pve/tank0
    server 10.16.28.202
    content rootdir,images,vztmpl,iso
    maxfiles 2
    options vers=4.1,nolock,noatime,tcp

nfs: tank0-ISOs
    export /tank0/corp/ISOs
    path /mnt/pve/tank0-ISOs
    server 10.16.28.202
    content iso
    options vers=4.1,nolock,noatime,tcp

nfs: tank0-proxmox
    export /tank0/corp/proxmox
    path /mnt/pve/tank0-proxmox
    server 10.16.28.202
    content rootdir,images,vztmpl
    maxfiles 0
    options vers=4.1,nolock,noatime,tcp

nfs: tank0-backups
    export /tank0/corp/backups
    path /mnt/pve/tank0-backups
    server 10.16.28.202
    content backup
    maxfiles 2
    options vers=4.1,nolock,noatime,tcp

nfs: cartman
    export /volume1/proxmox
    path /mnt/pve/cartman
    server cartman
    content images
    options vers=3

fstab:

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=D1D2-ED11 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/sdb1 /sdb ext4 errors=remount-ro 0 1
pve4:/tank0/corp/lab /lab   nfs rw,auto,noatime,bg,tcp,rsize=65536,wsize=65536 0 0
/dev/nvme_vg/nvme    /nvme    ext4    noatime,errors=remount-ro    00

 

[Stoiko Ivanov]

/mnt/pve/cartman does exist on all nodes, but on pve5, I can't `ls` or `df` because it's hung. if I `umount -l` and then mount it from the shell using `mount -t nfs cartman:/volume1/proxmox /mnt/pve/cartman`, it mounts and works in the shell, but not in the gui.

hm - that sounds like the connection to the nfs-share is not working too well for pve5 - and this usually results in quite a few problems (hung VMs, and some problems only can be resolved with a node-reboot)

check `dmesg` and `journalctl -b` for some pointers of where the problems might be

 

[cdukes]

hm - that sounds like the connection to the nfs-share is not working too well for pve5 - and this usually results in quite a few problems (hung VMs, and some problems only can be resolved with a node-reboot)

check `dmesg` and `journalctl -b` for some pointers of where the problems might be

dmesg says:

[Tue Apr  9 10:45:01 2019] nfs: server cartman not responding, timed out
[Tue Apr  9 10:45:01 2019] nfs: server cartman not responding, timed out
[Tue Apr  9 10:45:07 2019] nfs: server cartman not responding, timed out
[Tue Apr  9 10:45:07 2019] nfs: server cartman not responding, timed out

journalctl -b has:

Apr 09 10:49:16 pve5 pvestatd[3411]: got timeout
Apr 09 10:49:16 pve5 pvestatd[3411]: unable to activate storage 'cartman' - directory '/mnt/pve/cartman' does not exist or is unreachable

But these message make no sense. Does PVE store the ip for this box somewhere instead of the name? I've even removed that storage in the UI and added it via IP address. All other nodes connect, just not this one.
I have no VMs on this server at the moment and have rebooted several times.

 

[guletz]

Hi @kdukes,

Can you verify after your nfs is mounted using command line what show the "mount" output(on pve5)? And on any other node how is mount output(for the same nfs share)?

 

[cdukes]

Hi @kdukes,

Can you verify after your nfs is mounted using command line what show the "mount" output(on pve5)? And on any other node how is mount output(for the same nfs share)?

Working server (pve4):
cartman:/volume1/proxmox on /mnt/pve/cartman type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.28.2,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.16.28.2)

pve5 server:
cartman:/volume1/proxmox on /mnt/pve/cartman type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.28.2,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.16.28.2)

Side question: How do I force nfs v4?

 

[guletz]

Hi,

Try to use:

options vers=4.1,nolock,noatime,tcp

instead of:

options vers=3

for the cartman share!