NFS Storage mountpoint vanished

[Knuuut]

Hi,
after adding a second nfs-storage to the cluster, this storage fails after exactly 30 minutes. Cause the mountpoint /mnt/pve/nfs2 vanished, but still listed in the output of mount. This is reproduceable. The first nfs storage isn't affected at all. The second nfs server got the same hardware, software and configuration as the first one.

Any ideas are welcome.

Cheers Knuuut

 

[oguz]

Hi,

Can you post the output of:

nfsstat -m
pvesm status

 

[Knuuut]

nfsstat -m
/mnt/pve/nfs01 from x.x.x.101:/data/nfs01
 Flags: rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=x.x.x.101,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=x.x.x.101

/mnt/pve/nfs02 from x.x.x.102:/data/nfs02
 Flags: rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=x.x.x.102,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=x.x.x.102

This is before the vanish

 pvesm status
Name                      Type     Status           Total            Used       Available        %
Pool1             rbd     active      3475292973      1390462253      2084830720   40.01%
Pool2         rbd     active      3444390674      2626337362       818053312   76.25%
Pool3                    rbd     active     13342379975      5764396487      7577983488   43.20%
nfs01                   nfs     active     39056246784      2756675584     36299571200    7.06%
nfs02                   nfs     active     36524119040          890880     36523228160    0.00%
local                      dir     active        98559220         2620332        90889340    2.66%
local-lvm              lvmthin     active       811073536               0       811073536    0.00%

 

[oguz]

This is reproduceable.

What are the steps to reproduce? I might be able to help if I'm able to reproduce it myself.

Can you also post `pveversion -v`?

 

[Knuuut]

syslog

Apr 03 12:30:37 pmve-1901 pvedaemon[2089411]: unable to activate storage 'nfs02' - directory '/mnt/pve/nfs02' does not exist or is unreachable

What are the steps to reproduce?

Just setting up a nfs-storage via gui. No problems at all

After the mountpoint vanished, i do a umount -f /mnt/pve/nfs02 on every node, then the storage comes back. Even if I delete it completely from the storage and set it up from scratch, it always disappears after 30 minutes from all nodes.

pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-11-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)
pve-kernel-4.15: 5.3-2
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.13.13-2-pve: 4.13.13-33
ceph: 12.2.11-pve1
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-47
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-12
libpve-storage-perl: 5.0-39
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-23
pve-cluster: 5.0-33
pve-container: 2.0-35
pve-docs: 5.3-3
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-18
pve-firmware: 2.0-6
pve-ha-manager: 2.0-8
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-2
pve-xtermjs: 3.10.1-2
qemu-server: 5.0-47
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

 

[oguz]

I don't think this is a Proxmox related problem. Probably a network issue or a configuration issue on NFS.

but still listed in the output of mount.

We don't unmount so this is normal.

The second nfs server got the same hardware, software and configuration as the first one.

What kind of NFS box are you using? If possible, can you attach the syslog from the NFS? If nothing useful comes out of the syslog, you can try to enable debugging on NFS.

 

[Knuuut]

I don't think this is a Proxmox related problem. Probably a network issue or a configuration issue on NFS.

It has to be, because this nfs share is also mounted on another linux box, on which was no trouble at all.

Anyway, after countless mount, umount, create, delete, restarts of nfs service with no(!) change of configurations, this problem disappered and I don't know why. But if it stays like now, I'll be happy.

Thanx

Knuuut