NFS Storage Status Unkown

marh · 2025-01-30T12:48:42+0100

Hey there,

we have an issue with one of our Storages at the moment:

Specs:

3x PVE Nodes
1x Synology NFS Share

The setup works fine for 1 year right now, never had any issues.
Somehow one of our Storages (both are connected with the same Synology) is in Status "unkown".

Below you find some outputs of following commands:

System Log of PVE1:

Code:

Jan 30 12:39:58 pve01 pvedaemon[1823967]: storage 'SynologyNFS' is not online
Jan 30 12:40:04 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 12:40:04 pve01 pvestatd[1560]: status update time (10.662 seconds)
Jan 30 12:40:15 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 12:40:15 pve01 pvestatd[1560]: status update time (10.694 seconds)
Jan 30 12:40:25 pve01 pveproxy[3425947]: worker exit
Jan 30 12:40:25 pve01 pveproxy[1591]: worker 3425947 finished
Jan 30 12:40:25 pve01 pveproxy[1591]: starting 1 worker(s)
Jan 30 12:40:25 pve01 pveproxy[1591]: worker 3446358 started
Jan 30 12:40:26 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 12:40:26 pve01 pvestatd[1560]: status update time (10.675 seconds)
Jan 30 12:40:31 pve01 pvedaemon[1881175]: storage 'SynologyNFS' is not online
Jan 30 12:40:36 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 12:40:37 pve01 pvestatd[1560]: status update time (10.644 seconds)
Jan 30 12:40:47 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 12:40:47 pve01 pvestatd[1560]: status update time (10.629 seconds)

Code:

root@pve01:/mnt/pve/SynologyNFS# pvesm status
storage 'SynologyNFS' is not online
Name               Type     Status           Total            Used       Available        %
NUERZ01PBS          pbs     active    111325536640     10282185344    101043351296    9.24%
RECOMIG             nfs     active    111325536640     10282185344    101043351296    9.24%
SynologyNFS         nfs   inactive               0               0               0    0.00%
local               dir     active        69573880        16971084        49022892   24.39%
local-lvm       lvmthin     active       143773696           14377       143759318    0.01%

Code:

root@pve01:/mnt/pve/SynologyNFS# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

nfs: SynologyNFS
        export /volume1/NUERZ01NFS
        path /mnt/pve/SynologyNFS
        server 172.22.13.20
        content rootdir,snippets,vztmpl,iso,backup,images
        prune-backups keep-all=1

pbs: NUERZ01PBS
        datastore NUERZ01PBS
        server 172.22.11.100
        content backup
        fingerprint e2:c6:4c:60:a7:cc:e3:XXXXXXXXXXXXXXXXXXXX                                                                                                                                                            b4:07:12:76:e6:02:86:16:12:89:82:07
        namespace NUERZ01
        prune-backups keep-all=1
        username nuerz01@pbs

nfs: RECOMIG
        export /volume1/RECOMIG
        path /mnt/pve/RECOMIG
        server 172.22.11.20
        content snippets,rootdir,vztmpl,iso,images,backup
        prune-backups keep-all=1

Code:

Jan 30 11:52:00 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 11:52:11 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 11:52:21 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
Jan 30 11:52:32 pve01 pvestatd[1560]: storage 'SynologyNFS' is not online
root@pve01:~# journalctl | grep -i nfs > /tmp/nfslog.log
root@pve01:~# ping 172.22.13.20
PING 172.22.13.20 (172.22.13.20) 56(84) bytes of data.
64 bytes from 172.22.13.20: icmp_seq=1 ttl=64 time=0.261 ms
64 bytes from 172.22.13.20: icmp_seq=2 ttl=64 time=0.167 ms
64 bytes from 172.22.13.20: icmp_seq=3 ttl=64 time=0.217 ms
64 bytes from 172.22.13.20: icmp_seq=4 ttl=64 time=0.173 ms
^C
--- 172.22.13.20 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3069ms
rtt min/avg/max/mdev = 0.167/0.204/0.261/0.037 ms
root@pve01:~# cd /mnt/
pve/     vzsnap0/
root@pve01:~# cd /mnt/pve/
RECOMIG/     SynologyNFS/ SynologySMB/
root@pve01:~# cd /mnt/pve/SynologyNFS/
root@pve01:/mnt/pve/SynologyNFS# ls -l
total 0
drwxrwxrwx 1 root   root   152 Sep  7 10:52 dump
drwxrwxrwx 1 root   root   150 Jan  9 15:23 images
drwxrwxrwx 1   1026 users    0 Jul 10  2024 IMPORT_FROM_CLIENTS
drwxrwxrwx 1 root   root     0 Mar 12  2024 private
drwxrwxrwx 1 root   root     0 Mar 12  2024 snippets
drwxrwxrwx 1 root   root    16 Mar 12  2024 template
drwxrwxrwx 1 backup backup  80 Jan 30 02:02 tmp
root@pve01:/mnt/pve/SynologyNFS# mkdir TEST

I can add an directory on /mnt/pve/SynologyNFS/
Which also is shown on the Synology itself.

Do you have any known issues or tips how we can fix this, for reminder the Storage "RECOMIG" is also on the same Synology which has not the failure.

Code:

proxmox-ve: 8.2.0 (running kernel: 6.8.12-1-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-1
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
proxmox-kernel-6.8.8-3-pve-signed: 6.8.8-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.2
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.13-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.2-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Thanks for helping!

waltar · 2025-01-30T13:26:59+0100

What about a "systemctl restart pvestatd", wait 1min and in webui status of your syno is back fine ?

bbgeek17 · 2025-01-30T14:11:19+0100

The health probes are populated by "pvestatd" as @waltar mentioned.

The NFS health probes are based on successful run/output of "rpcinfo", "showmount" and "ping". Ensure those are working fine. Check the MTUs along the way.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

marh · 2025-01-30T14:31:53+0100

waltar said:
What about a "systemctl restart pvestatd", wait 1min and in webui status of your syno is back fine ?

I used a spare node which has no important VM's on it... with following result:

Code:

root@pve01:/mnt/pve/SynologyNFS# time pvesm nfsscan 172.22.13.20
/volume1/BUSCHPERSONAL 192.168.99.148
/volume1/NUERZ01NFS    172.22.11.0/24
/volume1/NUERZ01PBS    172.22.11.100,172.22.11.13,172.22.11.12,172.22.11.11
/volume1/RECOMIG       172.22.11.0/24

real    1m0.727s
user    0m0.581s
sys     0m0.093s
root@pve01:/mnt/pve/SynologyNFS#

bbgeek17 · 2025-01-30T14:36:49+0100

marh said:
real 1m0.727s

You have a network problem. PVE is using underlying, standard, Linux NFS stack. You will likely get the same results when you run:
/usr/sbin/rpcinfo -T tcp $ip nfs 4
/sbin/showmount --exports $server

You need to make sure the commands about are working reasonably.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

waltar · 2025-01-30T15:13:46+0100

Yes, network or nfs-server problem, should be answered in under 2sec (you could even ask other servers even without exports to check normal answer time).

ghusson · 2025-01-30T17:00:15+0100

Can you list the content of the NFS for example ? And in how many time is it done ?
time ls /mnt/pve/SynologyNFS

marh · 2025-01-30T18:56:49+0100

ghusson said:
Can you list the content of the NFS for example ? And in how many time is it done ?
time ls /mnt/pve/SynologyNFS

Code:

root@pve01:~# time ls /mnt/pve/SynologyNFS

dump  images  IMPORT_FROM_CLIENTS  private  snippets  template  TEST  tmp

real    0m0.003s
user    0m0.000s
sys     0m0.002s

root@pve01:~#

waltar said:
Yes, network or nfs-server problem, should be answered in under 2sec (you could even ask other servers even without exports to check normal answer time).

Why should then, the other NFS Share work which is also on the same Synology?

Search

Search

NFS Storage Status Unkown

marh

New Member

waltar

Renowned Member

bbgeek17

Distinguished Member

marh

New Member

bbgeek17

Distinguished Member

waltar

Renowned Member

ghusson

Renowned Member

marh

New Member

We value your privacy