NFS Shares: storage is not online (500)

I

idurey

New Member
Jun 14, 2012
4
0
1
I'm encountering a strange issue today after upgrading the system to the latest version:

I can mount an NFS share from the command line but when I attempt to browse the contents from the ProxMox UI I get "storage 'Backups' is not online (500)".

If I delete the share and re-add it using the UI the share does not appear in /mnt/pve/ as expected.

Here is /etc/pve/storage.cfg
Code: 
nfs: Backups        
        path /mnt/pve/Backups
        server FIRS01
        export /NFS
        options vers=3
        content backup
        maxfiles 10

Here is mounting the NFS share from command line and viewing contents of a test file:
Code: 
mount -t nfs firs01:/NFS/ /mnt/pve/Backups/
root@pmhost3:/mnt/pve# cd /mnt/pve/Backups/
root@pmhost3:/mnt/pve/Backups# ls
dump  test.txt
root@pmhost3:/mnt/pve/Backups# cat test.txt 
Successfully connected to NFS share.

Is there a log somewhere I can use to figure out why this isn't working?

Thanks,

Ian
 
dietmar said:
any hint in /var/log/syslog? What is the output of

# pvesm list Backups
Click to expand...


Well this is very odd. As a workaround overnight I removed the Backups storage and created a new NFS mount on each node at /mnt/pve/Backup2 and created a new Storage Dir using the UI called Backup2. I changed my backup jobs to back up to Backup2 and tested them, everything was working fine.

Today I read your message and sat down to do some troubleshooting. Once again I added a new NFS storage called Backups and was ready to comb through the syslog when suddenly everything works.

I made no other changes to the system and I did not reboot. Is it possible that changing the location which the backup jobs point to resolved the issue?

In any case, everything is fine now. Thank you for your time and your suggestions, I didn't think to look in syslog.
 
Hi,

I also get this error on my cluster after upgrading it from Proxmox 1.x to 2.0
The only workaround to re-enable the NFS shares (at least for some days) is to restart the nfs server on my NAS (Synology).

How ever, after the time the error comes back, the nfs shares seams to be offline again and I have to restart the nfs server once more.

Any suggestions?
Regards Peter
 
idurey said:
The best solution I found was to stop using ProxMox.
Click to expand...

Hmm... This is the only one issue with Proxmox until now, so I would really like to continue using it :)
Why are you so unhappy with Proxmox? Any other problems?

cu Peter
 
nik-weter said:
Had the same problem. Disabled ipv6 and nfs4. everything works.
Click to expand...

Hi proxmox check if the nfs storage is online with command

"/usr/bin/rpcinfo -p ipofnfsserver"

with a timeout of 2sec.


So maybe is the timeout too short for your storage under load ?
Can you try to change it in file (at the end) :

/usr/share/perl5/PVE/Storage/NFSPlugin.pm

and restart

/etc/init.d/pvestatd restart
/etc/init.d/pvedaemon restart
 
Sorry to excave an old post, but with fresh install of proxmox 3.1 I can reproduce this problem everytime between some Dell workstations, and a QNAP TS-469L NAS or any other linux servers/VM on the local network.

I tested a dell optiplex 745 with broadcom network chipset, and a dell T3600 with a TG3 network chipset.

Code: 
# rpcinfo -p 172.16.66.66
rpcinfo: can't contact pormapper: RPC: Remote system error - Connection timed out
So proxmox complains about storage being offline, I won't blame the proxmox storage daemon on this : rpcinfo -p should not timeout.

The problem is that :
- Other linux boxes on the network do not suffer from this problem.
- The proxmox server v3.0 upgraded to v3.1 do not suffer from this problem.
- Without -p option, "rpcinfo 172.16.66.66" finally works after a very long time.
- When running "rpcinfo -p serverip" Wireshark show a SYN but no ACK in reply (hence the timeout)
- When running "telnet serverip 111" Wireshark show a SYN and ACK in reply (explain me this)

On top of that, this mount command line works :
Code: 
# mount -t nfs -o vers=3,udp serverip:/Backup /mnt/pve/Backup

But this mount command doesn't works :
Code: 
# mount -t nfs -o vers=3 serverip:/Backup /mnt/pve/Backup

This problem feels very wrong and bizarre. The switch is an HP 2530-24G, and there is no trace of firewall installed.

The pveversion is :
Code: 
root@secours:~# pveversion -v
proxmox-ve-2.6.32: 3.1-109 (running kernel: 2.6.32-23-pve)
pve-manager: 3.1-3 (running version: 3.1-3/dc0e9b0e)
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-7
qemu-server: 3.1-1
pve-firmware: 1.0-23
libpve-common-perl: 3.0-6
libpve-access-control: 3.0-6
libpve-storage-perl: 3.0-10
pve-libspice-server1: 0.12.4-1
vncterm: 1.1-4
vzctl: 4.0-1pve3
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.0-2

The local rpc services show no lockd daemon running (nlockmgr) ; couldn't find any clue about how to start the rpc.lockd kernel process ; or why doesn't it runs.
Code: 
root@secours:~# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  51434  status
    100024    1   tcp  55586  status

Since the problem arises with any other server (not only the NAS), I guess the NAS is not to blame, and rebooting the NAS has no effect on this, I tried removing the IPV6 (following a tutorial on internet) with no success.

Does somebody have any clue about this problem ? It drives me nuts.
How did you managed to disable the NFSv4 and IPV6 on the proxmox host ?
 
Onliest to register. Recently i had problem with the access on my nfs storage.
I'm using Proxmox 4.2
The message that i receive was "NFS_Storage is not online"

I read on https://forums.freenas.org/index.php?threads/nfs-mount-times-out.7270/ :
"During the mount process, NFS apparently does a reverse DNS lookup to see if the address is valid. On a local network there is no way to do that unless you add the IP address of the client to the hosts file on the server. The hosts file even mentions that in the documentation in the beginning of the file. Once we did that the mount connected very quickly."

So I register my servers on /etc/hosts of FreeNAS. And the message changed to "NFS version ..."
Then I changed on FreeNAS to permit use NFS v4.

But when the FreeNAS reboot the /etc/hosts is rewrited, so to make the register of hosts permanent need to register the hosts on FreeNAS.
"System > Network", on field "Host name data base"

And it works.
 
Last edited:
Thanks Bruno - your post lead me to the issue I was having. I was getting the same Storage is not online message. I checked everything on Proxmox side and it was ok. When I saw your post I went to the network setting on the FreeNAS and noticed that the DNS 1 was not valid anymore (I replaced it a week ago and the ip address changed), the DNS2 and 3 had a valid public DNS ips 8.8.8.8 and 4.2.2.2 but I guess it was not enough, go figure...

Once I adjusted the primary DNS setting even with no host name database entry for Proxmox it started to work immediately.

I am on PVE 4.2 and FreeNAS 9.10

Thank you
 
  • Like
Reactions: Bruno Emanuel
I just experienced this issue on PVE 4.2. It seems that my session had expired. I checked via ssh on the server and all was fine. I logged out of the UI, logged back in and it was fine.
 
pretty old thread but I did experiment the same issue today,
i was able to connect in the shell with a classic mount, but got always an error on the Proxmox side,

I got it finally working by editing /etc/pve/storage.cfg
i noticed that proxmox was using nfs version 3, but my nfs server at ovh wanted nfs 4
so by just changing the line
options vers=3
to
options vers=4

then everything worked fine in proxmox interface

maybe it can help someone
 
  • Like
Reactions: peterge, Xabi and SamTzu
OMG, thank you, thank you for this post! I accidentally added my phile to SAN DHCP and it shattered my cluster, I was getting "storage offline" messages everywhere. {heart attack ends}
 
My Proxmox cluster had been in operation for 4 months when I encountered this error. It happened when I was mass-migrating disks from iSCSI to NFS.

Cluster Details
Proxmox version: 5.3
Storage: FreeNAS 11.2 serving iSCSI and NFS

Problem
I was switching the backing store of several VMs from iSCSI to NFS (QCoW2) storage when I encountered the error TASK ERROR: storage 'nfs-infra-ssd-1' is not online. In one of the PVE nodes, the NFS storage showed up as inactive.
Code: 
# pvesm status
...
nfs-infra-ssd-1                   nfs   inactive               0               0               0    0.00%

Solution
I didn't manage to figure out what was the issue but remounting the share made it active again.
Code: 
umount /mnt/pve/nfs-infra-ssd-1
mount -t nfs freenas.local:/mnt/pool-ssd-1/ds-infra-ssd-1 nfs-infra-ssd-1

There's no reason why this should work, but it did. Hope it helps someone out there who encountered this issue.

16 May 2019 update: The problem resurfaced on 1 out of 4 PVE nodes after an upgrade from 5.3 to 5.4 followed by a reboot. The above method didn't work but this instantly solved it:
Code: 
pvesm scan nfs ip.of.nfs.storage
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back