NFS Shares: storage is not online (500). Why?

P

PedroDelgado

New Member
Aug 17, 2015
18
0
1
I have a Proxmox 5.1-35 (172.16.10.101) and FreeNAS 11.0-U4 (172.16.10.10) server. Everything was fine until few days ago, maybe 4 days, Proxmox stopped seeing NFS shares on FreeNAS. The message in /var/log/syslog:

Code: 
pve01 pvestatd[1277]: storage 'Backups' is not online

This happened for all NFS shares (Backups, ISOS and Containers). As this post says I can ping the FreeNAS and I even can
Code: 
mount 172.16.10.10:/mnt/RAID/Backups /mnt/pve/Backups
so there is not a network issue but a configuration issue I guess. Previous command line allows me to start the VMs and Containers but I cannot do backups. I tried doing backups through command line with an alternate path using --dumpdir and the message is the same. I don't understand why because the --dumpdir should override the NFS share Backups.

Code: 
vzdump --all --compress gzip --dumpdir /home/

If I do
Code: 
df -h
, the output:

172.16.10.10:/mnt/RAID/Backups 1.8T 46G 1.7T 3% /mnt/pve/Backups

Proxmox can even see the capacity of the NFS share through command line, however it can't show this very same info through WebGUI. What??????

Info on /etc/pve/storage.cfg for Backups NFS:

Code: 
nfs: Backups

    export /mnt/RAID/Backups
    path /mnt/pve/Backups
    server 172.16.10.10
    content backup,rootdir,images
    maxfiles 0
    options vers=4
 
what does
Code: 
showmount -e 172.16.10.10
show ?
 
Hi Pedro,

I can remember, when I got this message, it was always a problem with the nameserver-entry at the freenas side. This ist also mentioned in some other threads in this forum.

E.g. the nameserver-entry pointed to a VM (domain controller) that was down at the moment.

The problem was solved as I put the IP of the always running default-gw at the freenas-config, but I know this is not the solution, if you want the FreeNAS-Box as domain-member with the virtualised DC. I didn't research this any further.

greets,
vmanz
 
  • Like
Reactions: PedroDelgado
vmanz said:
Hi Pedro,

I can remember, when I got this message, it was always a problem with the nameserver-entry at the freenas side. This ist also mentioned in some other threads in this forum.

E.g. the nameserver-entry pointed to a VM (domain controller) that was down at the moment.

The problem was solved as I put the IP of the always running default-gw at the freenas-config, but I know this is not the solution, if you want the FreeNAS-Box as domain-member with the virtualised DC. I didn't research this any further.

greets,
vmanz
Click to expand...

Right now I have not a DNS entry for the FreeNAS or Proxmox on the DC because I'am not pointing to the name, I'm using just the IPv4 addresses. I guess that's what you mean with nameserver-entry. I have the default-gw (pfSense) also as an IPv4 on FreeNAS.
 
PedroDelgado said:
rpc mount export: RPC: Timed out
Click to expand...
this is the reason that pve thinks the storage is offline

AFAIR, when using nfs and freenas, the freenas host, wants to look up the names of the client
so you can try to enter the hostname/ip of the pve host into the /etc/hosts of the freenas server, or do this in your normal dns server
 
  • Like
Reactions: PedroDelgado
dcsapak said:
this is the reason that pve thinks the storage is offline

AFAIR, when using nfs and freenas, the freenas host, wants to look up the names of the client
so you can try to enter the hostname/ip of the pve host into the /etc/hosts of the freenas server, or do this in your normal dns server
Click to expand...

The same @vmanz suggested, however, there was no problem weeks before why now. I'm going to add the host on /etc/hosts and will tell you guys the outcome. Thanks both.
 
Ok I know this is a Proxmox forum, not a FreeNAS but if I manually add an entry for 172.16.10.101 proxmox.domain on /etc/hosts this entry is deleted. More digging lead me to add entries using WebGUI in Host name data base field. However, once I added the entry on this field, /etc/hosts file do changes but nslookup command returns ;; connection timed out; no servers could be reached.

Code: 
nslookup
> server 127.0.0.1
Default server: 127.0.0.1
Address: 127.0.0.1#53
> proxmox.domain
;; connection timed out; no servers could be reached

Maybe I made a mistake, but seems to me FreeNAS is not getting the /etc/hosts entry for proxmox.domain.
My /etc/hosts file after WebGUI editing

Code: 
172.16.10.101 proxmox.domain proxmox
127.0.0.1         freenas.domain freenas
::1                    freenas.domain freenas

Below the picture using the WebGUI for filling Host name data base field:
diuQ5io
 
I'm also having this issue as of today been working fine since deployment 6 months ago and then all of a sudden its just stopped.

I have the same as PedroDelgado
Code:
showmount -e 172.16.10.10
Output on Proxmox:
rpc mount export: RPC: Timed out

And also the mount via CLI works fine..

Any one got an idea why this has suddenly happened?
 
webstaff said:
I'm also having this issue as of today been working fine since deployment 6 months ago and then all of a sudden its just stopped.

I have the same as PedroDelgado
Code:
showmount -e 172.16.10.10
Output on Proxmox:
rpc mount export: RPC: Timed out

And also the mount via CLI works fine..

Any one got an idea why this has suddenly happened?
Click to expand...

Did you tried adding the entry for proxmox on /etc/hosts file? I'm still stuck there.
 
You can have this kind of issues when PVE does not have rw access to NFS share (with ro PVE will mount NFS share but will not bring it online)
Check out on FreeNas that "Maproot User" property (in advanced tab) of NFS share (on Freenas) is set to root (or another user that has rw rights on the folder you share on Freenas box)

Hope it helps
 
  • Like
Reactions: Omer SAVAS
PedroDelgado said:
Did you tried adding the entry for proxmox on /etc/hosts file? I'm still stuck there.
Click to expand...
We did try that but it didn't work.
What we ended up doing was just reordering the DNS list in the Freenas webgui so that the DC's are below the main DNS server for the network.
Seems like Freenas if it fails on the primary DNS doesn't use the others that you provide, I'll have to investigate further in the new year as we're basically shut down for Christmas now.

regards
Dave
 
Whatever said:
You can have this kind of issues when PVE does not have rw access to NFS share (with ro PVE will mount NFS share but will not bring it online)
Check out on FreeNas that "Maproot User" property (in advanced tab) of NFS share (on Freenas) is set to root (or another user that has rw rights on the folder you share on Freenas box)

Hope it helps
Click to expand...

Hi @Whatever before the issue I was able to read and write from NFS without problem. Also if I do manually mount the NFS share through cli I'm also able to read write on the NFS. Thanks for your reply.
 
webstaff said:
We did try that but it didn't work.
What we ended up doing was just reordering the DNS list in the Freenas webgui so that the DC's are below the main DNS server for the network.
Seems like Freenas if it fails on the primary DNS doesn't use the others that you provide, I'll have to investigate further in the new year as we're basically shut down for Christmas now.

regards
Dave
Click to expand...

Same as we, I mean Christmas. But I really hope to solve this out. Its critical for me cause I have several services on Proxmox reading and writing from FreeNAS. I really appreciate your help guys.
 
Well as I say we got ours working fine by just making sure the first DNS in the network > global config was active. As I say we just reordered them so the DC's which rely on freenas to boot we're not the first in our list.
In our case that just mean using our pfsense DNS (router) as primary as thats always up and not tied to proxmox or freenas on a cold start.
I won't be leaving it like that but it will do for two weeks till the new year.
Regards
Dave
 
  • Like
Reactions: PedroDelgado
Oh boy, I have some news about this issue and I guess is a FreeNAS-DNS problem as @vmanz and @webstaff suggested. By the way happy new year to all of you guys.

So I came in into the office with the problem in my head, turn on the Proxmox server and all the same, maybe I was hoping in my mind the problem got solved by itself, sometimes you can wish that kind of things on new year.

Being serious, I plug display and keyboard to Proxmox and run this:

Code: 
qm start 100

root@pve01:~# storage 'Backups' is not online

Then,

Code: 
mount 172.16.10.10:/mnt/RAID/Backups /mnt/pve/Backups

All good, now

Code: 
qm start 100

All good too. The VM (pfSense) is now running. So far nothing is new, I posted this at the beginning. Here comes the new stuff. Following the suggestion about DNS on FreeNAS I logged into WebGUI and added the PDC (also DNS server) in the field "Nameserver 1" as follows:

https://imgur.com/6cNTnYV (Direct link because last time embedded images didn't work for me)

After that I logged into FreeNAS using PuTTY and:

Code: 
nslookup pve01.localdomain.net

Output: 172.16.10.101, FreeNAS is seeing the DNS server. Now I try to add a new NFS share in Proxmox but using the WegGUI this time. Turns out now I can do it, the Proxmox is listing the NFS shares on FreeNAS. So I added a second one named ISOS. All good, now I do on Proxmox

Code: 
reboot

The VMs and containers aren't autostarting, the issue is back. So I run again:

Code: 
mount 172.16.10.10:/mnt/RAID/Backups /mnt/pve/Backups

But this time I'm getting a Timed out response from mount.nfs. Then I go to FreeNAS and remove the Nameserver 1 field by live it blank and run again

Code: 
mount 172.16.10.10:/mnt/RAID/Backups /mnt/pve/Backups

This time it succeed, so by elimination, the problem is on the DNS field. It's a loop, FreeNAS is trying to reach 172.16.10.1 (DNS) but Proxmox can't start the PDC because FreeNAS won't find a reply coming from the PDC. So as Dave (@webstaff) suggested, this could be "fixed" by pointing to a physical device acting as a secondary DNS server. I'm using a simple TP-LINK switch, I don't even find an IP address to enter to it WebGUI so this won't work for me unless I figure out a way to get a physical DNS server.

So YES, I can confirm the "problem" is on FreeNAS and the DNS setting. However, why is mandatory for FreeNAS to use a DNS server for this? And why an entry in the file /etc/hosts/ won't do the job? I will mark the post as SOLVED when I get a workaround to fix this, but I'm pretty clear what is going on here. Thank you guys for helping me out. @vmanz, if I had pay attention to your post about DNS stuff would save a lot of time to everybody. Also @dcsapak
 
Could be a reasonable idea to install bind9 on Proxmox's base operating system (Debian Jessie)? @dcsapak, @webstaff and @vmanz could this replace pfSense router DNS and avoid too this issue? The thing is the IPv4 address for Proxmox is 172.16.10.101 and Windows Server 2012 (PDC and DNS server) is 172.16.10.1. Will this cause any problem between these two DNS servers? FreeNAS IPv4 address is 172.16.10.10. Still don't understand why /etc/hosts on FreeNAS can't do the job.
 
putting the dns server virtualized on a freenas share can lead to a "chicken egg problem". freenas can be picky about it especially if you've all this active directory, samba crap enabled.

if you often get these
Code: 
storage <STORAGE> is not online
stuff check also on the pve host:
Code: 
nfsstat -r
and look at the 'retrans' value
this can also indicate that nfs server on freenas can't spawn enough nfs kernel threads (rpc calls included)
http://doc.freenas.org/11/services.html#nfs
Code: 
vfs.nfsd.maxthreads
vfs.nfsd.minthreads
 
I've just discovered that no response after showmount -e <server> command on client could be due to DoS Defend setting enabled on TP-Link switch and the reason is SYN sPort Less 1024 rule.
Hope this help to somebody.
 
You must log in or register to reply here.
Top Bottom