NFS Shares: storage is not online (500)

Discussion in 'Proxmox VE: Installation and configuration' started by idurey, Sep 10, 2012.

  1. idurey

    idurey New Member

    Joined:
    Jun 14, 2012
    Messages:
    4
    Likes Received:
    0
    I'm encountering a strange issue today after upgrading the system to the latest version:

    I can mount an NFS share from the command line but when I attempt to browse the contents from the ProxMox UI I get "storage 'Backups' is not online (500)".

    If I delete the share and re-add it using the UI the share does not appear in /mnt/pve/ as expected.

    Here is /etc/pve/storage.cfg
    Code:
    nfs: Backups        
            path /mnt/pve/Backups
            server FIRS01
            export /NFS
            options vers=3
            content backup
            maxfiles 10
    
    Here is mounting the NFS share from command line and viewing contents of a test file:
    Code:
    mount -t nfs firs01:/NFS/ /mnt/pve/Backups/
    root@pmhost3:/mnt/pve# cd /mnt/pve/Backups/
    root@pmhost3:/mnt/pve/Backups# ls
    dump  test.txt
    root@pmhost3:/mnt/pve/Backups# cat test.txt 
    Successfully connected to NFS share.
    Is there a log somewhere I can use to figure out why this isn't working?

    Thanks,

    Ian
     
  2. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,447
    Likes Received:
    304
    any hint in /var/log/syslog? What is the output of

    # pvesm list Backups
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. idurey

    idurey New Member

    Joined:
    Jun 14, 2012
    Messages:
    4
    Likes Received:
    0

    Well this is very odd. As a workaround overnight I removed the Backups storage and created a new NFS mount on each node at /mnt/pve/Backup2 and created a new Storage Dir using the UI called Backup2. I changed my backup jobs to back up to Backup2 and tested them, everything was working fine.

    Today I read your message and sat down to do some troubleshooting. Once again I added a new NFS storage called Backups and was ready to comb through the syslog when suddenly everything works.

    I made no other changes to the system and I did not reboot. Is it possible that changing the location which the backup jobs point to resolved the issue?

    In any case, everything is fine now. Thank you for your time and your suggestions, I didn't think to look in syslog.
     
  4. PeterS

    PeterS New Member

    Joined:
    Oct 15, 2011
    Messages:
    5
    Likes Received:
    0
    Hi,

    I also get this error on my cluster after upgrading it from Proxmox 1.x to 2.0
    The only workaround to re-enable the NFS shares (at least for some days) is to restart the nfs server on my NAS (Synology).

    How ever, after the time the error comes back, the nfs shares seams to be offline again and I have to restart the nfs server once more.

    Any suggestions?
    Regards Peter
     
  5. idurey

    idurey New Member

    Joined:
    Jun 14, 2012
    Messages:
    4
    Likes Received:
    0
    The best solution I found was to stop using ProxMox.
     
  6. PeterS

    PeterS New Member

    Joined:
    Oct 15, 2011
    Messages:
    5
    Likes Received:
    0
    Hmm... This is the only one issue with Proxmox until now, so I would really like to continue using it :)
    Why are you so unhappy with Proxmox? Any other problems?

    cu Peter
     
  7. nik-weter

    nik-weter New Member

    Joined:
    Sep 24, 2012
    Messages:
    12
    Likes Received:
    0
    Had the same problem. Disabled ipv6 and nfs4. everything works.
     
  8. spirit

    spirit Well-Known Member

    Joined:
    Apr 2, 2010
    Messages:
    3,316
    Likes Received:
    132
    Hi proxmox check if the nfs storage is online with command

    "/usr/bin/rpcinfo -p ipofnfsserver"

    with a timeout of 2sec.


    So maybe is the timeout too short for your storage under load ?
    Can you try to change it in file (at the end) :

    /usr/share/perl5/PVE/Storage/NFSPlugin.pm

    and restart

    /etc/init.d/pvestatd restart
    /etc/init.d/pvedaemon restart
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. RobFantini

    RobFantini Active Member
    Proxmox Subscriber

    Joined:
    May 24, 2012
    Messages:
    1,490
    Likes Received:
    21
    Did you do that on NFS or the Proxmox system?
     
  10. nik-weter

    nik-weter New Member

    Joined:
    Sep 24, 2012
    Messages:
    12
    Likes Received:
    0
    On storage - just.
    On proxmox i do not remember.
    Now there is no access to the hypervisor.
     
  11. jbennet

    jbennet New Member

    Joined:
    Jul 19, 2013
    Messages:
    18
    Likes Received:
    0
    Sorry to excave an old post, but with fresh install of proxmox 3.1 I can reproduce this problem everytime between some Dell workstations, and a QNAP TS-469L NAS or any other linux servers/VM on the local network.

    I tested a dell optiplex 745 with broadcom network chipset, and a dell T3600 with a TG3 network chipset.

    Code:
    # rpcinfo -p 172.16.66.66
    rpcinfo: can't contact pormapper: RPC: Remote system error - Connection timed out
    
    So proxmox complains about storage being offline, I won't blame the proxmox storage daemon on this : rpcinfo -p should not timeout.

    The problem is that :
    - Other linux boxes on the network do not suffer from this problem.
    - The proxmox server v3.0 upgraded to v3.1 do not suffer from this problem.
    - Without -p option, "rpcinfo 172.16.66.66" finally works after a very long time.
    - When running "rpcinfo -p serverip" Wireshark show a SYN but no ACK in reply (hence the timeout)
    - When running "telnet serverip 111" Wireshark show a SYN and ACK in reply (explain me this)

    On top of that, this mount command line works :
    Code:
    # mount -t nfs -o vers=3,udp serverip:/Backup /mnt/pve/Backup
    But this mount command doesn't works :
    Code:
    # mount -t nfs -o vers=3 serverip:/Backup /mnt/pve/Backup
    This problem feels very wrong and bizarre. The switch is an HP 2530-24G, and there is no trace of firewall installed.

    The pveversion is :
    Code:
    root@secours:~# pveversion -v
    proxmox-ve-2.6.32: 3.1-109 (running kernel: 2.6.32-23-pve)
    pve-manager: 3.1-3 (running version: 3.1-3/dc0e9b0e)
    pve-kernel-2.6.32-23-pve: 2.6.32-109
    lvm2: 2.02.98-pve4
    clvm: 2.02.98-pve4
    corosync-pve: 1.4.5-1
    openais-pve: 1.1.4-3
    libqb0: 0.11.1-2
    redhat-cluster-pve: 3.2.0-2
    resource-agents-pve: 3.9.2-4
    fence-agents-pve: 4.0.0-1
    pve-cluster: 3.0-7
    qemu-server: 3.1-1
    pve-firmware: 1.0-23
    libpve-common-perl: 3.0-6
    libpve-access-control: 3.0-6
    libpve-storage-perl: 3.0-10
    pve-libspice-server1: 0.12.4-1
    vncterm: 1.1-4
    vzctl: 4.0-1pve3
    vzprocps: 2.0.11-2
    vzquota: 3.1-2
    pve-qemu-kvm: 1.4-17
    ksm-control-daemon: 1.1-1
    glusterfs-client: 3.4.0-2
    
    The local rpc services show no lockd daemon running (nlockmgr) ; couldn't find any clue about how to start the rpc.lockd kernel process ; or why doesn't it runs.
    Code:
    root@secours:~# rpcinfo -p
       program vers proto   port  service
        100000    4   tcp    111  portmapper
        100000    3   tcp    111  portmapper
        100000    2   tcp    111  portmapper
        100000    4   udp    111  portmapper
        100000    3   udp    111  portmapper
        100000    2   udp    111  portmapper
        100024    1   udp  51434  status
        100024    1   tcp  55586  status
    
    Since the problem arises with any other server (not only the NAS), I guess the NAS is not to blame, and rebooting the NAS has no effect on this, I tried removing the IPV6 (following a tutorial on internet) with no success.

    Does somebody have any clue about this problem ? It drives me nuts.
    How did you managed to disable the NFSv4 and IPV6 on the proxmox host ?
     
  12. jbennet

    jbennet New Member

    Joined:
    Jul 19, 2013
    Messages:
    18
    Likes Received:
    0
  13. Bruno Emanuel

    Bruno Emanuel New Member

    Joined:
    Apr 29, 2016
    Messages:
    7
    Likes Received:
    0
    Onliest to register. Recently i had problem with the access on my nfs storage.
    I'm using Proxmox 4.2
    The message that i receive was "NFS_Storage is not online"

    I read on https://forums.freenas.org/index.php?threads/nfs-mount-times-out.7270/ :
    "During the mount process, NFS apparently does a reverse DNS lookup to see if the address is valid. On a local network there is no way to do that unless you add the IP address of the client to the hosts file on the server. The hosts file even mentions that in the documentation in the beginning of the file. Once we did that the mount connected very quickly."

    So I register my servers on /etc/hosts of FreeNAS. And the message changed to "NFS version ..."
    Then I changed on FreeNAS to permit use NFS v4.

    But when the FreeNAS reboot the /etc/hosts is rewrited, so to make the register of hosts permanent need to register the hosts on FreeNAS.
    "System > Network", on field "Host name data base"

    And it works.
     
    #13 Bruno Emanuel, Jul 21, 2016
    Last edited: Jul 21, 2016
  14. brucexx

    brucexx Member

    Joined:
    Mar 19, 2015
    Messages:
    126
    Likes Received:
    4
    Thanks Bruno - your post lead me to the issue I was having. I was getting the same Storage is not online message. I checked everything on Proxmox side and it was ok. When I saw your post I went to the network setting on the FreeNAS and noticed that the DNS 1 was not valid anymore (I replaced it a week ago and the ip address changed), the DNS2 and 3 had a valid public DNS ips 8.8.8.8 and 4.2.2.2 but I guess it was not enough, go figure...

    Once I adjusted the primary DNS setting even with no host name database entry for Proxmox it started to work immediately.

    I am on PVE 4.2 and FreeNAS 9.10

    Thank you
     
    Bruno Emanuel likes this.
  15. icalredhat

    icalredhat New Member

    Joined:
    May 20, 2016
    Messages:
    19
    Likes Received:
    0
    check file hosts on your proxmox and freenas ..
     
  16. Cambell

    Cambell New Member

    Joined:
    Feb 8, 2017
    Messages:
    1
    Likes Received:
    0
    I just experienced this issue on PVE 4.2. It seems that my session had expired. I checked via ssh on the server and all was fine. I logged out of the UI, logged back in and it was fine.
     
  17. lao-tseu

    lao-tseu New Member

    Joined:
    Dec 9, 2011
    Messages:
    2
    Likes Received:
    2
    pretty old thread but I did experiment the same issue today,
    i was able to connect in the shell with a classic mount, but got always an error on the Proxmox side,

    I got it finally working by editing /etc/pve/storage.cfg
    i noticed that proxmox was using nfs version 3, but my nfs server at ovh wanted nfs 4
    so by just changing the line
    options vers=3
    to
    options vers=4

    then everything worked fine in proxmox interface

    maybe it can help someone
     
    Xabi and SamTzu like this.
  18. SamTzu

    SamTzu Member

    Joined:
    Mar 27, 2009
    Messages:
    356
    Likes Received:
    6
  19. bferrell

    bferrell New Member

    Joined:
    Nov 16, 2018
    Messages:
    19
    Likes Received:
    0
    OMG, thank you, thank you for this post! I accidentally added my phile to SAN DHCP and it shattered my cluster, I was getting "storage offline" messages everywhere. {heart attack ends}
     
  20. uno

    uno New Member
    Proxmox Subscriber

    Joined:
    Jun 5, 2018
    Messages:
    7
    Likes Received:
    0
    My Proxmox cluster had been in operation for 4 months when I encountered this error. It happened when I was mass-migrating disks from iSCSI to NFS.

    Cluster Details
    Proxmox version: 5.3
    Storage: FreeNAS 11.2 serving iSCSI and NFS

    Problem
    I was switching the backing store of several VMs from iSCSI to NFS (QCoW2) storage when I encountered the error TASK ERROR: storage 'nfs-infra-ssd-1' is not online. In one of the PVE nodes, the NFS storage showed up as inactive.
    Code:
    # pvesm status
    ...
    nfs-infra-ssd-1                   nfs   inactive               0               0               0    0.00%
    
    Solution
    I didn't manage to figure out what was the issue but remounting the share made it active again.
    Code:
    umount /mnt/pve/nfs-infra-ssd-1
    mount -t nfs freenas.local:/mnt/pool-ssd-1/ds-infra-ssd-1 nfs-infra-ssd-1
    
    There's no reason why this should work, but it did. Hope it helps someone out there who encountered this issue.

    16 May 2019 update: The problem resurfaced on 1 out of 4 PVE nodes after an upgrade from 5.3 to 5.4 followed by a reboot. The above method didn't work but this instantly solved it:
    Code:
    pvesm scan nfs ip.of.nfs.storage
    
     
    #20 uno, Apr 29, 2019
    Last edited: May 16, 2019
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice