Automount not working, but manual mount from shell works fine. -Sequel-

[PedroDelgado]

I will resume my post because this is a derivated question from a unsolved thread and I will ask for pardon. The unsolved thread is: https://forum.proxmox.com/threads/nfs-shares-storage-is-not-online-500-why.38778/

Situation:
Around 5 or 6 weeks ago I set up a Proxmox 5.1-35 (172.16.10.101) and FreeNAS 11.0-U4 (172.16.10.10) server. Everything was fine until no Virtual Machines (VMs) or (CTs) completed "Start at boot". None of them were running by the time Proxmox was already working. So I enter into the server room, sit on chair in front of Proxmox server's display and type:

> qm list
>qm start 100 (my pfSense firewall)

The output:

pve01 pvestatd[1277]: storage 'Backups' is not online

Here in this forum, and in the same above thread: https://forum.proxmox.com/threads/nfs-shares-storage-is-not-online-500-why.38778/ we came into the idea it was a FreeNAS problem, better said, FreeNAS wasn't seeing a DNS server to allow Proxmox mount NFS shares. The curious thing is FreeNAS allows Proxmox doing the mount using the command mount 172.16.10.10:/mnt/RAID/Backups /mnt/pve/Backups with no DNS request involved. Then I decided, before doing anything on my servers, to post the same issue on FreeNAS forum.

And the link is: https://forums.freenas.org/index.ph...visible-because-a-dns-issue-on-freenas.60508/

According to a very kind user named @Nick2253 the problem was automount options, which sounds correct because FreeNAS lets Proxmox do the mount through the command line. Specifically the way Proxmox is configured to do NFS automount at startup. I did more digging and realized /etc/fstab file has nothing related to NFS mounts.




At this thread https://forum.proxmox.com/threads/cifs-mount-in-etc-fstab-will-not-auto-mount-upon-restart.36841/ user @hec suggests to declare mounts in /etc/systemd/system/ so I get there and found no reference neither related to NFS shares (was hoping the NFS configuration files was there).

VERY IMPORTANT to consider:

I do can mount NFS shares using Proxmox WebGUI even when the DNS server is off and can do it through command line too.



However I can't do umount 172.16.10.10:/mnt/RAID/ISOS /mnt/pve/ISOS but this could be unrelated to the automount problem.

 

[stef1777]

Hi!

May be this will solve your problem.

You need for each hardware server running PVE to modify this:

In "/etc/apparmor.d/lxc/lxc-default-cgns" add these lines:

mount fstype=nfs*,
mount fstype=rpc_pipefs,

In "/etc/apparmor.d/lxc/lxc-default-with-mounting "add this line:

mount fstype=nfs*,

And reload apparmor:

service apparmor reload

Optional:

In LXC host running with Debian wheezy, you also may need to add this in /etc/default/rcS:

ASYNCMOUNTNFS=no

Not needed for Jessie or Stretch.

 

[PedroDelgado]

Tried that and then did a reboot. Turns out VMs and CTs won't autostart because the problem persist. I'm going to do a reinstall of Proxmox to see if I can rid of this issue. This is stressing me a lot. By the way thanks for your help

 

[silbro]

Has anyone solved this problem? I can recreate the exact same situation. I have a dns server as a VM. I added this dns to freenas as nameserver 1 and added my firewall as nameserver 2. Whenever I stop the dns server, it sais NFS share is offline. You cant start the dns server afterwards because it resides on the nfs share. luckily I had the dns server on a spare server so I could start it to get access to the nfs share again. This didn't happen on the proxmox 3.x.

 

[silbro]

Ok so I just found a workaround for the problem. If you remove all nameservers on freenas, save and then wait a minute you can start all vms again. As soon as you add the nameserver again and it's not available you get the same error. At least this way you can start your vms again.

 

[uno]

This is an old thread but I'll revive it for posterity.

TLDR
I encountered the same problem on my production cluster. The workaround is: run DNS servers outside of the Proxmox cluster or on iSCSI (and lose snapshots) or on local-lvm (and have limited HA). This way, at least the DNS servers can come up and subsequently for FreeNAS to permit NFSv3 mounts and then the rest of the NFS-backed Proxmox VMs can come up.

The Long Answer
My setup looks like this:

1. Dev: 4 Proxmox servers + FreeNAS
2. Production: 10 Proxmox server + TrueNAS
3. 90% of VMs are on NFS. The rest use iSCSI.

I enabled NFSv4 in FreeNAS (FreeNAS henceforth will refer to both FreeNAS and TrueNAS) and Proxmox mounts the shares at NFSv4.1. The daemon that checks if the storage is online is pvestatd. At the lower level, the command it uses to check is /sbin/showmount --no-headers --exports ip.of.nfs.server. showmount performs NFSv3-like checks on the storage.

Here's the problem. In FreeNAS (and hence FreeBSD), NFSv3 is hardcoded to require DNS. The iXSystems engineer confirmed this. NFSv4 has no such requirement. In my case,

1. mount -t nfsv4 ip.of.nfs.server:/share worked
2. mount -t nfs -o vers=3 ip.of.nfs.server:/share failed. The Linux NFS client will attempt mounting the highest version first so vers=3 is necessary.

After 1 whole day of debugging, I discovered by accident that when DNS was up, pvesm would report that the storage was online. DNS goes down and the storage "goes down". It doesn't actually go down per se, but because showmount uses NFSv3-style checks, it reports that the storage failed. NFSv4 mounts continue to work.

The story doesn't end here because I'm not satisfied that this can't be resolved. In Linux, one can configure nfsd to ignore not to perform name lookups. Search for Disable name lookup requests from NFS server at this page. I haven't seen such an option in FreeNAS but I'll have a chat with the engineer.

Presently, I have the DNS servers running on local-lvm.

Also, I feel that Proxmox needs to update pvestatd to with a NFSv4 way to check if the storage is online. Its iSCSI checks are also generating tons of spurious errors in FreeNAS' logs. Read iSCSI Reconnecting every 10 seconds to FreeNAS solution for the complete picture. I think both OSes - Proxmox and FreeNAS - can be improved.

 

[TheMrWorf]

I found a solution which is the "least" hacky:

1. Add your NFS server to the proxmox hosts file (FQDN and just hostname as usual)
2. Edit storage.cfg so you point to the FQDN

Step 2 is critical, without it, this wont work. Once this is done, you can confirm functionality by running

showmount -e my.nfs.server.local

It should now work, even though your DNS is down.