[SOLVED] NFS exports disappearing after 1-2 seconds

[vfrvgz]

Dear Proxmox community,

I'm at a loss here. I've been using PVE for quite some time now but after installing all pending OS updates from the community repository today, all of my NFS exports are no longer working.

After rebooting the host I noticed all NFS shares, shared via zfs sharenfs commands, are unavailable, even though the configuration was unchanged for months. One thing I found is that it seems the updates changed something in regards to hostname/IP handling. The configuration for one share was:

zfs set sharenfs="rw=@hostname3.redacted,rw=@hostname2.redacted,ro=hostname4.redacted,ro=@hostname1.redacted" pool1/data/redacted

Where hostname1 and hostname2 would resolve to the same IP address. This resulted in exportfs -r called by the nfs-server service exiting:

pve systemd[1]: Starting nfs-server.service - NFS server and services...
pve exportfs[16613]: exportfs: incompatible duplicated export entries:
pve exportfs[16613]: exportfs:         hostname1.redacted:/pool1/data/redacted (0x425) [IGNORED]
pve exportfs[16613]: exportfs:         hostname2.redacted:/pool1/data/redacted (0x424)
pve systemd[1]: Finished nfs-server.service - NFS server and services.

Since hostname2 is obsolete anyway, it was removed, nfs-server restarted and now the issue of exportfs -r exiting is gone and the shares are online and accessible after either running systemctl restart nfs-server or zfs share -a. They are also all visible when running showmount -e - the problem is: they disappear after a couple of seconds (in that case, showmount -e has no more entries), as soon as something is accessing the shares.

Would you please help me identifying the cause of this issue? I can't seem to find the correct logs or even the component that is responsible for removing the exports. I tried digging through journalctl, dmesg, etc. but couldn't find any meaningful entries. SUSE's KB states that NFS debug logging is pretty much unreadable to an average person so I didn't try that.

Any help is appreciated! Thanks!

// Update:
To make debugging easier, all nfs shares have now been removed using zfs set sharenfs=off to test with a single share. Behavior is the same, it will be removed from showmount -e after a couple of seconds without a client even trying to connect.

/etc/exports.d/zfs.exports file is also always empty. Only after restarting nfs-server it will be populated with the entries one would expect to see there:

# !!! DO NOT EDIT THIS FILE MANUALLY !!!

/zpool/data/redacted 10.10.10.10(sec=sys,rw,no_subtree_check,mountpoint)

So I believe this must be related to the current zfs version

zfs-2.2.2-pve1
zfs-kmod-2.2.2-pve1

Would downgrading to an older zfs version work for testing?

By the way: none of the zpools have been updated/upgraded since their creation.

 

[vfrvgz]

Downgrading zfs packages and reinstalling nfs-server did the trick, things are working again. Versions installed prior to today's updates were extracted from /var/log/dpkg.log to have the best chance of succeeding. Compatibility was verified using apt-cache policy first.

apt install libzfs4linux=2.1.13-pve1 zfs-initramfs=2.1.13-pve1 zfsutils-linux=2.1.13-pve1 zfs-zed=2.1.13-pve1
apt reinstall nfs-kernel-server

If someone from development wants some more info for investigation, let me know.

 

[george-f-s]

We've noticed a similar issue on our pve v 8.1.10 cluster, after performing step-upgrades from 6 ->7, 7->8. Issue caused by a manual update/upgrade of the NFS packages during PVE upgrade. Re-installing known good versions of nfs packages fixes it.