ZFS cannot destroy XXX: dataset is busy -- Cause is nfsd

denos

Well-Known Member
Jul 27, 2015
83
41
58
Summary
There are a number of posts about being unable to destroy datasets that ZFS reports as busy even when lsof, fuser and cat /proc/*/mounts show no references.

I have traced this to nfsd kernel processes which are launched when the container starts but aren't cleaned up when the container is shutdown. This is likely only possible with privileged containers.

It is reproducible on Proxmox 4.x and 5.x.

How to Duplicate
Before starting confirm that you have no nfsd kernel threads running on the Proxmox hypervisor:
Code:
ps -face | grep nfsd
Launch a priviledged CentOS 6 container with an apparmor profile that allows NFS:
Code:
  mount fstype=nfs*,
Enter the container:
Code:
pct enter ID
Install and start NFS:
Code:
yum install nfs-utils nfs-utils-lib
/etc/init.d/nfs start

On the hypervisor, note that there are now nfsd kernel threads:
Code:
root@HYPERVISOR:/# ps -face | grep nfs
root     37285     2 TS   19 12:00 ?        00:00:00 [nfsd]
root     37286     2 TS   19 12:00 ?        00:00:00 [nfsd]
root     37287     2 TS   19 12:00 ?        00:00:00 [nfsd]
root     37288     2 TS   19 12:00 ?        00:00:00 [nfsd]
root     37289     2 TS   19 12:00 ?        00:00:00 [nfsd]
root     37290     2 TS   19 12:00 ?        00:00:00 [nfsd]
root     37291     2 TS   19 12:00 ?        00:00:00 [nfsd]
root     37292     2 TS   19 12:00 ?        00:00:00 [nfsd]

Shutdown or stop the container and confirm that the nfsd processes remain. If your container was on a ZFS filesystem. Now try to destroy it (pct destroy will also fail but the zfs command is shown for clarity):
Code:
root@HYPERVISOR:/# zfs destroy -r -f POOL/subvol-XXXX-disk-1 cannot destroy 'POOL/subvol-XXXX-disk-1': dataset is busy

Check for references using lsof, fuser or grep for the subvol in /proc/*/mounts. fuser -vm will indicate the kernel has a reference with no details. The other tools will return nothing.

Forcibly kill the nfsd processes:
Code:
killall -9 nfsd
# Confirm that they're gone
ps -face | grep nfsd
# Now the destroy will work
root@HYPERVISOR:/# zfs destroy -r -f POOL/subvol-XXXX-disk-1
root@HYPERVISOR:/#

Workaround 1
nfsd isn't needed for client mounts -- it's only required to serve NFS filesystems from an LXC container. It can be prevented from starting by changing the apparmor profile to only allow nfs (not nfsd):
Code:
  mount fstype=nfs,

Workaround 2
Tell the container to not launch nfsd threads by editing /etc/sysconfig/nfs and setting:
Code:
RPCNFSDCOUNT=0

A better option...
Would be to have the nfsd processes cleaned up on container shutdown. I poked around the /proc filesystem for the nfsd processes but didn't see anything obvious (cgroups, file descriptors, etc.) that would conclusively associate them with the container. If you know the approximate start time of the container you can reconcile that against the start times of the nfsd processes but that's not going to be helpful if multiple containers are started at roughly the same time.

I'm hoping the Proxmox team will have some ideas here as this issue will be unavoidable if someone needs to run an NFS server inside a container.
 
Last edited:
if you want NFS mounts in containers, mount them on the host and bind mount them into the container. anything else is not supported and will lead to problems such as you describe.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!