Summary
There are a number of posts about being unable to destroy datasets that ZFS reports as busy even when lsof, fuser and cat /proc/*/mounts show no references.
I have traced this to nfsd kernel processes which are launched when the container starts but aren't cleaned up when the container is shutdown. This is likely only possible with privileged containers.
It is reproducible on Proxmox 4.x and 5.x.
How to Duplicate
Before starting confirm that you have no nfsd kernel threads running on the Proxmox hypervisor:
Launch a priviledged CentOS 6 container with an apparmor profile that allows NFS:
Enter the container:
Install and start NFS:
On the hypervisor, note that there are now nfsd kernel threads:
Shutdown or stop the container and confirm that the nfsd processes remain. If your container was on a ZFS filesystem. Now try to destroy it (pct destroy will also fail but the zfs command is shown for clarity):
Check for references using lsof, fuser or grep for the subvol in /proc/*/mounts. fuser -vm will indicate the kernel has a reference with no details. The other tools will return nothing.
Forcibly kill the nfsd processes:
Workaround 1
nfsd isn't needed for client mounts -- it's only required to serve NFS filesystems from an LXC container. It can be prevented from starting by changing the apparmor profile to only allow nfs (not nfsd):
Workaround 2
Tell the container to not launch nfsd threads by editing /etc/sysconfig/nfs and setting:
A better option...
Would be to have the nfsd processes cleaned up on container shutdown. I poked around the /proc filesystem for the nfsd processes but didn't see anything obvious (cgroups, file descriptors, etc.) that would conclusively associate them with the container. If you know the approximate start time of the container you can reconcile that against the start times of the nfsd processes but that's not going to be helpful if multiple containers are started at roughly the same time.
I'm hoping the Proxmox team will have some ideas here as this issue will be unavoidable if someone needs to run an NFS server inside a container.
There are a number of posts about being unable to destroy datasets that ZFS reports as busy even when lsof, fuser and cat /proc/*/mounts show no references.
I have traced this to nfsd kernel processes which are launched when the container starts but aren't cleaned up when the container is shutdown. This is likely only possible with privileged containers.
It is reproducible on Proxmox 4.x and 5.x.
How to Duplicate
Before starting confirm that you have no nfsd kernel threads running on the Proxmox hypervisor:
Code:
ps -face | grep nfsd
Code:
mount fstype=nfs*,
Code:
pct enter ID
Code:
yum install nfs-utils nfs-utils-lib
/etc/init.d/nfs start
On the hypervisor, note that there are now nfsd kernel threads:
Code:
root@HYPERVISOR:/# ps -face | grep nfs
root 37285 2 TS 19 12:00 ? 00:00:00 [nfsd]
root 37286 2 TS 19 12:00 ? 00:00:00 [nfsd]
root 37287 2 TS 19 12:00 ? 00:00:00 [nfsd]
root 37288 2 TS 19 12:00 ? 00:00:00 [nfsd]
root 37289 2 TS 19 12:00 ? 00:00:00 [nfsd]
root 37290 2 TS 19 12:00 ? 00:00:00 [nfsd]
root 37291 2 TS 19 12:00 ? 00:00:00 [nfsd]
root 37292 2 TS 19 12:00 ? 00:00:00 [nfsd]
Shutdown or stop the container and confirm that the nfsd processes remain. If your container was on a ZFS filesystem. Now try to destroy it (pct destroy will also fail but the zfs command is shown for clarity):
Code:
root@HYPERVISOR:/# zfs destroy -r -f POOL/subvol-XXXX-disk-1 cannot destroy 'POOL/subvol-XXXX-disk-1': dataset is busy
Check for references using lsof, fuser or grep for the subvol in /proc/*/mounts. fuser -vm will indicate the kernel has a reference with no details. The other tools will return nothing.
Forcibly kill the nfsd processes:
Code:
killall -9 nfsd
# Confirm that they're gone
ps -face | grep nfsd
# Now the destroy will work
root@HYPERVISOR:/# zfs destroy -r -f POOL/subvol-XXXX-disk-1
root@HYPERVISOR:/#
Workaround 1
nfsd isn't needed for client mounts -- it's only required to serve NFS filesystems from an LXC container. It can be prevented from starting by changing the apparmor profile to only allow nfs (not nfsd):
Code:
mount fstype=nfs,
Workaround 2
Tell the container to not launch nfsd threads by editing /etc/sysconfig/nfs and setting:
Code:
RPCNFSDCOUNT=0
A better option...
Would be to have the nfsd processes cleaned up on container shutdown. I poked around the /proc filesystem for the nfsd processes but didn't see anything obvious (cgroups, file descriptors, etc.) that would conclusively associate them with the container. If you know the approximate start time of the container you can reconcile that against the start times of the nfsd processes but that's not going to be helpful if multiple containers are started at roughly the same time.
I'm hoping the Proxmox team will have some ideas here as this issue will be unavoidable if someone needs to run an NFS server inside a container.
Last edited: