PVE 9.2.2 unable to remove CT on zfs with "dataset is busy" error

Madhatter

Renowned Member
Apr 8, 2012
43
2
73
I'm not sure if this is a new problem as I havent tried to remove a CT for some while.

But today I wanted to remove a few CTs which are running on a NVME ZFS and all result in

"TASK ERROR: zfs error: cannot destroy 'NVME-Pool/subvol-121-disk-0': dataset is busy"

Similar behavior on that NVME ZFS with a "Directory" Storage in Proxmox

I also have CTs on another internal ZFS NVMEs where it all works well.

zfs list snippet
NVME-Pool/subvol-121-disk-0 611M 7.40G 611M /NVME-Pool/subvol-121-disk-0

zfs version
zfs-2.4.2-pve1
zfs-kmod-2.4.2-pve1

pveversion -v
proxmox-ve: 9.2.0 (running kernel: 7.0.2-6-pve)
pve-manager: 9.2.2 (running version: 9.2.2/b9984c6d90a4bd80)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.2-6
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-7.0.2-5-pve-signed: 7.0.2-5
proxmox-kernel-7.0.2-2-pve-signed: 7.0.2-2
proxmox-kernel-6.17: 6.17.13-11
proxmox-kernel-6.17.13-11-pve-signed: 6.17.13-11
proxmox-kernel-6.14: 6.14.11-9
proxmox-kernel-6.14.11-9-pve-signed: 6.14.11-9
amd64-microcode: 3.20251202.1
ceph-fuse: 19.2.3-pve1
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.5
libpve-cluster-perl: 9.1.5
libpve-common-perl: 9.1.12
libpve-guest-common-perl: 6.0.3
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.5
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.5
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-1
proxmox-backup-client: 4.2.0-1
proxmox-backup-file-restore: 4.2.0-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.2
pve-cluster: 9.1.5
pve-container: 6.1.10
pve-docs: 9.2.1
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-3
pve-ha-manager: 5.2.4
pve-i18n: 3.7.4
pve-qemu-kvm: 11.0.0-3
pve-xtermjs: 6.0.0-1
qemu-server: 9.1.15
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.2-pve1

Migrating the disk results into the same error as does trying to remove manually with zfs destroy

I'm out of ideas what to do.
Anyone with an Idea what the issue could be?
 
What is the output of:
grep NVME-Pool/subvol-121-disk-0 /proc/*/mounts
fuser -vm /NVME-Pool/subvol-121-disk-0
zfs holds NVME-Pool/subvol-121-disk-0
 
Last edited:
grep NVME-Pool/subvol-121-disk-0 /proc/*/mounts
/proc/987/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/988/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/989/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/990/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/991/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/992/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/993/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/994/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/995/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/996/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/997/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/998/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/999/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/99/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/self/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0
/proc/thread-self/mounts:NVME-Pool/subvol-121-disk-0 /NVME-Pool/subvol-121-disk-0 zfs rw,noatime,xattr,posixacl,casesensitive 0 0



# fuser -am /dev/zvol/NVME-Pool/subvol-121-disk-0
Specified filename /dev/zvol/NVME-Pool/subvol-121-disk-0 does not exist.
/dev/zvol/NVME-Pool/subvol-121-disk-0:

# zfs holds NVME-Pool/subvol-121-disk-0
'NVME-Pool/subvol-121-disk-0' is not a snapshot
NAME TAG TIMESTAMP
no datasets available


Thanks for picking that up ..

I did the obligatory reboot also without any success...
 
Another interesting observation. I went to a CT, moved the disk to another storage, it moved fine but wasnt able to delete the old source.
then I reassigned the old disk to any other CT (in my case I build a dummy CT) and then from theree I was able to delete the Disk on the NVME Pool.

very weird ..
 
# fuser -am /dev/zvol/NVME-Pool/subvol-121-disk-0
Specified filename /dev/zvol/NVME-Pool/subvol-121-disk-0 does not exist.
/dev/zvol/NVME-Pool/subvol-121-disk-0:
Sorry this should be fuser -vam /NVME-Pool/subvol-121-disk-0 for LXC.

Also make sure to run these when the LXC is not running.

Basically you want to find out which process(es) are still holding the disk of the LXC when it is not running, preventing the removal or migration.
 
# fuser -vam /NVME-Pool/subvol-102-disk-0
USER PID ACCESS COMMAND
/NVME-Pool/subvol-102-disk-0:
root kernel mount /NVME-Pool

LXC was not running.

I was able to whack the CT from proxmox .. so only the storage is left but even manually I cant get rid of it.

zfs destroy -f NVME-Pool/subvol-102-disk-0
cannot destroy 'NVME-Pool/subvol-102-disk-0': dataset is busy

the other CTs I was able to get off the ZFS with my workaround ..