'zfs destroy' of single disk killed all VEs

[Riesling.Dry]

with disk-space having become short, I wanted to clean up my server a bit. I deleted a virtual disk, that was not actually being used (VE1100 uses subvol-1100-disk-2) and could not be deleted via GUI, so I did a:

zfs destroy -f rpool/subvol-1100-disk-1

as a result all other subvolumes sort of vanished, no VE could be started, cuz the files of the subvolumes/disks kinda disappeared, e.g.:

lxc-start -lDEBUG -o logme.log -F -n 222
cannot open directory //rpool/subvol-222-disk-1: No such file or directory
lxc-start: conf.c: run_buffer: 347 Script exited with status 2
lxc-start: start.c: lxc_init: 450 Failed to run lxc.hook.pre-start for container "222".
lxc-start: start.c: __lxc_start: 1313 Failed to initialize container "222".
lxc-start: tools/lxc_start.c: main: 360 The container failed to start.
lxc-start: tools/lxc_start.c: main: 364 Additional information can be obtained by setting the --logfile and --logpriority options.

I had to restore from backup to make the VEs work again. This is just a test server and I am glad this didn't happen on my production system.

Posting this JFYI, dunno if this is a bug.

 

[dietmar]

as a result all other subvolumes sort of vanished, no VE could be started

That is only possible if you use linked clones? In that case it is a bad idea to force removal of the base volume using the force flag. But this is not a zfs problem.

 

[fabian]

That is only possible if you use linked clones? In that case it is a bad idea to force removal of the base volume using the force flag. But this is not a zfs problem.

even then ZFS does not remove a dataset with children unless you pass -r (to destroy snapshots) or -R (to destroy snapshots and clones).. -f is just for force unmounting the dataset.

if you want to find out what happened, "zpool history rpool" will probably give you the answer..

 

[Riesling.Dry]

...linked clones?

No idea - it's a simple, standard proxmox VE 4.3-12 w. some containers

@fabian:

root@pve:~# zpool history rpool
...
2016-12-12.12:17:12 zfs destroy -r rpool/subvol-666-disk-2
2016-12-12.12:17:26 zfs destroy -r rpool/vm-555-disk-1
2016-12-12.12:17:34 zfs destroy -r rpool/vm-777-disk-1
2016-12-12.12:38:02 zfs destroy -f rpool/subvol-1100-disk-1
2016-12-12.12:42:29 zpool import -N rpool
2016-12-12.12:50:19 zpool import -N rpool
2016-12-12.12:55:53 zfs create -o acltype=posixacl -o xattr=sa -o refquota=83886080k rpool/subvol-222-disk-2
2016-12-12.12:56:08 zfs destroy -r rpool/subvol-222-disk-1
2016-12-12.13:01:21 zfs create -o acltype=posixacl -o xattr=sa -o refquota=260046848k rpool/subvol-100-disk-2
2016-12-12.13:01:32 zfs destroy -r rpool/subvol-100-disk-1
2016-12-12.13:08:33 zfs create -o acltype=posixacl -o xattr=sa -o refquota=104857600k rpool/subvol-333-disk-2
2016-12-12.13:08:41 zfs destroy -r rpool/subvol-333-disk-1
2016-12-12.13:17:23 zfs create -o acltype=posixacl -o xattr=sa -o refquota=8388608k rpool/subvol-444-disk-2
2016-12-12.13:17:41 zfs destroy -r rpool/subvol-444-disk-1
2016-12-12.13:39:49 zfs create -o acltype=posixacl -o xattr=sa -o refquota=335544320k rpool/subvol-1100-disk-1
2016-12-12.13:39:54 zfs destroy -r rpool/subvol-1100-disk-2
2016-12-12.14:00:41 zfs create -o acltype=posixacl -o xattr=sa -o refquota=8388608k rpool/subvol-555-disk-1
2016-12-12.14:01:53 zfs destroy -r rpool/subvol-555-disk-1

...where rpool/subvol-1100-disk-1 in line #1 is the one I intentionally killed via console (bc I couldn't via GUI, option ghosted out).
100, 222, 333, 444 and 1100 "vanished" - I didn't ask to kill them, neither via console, nor via GUI.
Initially I tried to remove 555, 666 and 777 via GUI which didn't work at first, not even after reboot. Destroy generated errors. After I restored 100, 222, 333, 444 and 1100 from backup, also 555, 666 and 777 went away, afair without again doing anything to remove them.

~R.

 

[fabian]

well the log shows that you destroyed the 555 666 and 777 volumes (probably with "remove disk"), then manually destroyed the 1100-disk-1 volume, and then created and destroyed the other disks (probably an attempted and failed restore from backup)

 

[Riesling.Dry]

I reckon the "destructions" of 100, 222, 333, 444 and 1100 happened automatically, when I restored them from backup which re-created the disks/subvolumes. I never ordered to destroy these. Restore happened only once for each VE and without errors.

 

[fabian]

ah sorry, I misread. yes, the destructions were destructions of the old volumes after the restore was done! maybe the datasets where unmounted after the manual deletion? I see two imports in the history, did you manually import the pool? did you reboot?

 

[Riesling.Dry]

except for the "zfs destroy -f rpool/subvol-1100-disk-1" I did nothing manually. Yes, rebooted twice.