Missing Container Disks on ZFS Pool

[kurtk]

Hello All,

I'm experiencing an issue where 2 of my container disks are missing from the GUI/storage manager and not listed or mounted anywhere. The disks in question are "subvol-101-disk-1" and "subvol-102-disk-1". The directories and filesystems are created in the zfspool, and the containers do actually boot, but I can't manage them at all:

root@pve:/zfs_pool# ls -lah
total 23K
drwxr-xr-x  6 root root    6 Aug 14 17:51 .
drwxr-xr-x 24 root root 4.0K Sep 17 16:18 ..
drwxr-xr-x 21 root root   21 Jul 19 14:21 subvol-101-disk-1
drwxr-xr-x 21 root root   21 Sep 17 15:27 subvol-102-disk-1
drwxrwxrwx  5 1000 1000    5 Aug 17 00:10 subvol-103-disk-1
drwxr-xr-x  4 root root    4 Aug 14 18:20 subvol-105-disk-1

root@pve:/zfs_pool# zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
zfs_pool                     723G  4.56T  6.80G  /zfs_pool
zfs_pool/subvol-103-disk-1   566G   958G   566G  /zfs_pool/subvol-103-disk-1
zfs_pool/subvol-105-disk-1  1.17G  18.8G  1.17G  /zfs_pool/subvol-105-disk-1
zfs_pool/vm-200-disk-1      85.0G  4.62T  25.9G  -
zfs_pool/vm-201-disk-1      42.5G  4.58T  18.6G  -
zfs_pool/vm-202-disk-1      21.3G  4.58T  4.16G  -

root@pve:/zfs_pool# zfs set mountpoint=/zfs_pool/subvol-102-disk-1 zfs_pool/subvol-102-disk-1
cannot open 'zfs_pool/subvol-102-disk-1': dataset does not exist

root@pve:/zfs_pool# pvesm list zfs1
zfs1:subvol-103-disk-1 subvol 1638272325387 103
zfs1:subvol-105-disk-1 subvol 21474836480 105
zfs1:subvol-109-disk-0 subvol 21474836480 109
zfs1:vm-200-disk-1       raw 85899345920 200
zfs1:vm-201-disk-1       raw 42949672960 201
zfs1:vm-202-disk-1       raw 21474836480 202

root@pve:/zfs_pool# cat /etc/pve/lxc/102.conf | grep rootfs
rootfs: zfs1:subvol-102-disk-1,size=16G

Any ideas on what happened here, and how I can fix it?
Thanks!

 

[Alwin]

drwxr-xr-x 21 root root 21 Jul 19 14:21 subvol-101-disk-1 drwxr-xr-x 21 root root 21 Sep 17 15:27 subvol-102-disk-1

Is there any content in those subfolders? Is anything in the logs (syslog/journal)?

 

[LnxBil]

Also check your zpool history when the datasets were deleted.

 

[kurtk]

Is there any content in those subfolders? Is anything in the logs (syslog/journal)?

yep, the whole file structure is there, the containers will boot.

root@pve:/zfs_pool/subvol-102-disk-1# pwd
/zfs_pool/subvol-102-disk-1
root@pve:/zfs_pool/subvol-102-disk-1# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Also check your zpool history when the datasets were deleted.

checked zpool history, no entries for either of the missing disks, so looks like the disks/datasets were never properly created.
can i just do zfs create -o xattr=sa -o refquota=16000000k zfs_pool/subvol-102-disk-1 and then mount the directory to it?

 

[Alwin]

Then it looks more like they are not being mounted, for whatever reason. Is anything in the logs (syslog/journal)?

 

[kurtk]

Then it looks more like they are not being mounted, for whatever reason. Is anything in the logs (syslog/journal)?

Not that I could find. However, I was able to resolve this by creating the dataset, mounting it, and then using "Move Volume" from within PVE to move it from zfs to local and back again (otherwise pve was reporting the bootdisk size as the entire 4.58T, as was df -h inside the container):

zfs create -o xattr=sa -o refquota=16000000k zfs_pool/subvol-102-disk-1
zfs set mountpoint=/zfs_pool/subvol-102-disk-1 zfs_pool/subvol-102-disk-1