Zpools not mounting after reboot

[VMTester]

Hello all together,

I have following problem:
After stopping and restarting the system my zfs-pools stop mounting.
Before made an update and in between I plugged in a Microsemi Adaptec HBA 1100 (https://storage.microsemi.com/de-de/products/hba1100/)

The only possibility to mount at the moment is via
zfs mount -a -O

I already tried:
zpool set cachefile=/etc/zfs/zpool.cache maschinencache1
zpool set cachefile=/etc/zfs/zpool.cache maschinenpool1
from this thread: Zpools not mounting on boot - without success

Spoiler: zfs mount -a

cannot mount '/maschinencache1': directory is not empty
cannot mount '/maschinenpool1': directory is not empty

Spoiler: pveversion

pve-manager/6.1-5/9bf06119 (running kernel: 5.3.13-1-pve)

Spoiler: zfs get mountpoint

NAME PROPERTY VALUE SOURCE
maschinencache1 mountpoint /maschinencache1 default
maschinencache1/zwischenspeicher mountpoint /maschinencache1/zwischenspeicher default
maschinenpool1 mountpoint /maschinenpool1 default
maschinenpool1/vmdata mountpoint /maschinenpool1/vmdata default
rpool mountpoint /rpool default
rpool/ROOT mountpoint /rpool/ROOT default
rpool/ROOT/pve-1 mountpoint / local
rpool/data mountpoint /rpool/data default
rpool/swap mountpoint - -

Spoiler: zfs get mounted

[19:06:22][root@heimserver:~]$ zfs get mounted
NAME PROPERTY VALUE SOURCE
maschinencache1 mounted no -
maschinencache1/zwischenspeicher mounted no -
maschinenpool1 mounted no -
maschinenpool1/vmdata mounted no -
rpool mounted yes -
rpool/ROOT mounted yes -
rpool/ROOT/pve-1 mounted yes -
rpool/data mounted yes -
rpool/swap mounted - -

Spoiler: zpool status -v

pool: maschinencache1
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0 days 00:10:30 with 0 errors on Sun Jan 12 00:34:31 2020
config:

NAME STATE READ WRITE CKSUM
maschinencache1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5002538e40477e93 ONLINE 0 0 0
wwn-0x5002538e40477ea4 ONLINE 0 0 0

errors: No known data errors

pool: maschinenpool1
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0 days 00:16:43 with 0 errors on Sun Jan 12 00:40:45 2020
config:

NAME STATE READ WRITE CKSUM
maschinenpool1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5002538e404e960f ONLINE 0 0 0
wwn-0x5002538e404e9599 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0 days 00:01:03 with 0 errors on Sun Jan 12 00:25:06 2020
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sde2 ONLINE 0 0 0

errors: No known data errors

Spoiler: zfs setup-parameter from documentation

zpool create -f -o ashift=12 maschinenpool1 mirror /dev/sdc /dev/sdd
zpool create -f -o ashift=12 maschinencache1 mirror /dev/sde /dev/sdf
zfs create maschinenpool1/vmdata
zfs create maschinencache1/zwischenspeicher

Many VMs (disk-images), all ISOs and Templates are at "vmdata"
and
some disk-images are at "zwischenspeicher"

I am glad about any help to fix this issue.
Thanks in advance!

 

[wolfgang]

Hi,
I guess the problem is as the zfs mount command says your dir is not empty.
This is the reason why it does not mount.
You have to export the pool and clean up.
Then it will work again.

 

[VMTester]

Hi,

I guess the problem is as the zfs mount command says your dir is not empty.
This is the reason why it does not mount.

I read the messages and the problem is:
I never use to copy or move data in these directories. Neither maschinencache1 nor maschinenpool1.
BTW: I don' see any data inside the directories

You have to export the pool and clean up.
Then it will work again.

Do you have a recommendation for a howto, manual or tutorial doing this without loosing any data?

And very important:
How can I prevent this happening again?

 

[wolfgang]

How can I prevent this happening again?

This normally happened if the pool is not mounted and Proxmox VE tries to access the mount.