zpool won't auto mount anymore after kernel update

L

Larrikin

New Member
Apr 15, 2024
11
1
3
Hi - I am running Proxmox Community Edition. I updated my packages today.

I am running Linux pve 6.5.13-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-5 (2024-04-05T11:03Z) x86_64 GNU/Linux

I run ZFS on all my drives. The NVME drives mount just fine.

However, I have 8 x 16TB spinning SATA drives that used to mount on boot, but no longer. Every time I reboot, I have to import my pool (called tank) for it to be recognised by Proxmox. I could create a service to automount it, but thought I'd raise it here first. I saw in another thread that someone else is having the same issue.

@Stoiko Ivanov I've included you as you stated in that previous thread to start a new thread (that was someone else who also is having the same problem).
 
gfngfn256 said:
Show your cat /etc/pve/storage.cfg output
Click to expand...

Code: 
root@pve:~# cat /etc/pve/storage.cfg


dir: local


    path /var/lib/vz


    content backup,vztmpl,iso





zfspool: local-zfs


    pool rpool/data


    content rootdir,images


    sparse 1





zfspool: vms


    pool vms


    content images,rootdir


    mountpoint /vms


    nodes pve





pbs: pbs


    datastore NAS


    server 10.1.1.91


    content backup


    fingerprint <finger print inserted here>


    prune-backups keep-all=1


    username root@pam
 
Last edited:
In future try & post your output in CODE Tags, its easier to read.

So am I correct in assuming your pool (called tank) has not been configured as a storage type.
 
I tried to use markup but clearly missed that in the GUI there is insert code. Sorry!

I don't mount it as a storage type. I use a VM with NFS connecting to the zpool tank.
 
This is the thing, it shows its setup correctly.

Code: 
root@pve:~# zpool get cachefile
NAME   PROPERTY   VALUE      SOURCE
rpool  cachefile  -          default
tank   cachefile  -          default
vms    cachefile  none       local
root@pve:~#
 
Please share your journal since boot (until all services are up) - `journalctl -b` maybe this will tell us what's going on.
The issue in the other thread was related to the new 6.8 kernel - does your pool get imported with any particular kernel you've installed, and is not imported with another? (else it's probably another issue)

How did you create the pool initially?
 
Ok looks good. What does systemctl status zfs-import.service zfs-import-cache.service zfs-import-scan.service show
 
I got the relevant section from the command you asked me to type.

Code: 
Apr 15 13:46:54 pve zed[61134]: eid=16 class=config_sync pool='tank'
Apr 15 13:46:54 pve systemd[1]: zfs-import-cache.service: Deactivated successfully.
Apr 15 13:46:54 pve systemd[1]: Stopped zfs-import-cache.service - Import ZFS pools by cache file.
Apr 15 13:46:54 pve systemd[1]: Stopping zfs-import-cache.service - Import ZFS pools by cache file...
Apr 15 13:46:54 pve systemd[1]: Starting zfs-import-cache.service - Import ZFS pools by cache file...
Apr 15 13:46:54 pve zpool[61136]: no pools available to import
Apr 15 13:46:54 pve systemd[1]: Finished zfs-import-cache.service - Import ZFS pools by cache file.
root@pve:~#

The pool was imported into Proxmox when I successfully installed it and it upgraded it (initially created by Ubuntu). Until today when I applied a kernel upgrade, everything worked fine when rebooting.
 
gfngfn256 said:
Ok looks good. What does systemctl status zfs-import.service zfs-import-cache.service zfs-import-scan.service show
Click to expand...
Code: 
root@pve:~# systemctl status zfs-import.service zfs-import-cache.service zfs-import-scan.service
○ zfs-import.service
     Loaded: masked (Reason: Unit zfs-import.service is masked.)
     Active: inactive (dead)

● zfs-import-cache.service - Import ZFS pools by cache file
     Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; preset: enabled)
     Active: active (exited) since Mon 2024-04-15 13:46:54 AEST; 4h 28min ago
       Docs: man:zpool(8)
    Process: 61136 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN $ZPOOL_IMPORT_OPTS (code=exited, status=0/SUCCESS)
   Main PID: 61136 (code=exited, status=0/SUCCESS)
        CPU: 10ms

Apr 15 13:46:54 pve systemd[1]: Starting zfs-import-cache.service - Import ZFS pools by cache file...
Apr 15 13:46:54 pve zpool[61136]: no pools available to import
Apr 15 13:46:54 pve systemd[1]: Finished zfs-import-cache.service - Import ZFS pools by cache file.

○ zfs-import-scan.service - Import ZFS pools by device scanning
     Loaded: loaded (/lib/systemd/system/zfs-import-scan.service; enabled; preset: disabled)
     Active: inactive (dead)
  Condition: start condition failed at Mon 2024-04-15 12:40:22 AEST; 5h 34min ago
             └─ ConditionFileNotEmpty=!/etc/zfs/zpool.cache was not met
       Docs: man:zpool(8)

Apr 15 12:40:22 pve systemd[1]: zfs-import-scan.service - Import ZFS pools by device scanning was skipped because of an unmet condition check (ConditionFileNotEmpty=!/etc/zfs/zpool.cache).
root@pve:~#
 
Pools defined in storage.cfg have their (own) mechanism for activate_storage causing a zpool import <pool>

You however have a pool not defined in storage.cfg, but rely on the zfs-import-cache.service

What I would try in your case is making a separate cachefile for that pool, so maybe:
Code: 
zpool set cachefile=/etc/zfs/zpool2.cache tank

# I believe this will create the file /etc/zfs/zpool2.cache automatically
Note: I don't currently do this, but IIRC did something (successfully) similar in the past.
 
gfngfn256 said:
Pools defined in storage.cfg have their (own) mechanism for activate_storage causing a zpool import <pool>

You however have a pool not defined in storage.cfg, but rely on the zfs-import-cache.service

What I would try in your case is making a separate cachefile for that pool, so maybe:
Code: 
zpool set cachefile=/etc/zfs/zpool2.cache tank

# I believe this will create the file /etc/zfs/zpool2.cache automatically
Note: I don't currently do this, but IIRC did something (successfully) similar in the past.
Click to expand...
Thanks.

That has created this:

Code: 
root@pve:~# zpool get cachefile
NAME   PROPERTY   VALUE                  SOURCE
rpool  cachefile  -                      default
tank   cachefile  /etc/zfs/zpool2.cache  local
vms    cachefile  none                   local
root@pve:~#
 
gfngfn256 said:
Pools defined in storage.cfg have their (own) mechanism for activate_storage causing a zpool import <pool>

You however have a pool not defined in storage.cfg, but rely on the zfs-import-cache.service

What I would try in your case is making a separate cachefile for that pool, so maybe:
Code: 
zpool set cachefile=/etc/zfs/zpool2.cache tank

# I believe this will create the file /etc/zfs/zpool2.cache automatically
Note: I don't currently do this, but IIRC did something (successfully) similar in the past.
Click to expand...

I rebooted (twice) and its made things more unstable. Definitely doesn't auto mount.

Did the normal import command again, but couldn't see tank on my PVE, but I could see it within my VM.

Rebooted PVE again, and this time I couldn't see it on my PVE or my VM.

I did the import again of tank, and this time it showed up on PVE.

It still wasn't showing up in my VM. So I rebooted just the VM and now it shows.

This is getting my heart racing :)
 
Another option I guess is to create a /etc/systemd/system/zfs-tank-delayed-import.service

Code: 
[Unit]
Description=Delayed ZFS pool import for tank
After=local-fs.target
Wants=local-fs.target

[Service]
Type=oneshot
ExecStart=/bin/sleep 10
ExecStart=/sbin/zpool import tank
ExecStart=/sbin/zfs mount tank
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
 
Larrikin said:
Another option I guess is to create a /etc/systemd/system/zfs-tank-delayed-import.service
Click to expand...
Well you originally mentioned this in your first post. But obviously this is just a workaround.
We still don't know the root cause for this.

I would first reset the changes we've made by:
Code: 
zpool set cachefile=/etc/zfs/zpool.cache tank
You never know, maybe it will somehow reset itself & finally start working. (You can always hope!).

You may have to open a bug report.
 
gfngfn256 said:
Well you originally mentioned this in your first post. But obviously this is just a workaround.
We still don't know the root cause for this.

I would first reset the changes we've made by:
Code: 
zpool set cachefile=/etc/zfs/zpool.cache tank
You never know, maybe it will somehow reset itself & finally start working. (You can always hope!).

You may have to open a bug report.
Click to expand...

Done in terms of reverting.

In terms of opening a bug report, I may wait to see what @Stoiko Ivanov says as he did ask me some questions so I am hoping he might come back into this thread and provide some advice. More than happy to file a bug report if required.

Thanks for all your efforts with me on this. I might just avoid rebooting pve for a while :). It doesn't need rebooting much, only when a kernel update occurs which is what started this whole thing. That happened today.
 
  • Like
Reactions: gfngfn256
How is this `tank` pool used exactly in your setup (as it's not a storage for vm-images)? How is it used inside a VM?

else pools in PVE get imported by various mechanisms:
* per default: if it is listed in /etc/zfs/zpool.cache (this happens if you manually import a pool on the cli, but can be racy, because it gets removed there when the pool is exported
* for pools that are used as VM-disk storage and listed in storage.cfg - by pvestatd (which calls activate_storage)
* when a guest-disk is on that pool and the guest gets started (also activate_storage)
* when you create the pool through the GUI a systemd-unit is enabled `zfs-import@<poolname>.service` which runs the import command on boot (maybe this would be the least invasive way for you to get your pool imported)

Currently I don't see where there is a bug here?

I hope this helps!
 
  • Like
Reactions: gfngfn256
As a workaround, try adding this to /etc/rc.local and reboot:

zpool import -a -f -d /dev/disk/by-id
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back