Failed to start zfs-import@nvme_pool.service - Import ZFS pool

Fazio8

New Member
Jan 9, 2023
19
1
3
Hello,
I just found out a timeout during the boot of my Proxmox 8.0.4.
SCR-20231025-pfbb Large.jpeg

The system both fine and VM/LXC are running properly but the boot is delayed for 90 seconds for each boot.
Checking the services I don't see any useful information:

Code:
[    7.443977] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[    7.460678] iwlwifi 0000:00:14.3: base HW address: 84:7b:57:61:de:15
[    7.482262] iwlwifi 0000:00:14.3 wlo1: renamed from wlan0
[    7.491370] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC233: line_outs=1 (0x21/0x0/0x0/0x0/0x0) type:hp
[    7.491373] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.491375] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.491376] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    7.491377] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    7.491378] snd_hda_codec_realtek hdaudioC0D0:      Mic=0x19
[    7.585873] input: HDA Intel PCH Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input6
[    7.586145] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input7
[    7.586196] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[    7.586228] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[    7.586267] input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[    7.586360] input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[    7.972756]  zd16: p1 p14 p15
[    7.990782]  zd64: p1
[    7.999215]  zd80: p1 p2 p3 p4 p5
[    8.002137]  zd96: p1 p14 p15
[    8.011872]  zd112: p1 p2 p3 p4 p5 p6 p7 p8
[    8.028437]  zd144: p1
[   96.509465] audit: type=1400 audit(1698253563.768:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=2629 comm="apparmor_parser"
[   96.510404] audit: type=1400 audit(1698253563.768:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=2627 comm="apparmor_parser"


...

root@NUC2:~# systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; preset: enabled)
     Active: active (exited) since Wed 2023-10-25 19:04:35 CEST; 1h 44min left
       Docs: man:zfs(8)
    Process: 2318 ExecStart=/sbin/zfs mount -a (code=exited, status=0/SUCCESS)
   Main PID: 2318 (code=exited, status=0/SUCCESS)
        CPU: 52ms
        
root@NUC2:~# systemctl status zfs-import@nvme_pool.service
× zfs-import@nvme_pool.service - Import ZFS pool nvme_pool
     Loaded: loaded (/lib/systemd/system/zfs-import@.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-10-25 19:04:35 CEST; 1h 51min left
       Docs: man:zpool(8)
    Process: 1194 ExecStart=/sbin/zpool import -N -d /dev/disk/by-id -o cachefile=none nvme_pool (code=exited, status=1/FAILURE)
   Main PID: 1194 (code=exited, status=1/FAILURE)
        CPU: 13ms

Oct 25 19:04:34 NUC2 systemd[1]: Starting zfs-import@nvme_pool.service - Import ZFS pool nvme_pool...
Oct 25 19:04:35 NUC2 zpool[1194]: cannot import 'nvme_pool': no such pool available
Oct 25 19:04:35 NUC2 systemd[1]: zfs-import@nvme_pool.service: Main process exited, code=exited, status=1/FAILURE
Oct 25 19:04:35 NUC2 systemd[1]: zfs-import@nvme_pool.service: Failed with result 'exit-code'.
Oct 25 19:04:35 NUC2 systemd[1]: Failed to start zfs-import@nvme_pool.service - Import ZFS pool nvme_pool.

As I said, the pool is OK and no issues are detected, I guess the pool is mounted later?
Code:
root@NUC2:~# zpool status -v
  pool: nvme_pool
 state: ONLINE
  scan: scrub repaired 0B in 00:04:19 with 0 errors on Sun Oct  8 00:28:20 2023
config:

    NAME                                            STATE     READ WRITE CKSUM
    nvme_pool                                       ONLINE       0     0     0
      nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:07:54 with 0 errors on Sun Oct  8 00:31:56 2023
config:

    NAME                                                STATE     READ WRITE CKSUM
    rpool                                               ONLINE       0     0     0
      ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part3  ONLINE       0     0     0

errors: No known data errors

root@NUC2:~# cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content iso,images,backup,snippets,vztmpl,rootdir
    shared 0

zfspool: local-zfs
    pool rpool/data
    content images,rootdir
    sparse 1

zfspool: nvme_pool
    pool nvme_pool
    content images,rootdir
    mountpoint /nvme_pool
    sparse 0

dir: wd
    path /mnt/pve/wd
    content rootdir,vztmpl,snippets,backup,iso,images
    is_mountpoint 1
    nodes NUC2

pbs: pbs
    datastore pbs
    server ####
    content backup
    fingerprint ####
    prune-backups keep-all=1
    username root@pam
    
root@NUC2:~# ls -l /dev/disk/by-id

total 0
lrwxrwxrwx 1 root root  9 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part3 -> ../../sda3
lrwxrwxrwx 1 root root 13 Oct 25  2023 nvme-eui.002538b921b6401c -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-eui.002538b921b6401c-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-eui.002538b921b6401c-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root 13 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z_1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z_1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z_1-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root  9 Oct 25  2023 usb-WD_My_Passport_0740_575848314136315339383739-0:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct 25  2023 usb-WD_My_Passport_0740_575848314136315339383739-0:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Oct 25  2023 wwn-0x50026b7282dbd8e3 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 25  2023 wwn-0x50026b7282dbd8e3-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 25  2023 wwn-0x50026b7282dbd8e3-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 25  2023 wwn-0x50026b7282dbd8e3-part3 -> ../../sda3

Any hint?
Thank you!
 
Found out that this was due to some old storage not visible on GUI but present on the node, I removed it by
Code:
systemctl disable mnt-pve-storage.mount
Removed "/etc/systemd/system/multi-user.target.wants/mnt-pve-storage.mount".
 
It seems that I fixed another error but not the main one:
Code:
Oct 25 18:17:18 NUC2 systemd[1]: Starting zfs-import-cache.service - Import ZFS pools by cache file...
░░ Subject: A start job for unit zfs-import-cache.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import-cache.service has begun execution.
░░
░░ The job identifier is 107.
Oct 25 18:17:18 NUC2 systemd[1]: zfs-import-scan.service - Import ZFS pools by device scanning was skipped because of an unmet condition check (ConditionFileNotEmpty=!/etc/zfs/zpool.cache).
░░ Subject: A start job for unit zfs-import-scan.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import-scan.service has finished successfully.
░░
░░ The job identifier is 103.
Oct 25 18:17:18 NUC2 systemd[1]: Starting zfs-import@nvme_pool.service - Import ZFS pool nvme_pool...
░░ Subject: A start job for unit zfs-import@nvme_pool.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import@nvme_pool.service has begun execution.
░░
░░ The job identifier is 105.
Oct 25 18:17:18 NUC2 systemd[1]: Reached target sound.target - Sound Card.
░░ Subject: A start job for unit sound.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit sound.target has finished successfully.
░░
░░ The job identifier is 188.
Oct 25 18:17:18 NUC2 zpool[1124]: cannot import 'nvme_pool': no such pool available
Oct 25 18:17:18 NUC2 systemd[1]: zfs-import@nvme_pool.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit zfs-import@nvme_pool.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 25 18:17:18 NUC2 systemd[1]: zfs-import@nvme_pool.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit zfs-import@nvme_pool.service has entered the 'failed' state with result 'exit-code'.
Oct 25 18:17:18 NUC2 systemd[1]: Failed to start zfs-import@nvme_pool.service - Import ZFS pool nvme_pool.
░░ Subject: A start job for unit zfs-import@nvme_pool.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import@nvme_pool.service has finished with a failure.
░░
░░ The job identifier is 105 and the job result is failed.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!