Failed to start zfs-import@nvme_pool.service - Import ZFS pool

Fazio8

New Member
Jan 9, 2023
24
1
3
Hello,
I just found out a timeout during the boot of my Proxmox 8.0.4.
SCR-20231025-pfbb Large.jpeg

The system both fine and VM/LXC are running properly but the boot is delayed for 90 seconds for each boot.
Checking the services I don't see any useful information:

Code:
[    7.443977] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[    7.460678] iwlwifi 0000:00:14.3: base HW address: 84:7b:57:61:de:15
[    7.482262] iwlwifi 0000:00:14.3 wlo1: renamed from wlan0
[    7.491370] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC233: line_outs=1 (0x21/0x0/0x0/0x0/0x0) type:hp
[    7.491373] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.491375] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.491376] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    7.491377] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    7.491378] snd_hda_codec_realtek hdaudioC0D0:      Mic=0x19
[    7.585873] input: HDA Intel PCH Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input6
[    7.586145] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input7
[    7.586196] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[    7.586228] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[    7.586267] input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[    7.586360] input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[    7.972756]  zd16: p1 p14 p15
[    7.990782]  zd64: p1
[    7.999215]  zd80: p1 p2 p3 p4 p5
[    8.002137]  zd96: p1 p14 p15
[    8.011872]  zd112: p1 p2 p3 p4 p5 p6 p7 p8
[    8.028437]  zd144: p1
[   96.509465] audit: type=1400 audit(1698253563.768:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=2629 comm="apparmor_parser"
[   96.510404] audit: type=1400 audit(1698253563.768:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=2627 comm="apparmor_parser"


...

root@NUC2:~# systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; preset: enabled)
     Active: active (exited) since Wed 2023-10-25 19:04:35 CEST; 1h 44min left
       Docs: man:zfs(8)
    Process: 2318 ExecStart=/sbin/zfs mount -a (code=exited, status=0/SUCCESS)
   Main PID: 2318 (code=exited, status=0/SUCCESS)
        CPU: 52ms
        
root@NUC2:~# systemctl status zfs-import@nvme_pool.service
× zfs-import@nvme_pool.service - Import ZFS pool nvme_pool
     Loaded: loaded (/lib/systemd/system/zfs-import@.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-10-25 19:04:35 CEST; 1h 51min left
       Docs: man:zpool(8)
    Process: 1194 ExecStart=/sbin/zpool import -N -d /dev/disk/by-id -o cachefile=none nvme_pool (code=exited, status=1/FAILURE)
   Main PID: 1194 (code=exited, status=1/FAILURE)
        CPU: 13ms

Oct 25 19:04:34 NUC2 systemd[1]: Starting zfs-import@nvme_pool.service - Import ZFS pool nvme_pool...
Oct 25 19:04:35 NUC2 zpool[1194]: cannot import 'nvme_pool': no such pool available
Oct 25 19:04:35 NUC2 systemd[1]: zfs-import@nvme_pool.service: Main process exited, code=exited, status=1/FAILURE
Oct 25 19:04:35 NUC2 systemd[1]: zfs-import@nvme_pool.service: Failed with result 'exit-code'.
Oct 25 19:04:35 NUC2 systemd[1]: Failed to start zfs-import@nvme_pool.service - Import ZFS pool nvme_pool.

As I said, the pool is OK and no issues are detected, I guess the pool is mounted later?
Code:
root@NUC2:~# zpool status -v
  pool: nvme_pool
 state: ONLINE
  scan: scrub repaired 0B in 00:04:19 with 0 errors on Sun Oct  8 00:28:20 2023
config:

    NAME                                            STATE     READ WRITE CKSUM
    nvme_pool                                       ONLINE       0     0     0
      nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:07:54 with 0 errors on Sun Oct  8 00:31:56 2023
config:

    NAME                                                STATE     READ WRITE CKSUM
    rpool                                               ONLINE       0     0     0
      ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part3  ONLINE       0     0     0

errors: No known data errors

root@NUC2:~# cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content iso,images,backup,snippets,vztmpl,rootdir
    shared 0

zfspool: local-zfs
    pool rpool/data
    content images,rootdir
    sparse 1

zfspool: nvme_pool
    pool nvme_pool
    content images,rootdir
    mountpoint /nvme_pool
    sparse 0

dir: wd
    path /mnt/pve/wd
    content rootdir,vztmpl,snippets,backup,iso,images
    is_mountpoint 1
    nodes NUC2

pbs: pbs
    datastore pbs
    server ####
    content backup
    fingerprint ####
    prune-backups keep-all=1
    username root@pam
    
root@NUC2:~# ls -l /dev/disk/by-id

total 0
lrwxrwxrwx 1 root root  9 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 25  2023 ata-KINGSTON_SA400S37240G_50026B7282DBD8E3-part3 -> ../../sda3
lrwxrwxrwx 1 root root 13 Oct 25  2023 nvme-eui.002538b921b6401c -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-eui.002538b921b6401c-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-eui.002538b921b6401c-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root 13 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z_1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z_1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z_1-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Oct 25  2023 nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T934934Z-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root  9 Oct 25  2023 usb-WD_My_Passport_0740_575848314136315339383739-0:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct 25  2023 usb-WD_My_Passport_0740_575848314136315339383739-0:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Oct 25  2023 wwn-0x50026b7282dbd8e3 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 25  2023 wwn-0x50026b7282dbd8e3-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 25  2023 wwn-0x50026b7282dbd8e3-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 25  2023 wwn-0x50026b7282dbd8e3-part3 -> ../../sda3

Any hint?
Thank you!
 
Found out that this was due to some old storage not visible on GUI but present on the node, I removed it by
Code:
systemctl disable mnt-pve-storage.mount
Removed "/etc/systemd/system/multi-user.target.wants/mnt-pve-storage.mount".
 
It seems that I fixed another error but not the main one:
Code:
Oct 25 18:17:18 NUC2 systemd[1]: Starting zfs-import-cache.service - Import ZFS pools by cache file...
░░ Subject: A start job for unit zfs-import-cache.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import-cache.service has begun execution.
░░
░░ The job identifier is 107.
Oct 25 18:17:18 NUC2 systemd[1]: zfs-import-scan.service - Import ZFS pools by device scanning was skipped because of an unmet condition check (ConditionFileNotEmpty=!/etc/zfs/zpool.cache).
░░ Subject: A start job for unit zfs-import-scan.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import-scan.service has finished successfully.
░░
░░ The job identifier is 103.
Oct 25 18:17:18 NUC2 systemd[1]: Starting zfs-import@nvme_pool.service - Import ZFS pool nvme_pool...
░░ Subject: A start job for unit zfs-import@nvme_pool.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import@nvme_pool.service has begun execution.
░░
░░ The job identifier is 105.
Oct 25 18:17:18 NUC2 systemd[1]: Reached target sound.target - Sound Card.
░░ Subject: A start job for unit sound.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit sound.target has finished successfully.
░░
░░ The job identifier is 188.
Oct 25 18:17:18 NUC2 zpool[1124]: cannot import 'nvme_pool': no such pool available
Oct 25 18:17:18 NUC2 systemd[1]: zfs-import@nvme_pool.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit zfs-import@nvme_pool.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 25 18:17:18 NUC2 systemd[1]: zfs-import@nvme_pool.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit zfs-import@nvme_pool.service has entered the 'failed' state with result 'exit-code'.
Oct 25 18:17:18 NUC2 systemd[1]: Failed to start zfs-import@nvme_pool.service - Import ZFS pool nvme_pool.
░░ Subject: A start job for unit zfs-import@nvme_pool.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit zfs-import@nvme_pool.service has finished with a failure.
░░
░░ The job identifier is 105 and the job result is failed.