zfs-import@asgard.service fails to start at boot?

[PeterAndersson]

Hi I just installed proxmox(7.1-2), setup a zfs pool.
But on every boot the service zfs-import@asgard.service fails to start, but if I look in the /asgard directory everything seems fine?

I am fairly new to Proxmox, Linux and ZFS so I hope I have gathered all the logs that can be of use, if not just tell me what more could be of interest.

root@proxmox:~# systemctl --failed
  UNIT                      LOAD   ACTIVE SUB    DESCRIPTION
● zfs-import@asgard.service loaded failed failed Import ZFS pool asgard

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.

root@proxmox:~# systemctl status zfs-import@asgard.service
● zfs-import@asgard.service - Import ZFS pool asgard
     Loaded: loaded (/lib/systemd/system/zfs-import@.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2021-12-13 20:26:53 CET; 9min ago
       Docs: man:zpool(8)
    Process: 1112 ExecStart=/sbin/zpool import -N -d /dev/disk/by-id -o cachefile=none asgard (code=exited, status=1/FAILURE)
   Main PID: 1112 (code=exited, status=1/FAILURE)
        CPU: 11ms

root@proxmox:~# journalctl -u zfs-import@asgard.service
-- Journal begins at Fri 2021-12-10 19:04:34 CET, ends at Mon 2021-12-13 20:27:21 CET. --
Dec 12 18:29:29 proxmox systemd[1]: Starting Import ZFS pool asgard...
Dec 12 18:29:29 proxmox zpool[1112]: cannot import 'asgard': no such pool available
Dec 12 18:29:29 proxmox systemd[1]: zfs-import@asgard.service: Main process exited, code=exited, status=1/FAILURE
Dec 12 18:29:29 proxmox systemd[1]: zfs-import@asgard.service: Failed with result 'exit-code'.
Dec 12 18:29:29 proxmox systemd[1]: Failed to start Import ZFS pool asgard.

root@proxmox:~# zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
asgard  18.2T  1.59G  18.2T        -         -     0%     0%  1.00x    ONLINE  -

root@proxmox:~# zpool status
  pool: asgard
 state: ONLINE
config:

        NAME                                  STATE     READ WRITE CKSUM
        asgard                                ONLINE       0     0     0
          raidz2-0                            ONLINE       0     0     0
            ata-TOSHIBA_HDWG440_9190A00KFZ0G  ONLINE       0     0     0
            ata-TOSHIBA_HDWG440_9190A00EFZ0G  ONLINE       0     0     0
            ata-TOSHIBA_HDWG440_91U0A06JFZ0G  ONLINE       0     0     0
            ata-TOSHIBA_HDWG440_X180A08DFZ0G  ONLINE       0     0     0
            ata-TOSHIBA_HDWG440_9170A007FZ0G  ONLINE       0     0     0

errors: No known data errors

Starting the service after boot also fails.

So what is wrong here?

Also a minor question, when I created a VM I selected the pool as storage, but no file has been created in the pool for the VM, where are VM Disks stored when a zfs pool is selected?

Edit: the pool was created using the web ui.

 

[dcsapak]

can you post the log of a boot (e.g. with 'journalctl -b' for the current one)?

 

[PeterAndersson]

It's fairly large so I attached it as a file.

 

[dcsapak]

it seeems that the disks just take a bit longer to initialize before the import service would find it (~2 seconds) but it imports it later regardless
so it's just a cosmetic problem

 

[PeterAndersson]

Is there any way to fix this issue, as I'm planning on adding monitoring and this would cause it to get a error..
I tried to add ZFS_INITRD_PRE_MOUNTROOT_SLEEP='15' and ZFS_INITRD_POST_MODPROBE_SLEEP='15' into /etc/default/zfs then run

update-initramfs -u

. but that didn't help.

 

[dcsapak]

ok after looking at the logs again, i assume the reason for this behaviour is that there are two services trying to import the pool: the zfs-import@asgard.service and the zfs-import-cache.service
the latter one only starts when the cachefile exists and is not empty (/etc/zfs/zpool.cache) which should not happen normally, except if you manually imported the pool once

so if this is your only pool (and it's not your root pool), it should be possible to remove that cache file and the normal service should be able to import the pool OR you could disable
the zfs-import@ service and let the pool import via cachefile

in any case, make sure that you know how to reverse those changes if something goes wrong

 

[PeterAndersson]

Hi again, tried to rename the /etc/zfs/zpool.cache to /etc/zfs/zpool.cache.backup but sadly after a reboot the failed service persists.

But disabeling the service worked and the /asgard directory is still there, so that seems to have done the trick..

I'm a bit confused what i did to cause this error but thanks for helping me resolve it!

 

[dcsapak]

Hi again, tried to rename the /etc/zfs/zpool.cache to /etc/zfs/zpool.cache.backup but sadly after a reboot the failed service persists.

mhmm interesting... if you want you can post the journal from that boot and i can check if i see more..

but thanks for helping me resolve it!

no problem

 

[PeterAndersson]

I renamed the zpool.cache file and enabled the service again, and did a restart.
Im guessing this is the interesting part?

Dec 17 10:12:56 proxmox kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
Dec 17 10:12:56 proxmox systemd[1]: Created slice system-lvm2\x2dpvscan.slice.
Dec 17 10:12:56 proxmox systemd[1]: Starting LVM event activation on device 8:35...
Dec 17 10:12:56 proxmox systemd[1]: Finished Helper to synchronize boot up for ifupdown.
Dec 17 10:12:56 proxmox systemd[1]: Finished Wait for udev To Complete Device Initialization.
Dec 17 10:12:56 proxmox systemd[1]: Condition check resulted in Import ZFS pools by cache file being skipped.
Dec 17 10:12:56 proxmox systemd[1]: Starting Import ZFS pools by device scanning...
Dec 17 10:12:56 proxmox systemd[1]: Starting Import ZFS pool asgard...
Dec 17 10:12:56 proxmox lvm[1111]:   pvscan[1111] PV /dev/sdc3 online, VG pve is complete.
Dec 17 10:12:56 proxmox lvm[1111]:   pvscan[1111] VG pve skip autoactivation.
Dec 17 10:12:56 proxmox systemd[1]: Finished LVM event activation on device 8:35.
Dec 17 10:12:58 proxmox zpool[1112]: cannot import 'asgard': pool already exists
Dec 17 10:12:58 proxmox systemd[1]: zfs-import-scan.service: Main process exited, code=exited, status=1/FAILURE
Dec 17 10:12:58 proxmox systemd[1]: zfs-import-scan.service: Failed with result 'exit-code'.
Dec 17 10:12:58 proxmox systemd[1]: Failed to start Import ZFS pools by device scanning.
Dec 17 10:12:58 proxmox kernel:  zd0: p1 p2 p3
Dec 17 10:12:58 proxmox systemd[1]: Finished Import ZFS pool asgard.
Dec 17 10:12:58 proxmox systemd[1]: Reached target ZFS pool import target.
Dec 17 10:12:58 proxmox systemd[1]: Starting Mount ZFS filesystems...
Dec 17 10:12:58 proxmox systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
Dec 17 10:12:58 proxmox zvol_wait[1657]: Testing 1 zvol links
Dec 17 10:12:58 proxmox zvol_wait[1657]: All zvol links are now present.
Dec 17 10:12:58 proxmox systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
Dec 17 10:12:58 proxmox systemd[1]: Reached target ZFS volumes are ready.
Dec 17 10:12:58 proxmox systemd[1]: Finished Mount ZFS filesystems.

 

[dcsapak]

are you sure you enable the correct service (zfs-import@asgard.service) and not 'zfs-import-scan.service' ? at least the error message is different ('already exists' vs 'no such pool') ... but otherwise the log does not help more really

 

[PeterAndersson]

I'm 99% that I enabled/disabled the correct service as the command systemctl --failed reflects the change correctly after a reboot.
But disabling the service at least removes/hides the problem.

The big question is if the zfs pool fails for some reason, will some other service get a failed state?

 

[dcsapak]

i am not sure, this depends on how it gets imported in the first place (what i wanted to find out), but the logs are sadly of limited help...
if it gets imported by a service, then most probably yes..