[SOLVED] ZFS Pool import fails - problem with cache file. How to fix?

[reckless]

Up-to-date Proxmox, latest versions as of today, already rebooted the server. I used to have no problem automatically importing all my ZFS pools on boot, but recently I have to do a manual zpool import -a every time after I reboot Proxmox. That command imports all the pools just fine. But it's quite annoying to do this manually every time and I want to fix it, so that it automatically boots and imports all the ZFS pools. This is the output from systemctl status zfs-import-cache:

root@proxmox:~# systemctl status zfs-import-cache
● zfs-import-cache.service - Import ZFS pools by cache file
   Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-12-18 10:02:43 CST; 11min ago
     Docs: man:zpool(8)
  Process: 1129 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=1/FAILURE)
Main PID: 1129 (code=exited, status=1/FAILURE)

Dec 18 10:02:43 proxmox zpool[1129]: cannot import 'mypool': no such pool or dataset
Dec 18 10:02:43 proxmox zpool[1129]:         Destroy and re-create the pool from
Dec 18 10:02:43 proxmox zpool[1129]:         a backup source.
Dec 18 10:02:43 proxmox zpool[1129]:         Destroy and re-create the pool from
Dec 18 10:02:43 proxmox zpool[1129]:         a backup source.
Dec 18 10:02:43 proxmox zpool[1129]:         Destroy and re-create the pool from
Dec 18 10:02:43 proxmox zpool[1129]:         a backup source.
Dec 18 10:02:43 proxmox systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 10:02:43 proxmox systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Dec 18 10:02:43 proxmox systemd[1]: Failed to start Import ZFS pools by cache file.

root@proxmox:~# systemctl status zfs-import-scan
● zfs-import-scan.service - Import ZFS pools by device scanning
   Loaded: loaded (/lib/systemd/system/zfs-import-scan.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:zpool(8)

Any idea how to fix this?

 

[Stoiko Ivanov]

Hmm - seems the cache-file is corrupt - we've had a few requests for this on the forum:
e.g. https://forum.proxmox.com/threads/u...c-containers-will-not-start.56102/post-268741

put shortly:

zpool set cachefile=/etc/zfs/zpool.cache rpool
update-initramfs -k all -u
reboot

(run the zpool command for each of your zfs pools)

I hope this helps!

 

[reckless]

I did exactly that but unfortunately, that didn't do anything. Rebooted and I got exactly the same issue. Like previously, zpool import -a works fine and imports everything, but I want it to mount at boot like it used to without manual intervention.

Here's the syslog from the start of the boot: https://pastebin.com/BphJqi4x

Any other ideas of what it might be and how to fix it?

 

[reckless]

Anybody have an idea on how to fix this?

 

[Stoiko Ivanov]

Dec 22 10:05:08 proxmox zpool[1135]: cannot import 'main': no such pool or dataset
Dec 22 10:05:08 proxmox zpool[1135]: cannot import 'nvr': no such pool or dataset
Dec 22 10:05:08 proxmox zpool[1135]: cannot import 'plex': no such pool or dataset
Dec 22 10:05:08 proxmox kernel: scsi 0:0:7:0: Direct-Access     ATA      ST10000NE0004-1Z EN01 PQ: 0 ANSI: 6
Dec 22 10:05:08 proxmox kernel: scsi 0:0:7:0: SATA: handle(0x0011), sas_addr(0x5003048017f19b47), phy(7), device_name(0x0000000000000000)
Dec 22 10:05:08 proxmox kernel: scsi 0:0:7:0: enclosure logical id (0x5003048017f19b7f), slot(7) 
Dec 22 10:05:08 proxmox kernel: scsi 0:0:7:0: enclosure level(0x0000), connector name(     )
Dec 22 10:05:08 proxmox kernel: scsi 0:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Dec 22 10:05:08 proxmox zpool[1135]:     Destroy and re-create the pool from
Dec 22 10:05:08 proxmox zpool[1135]:     a backup source.
Dec 22 10:05:08 proxmox zpool[1135]:     Destroy and re-create the pool from
Dec 22 10:05:08 proxmox zpool[1135]:     a backup source.
Dec 22 10:05:08 proxmox zpool[1135]:     Destroy and re-create the pool from
Dec 22 10:05:08 proxmox zpool[1135]:     a backup source.
Dec 22 10:05:08 proxmox systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Dec 22 10:05:08 proxmox systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Dec 22 10:05:08 proxmox systemd[1]: Failed to start Import ZFS pools by cache file.
Dec 22 10:05:08 proxmox systemd[1]: Reached target ZFS pool import target.
Dec 22 10:05:08 proxmox systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
Dec 22 10:05:08 proxmox systemd[1]: Starting Mount ZFS filesystems...

seems one of the disks (or the whole enclosure) becomes available after ZFS gets loaded and the system tries to import all pools:
Try setting a rootdelay of 10 on the commandline - see https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox
If you're booting with systemd-boot (ZFS on root and the system was setup with PVE 6.x with our installer you need to add the parameter to /etc/kernel/cmdline instead of /etc/default/grub - see https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot)

I hope this helps!

 

[reckless]

Seems one of the disks (or the whole enclosure) becomes available after ZFS gets loaded and the system tries to import all pools:
Try setting a rootdelay of 10 on the commandline - see https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox

Adding rootdelay to the command line ended up fixing the issue, my pools now automatically restore upon reboot.

Thank you Stoiko!

 

[maciej_kawala]

Hmm - seems the cache-file is corrupt - we've had a few requests for this on the forum:
e.g. https://forum.proxmox.com/threads/u...c-containers-will-not-start.56102/post-268741

put shortly:

zpool set cachefile=/etc/zfs/zpool.cache rpool
update-initramfs -k all -u
reboot

(run the zpool command for each of your zfs pools)

I hope this helps!

Hello,
What's the problem?

root@proxmox:~# systemctl status zfs-import-cache
● zfs-import-cache.service - Import ZFS pools by cache file
Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2022-10-03 16:33:56 CEST; 6min ago
Docs: man:zpool(8)
Process: 739 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN $ZPOOL_IMPORT_OPTS (code=exited, status=0/SUCCESS)
Main PID: 739 (code=exited, status=0/SUCCESS)
CPU: 52ms

Oct 03 16:33:52 proxmox systemd[1]: Starting Import ZFS pools by cache file...
Oct 03 16:33:56 proxmox systemd[1]: Finished Import ZFS pools by cache file.
root@proxmox:~# systemctl status zfs-import-scan
● zfs-import-scan.service - Import ZFS pools by device scanning
Loaded: loaded (/lib/systemd/system/zfs-import-scan.service; enabled; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Mon 2022-10-03 16:33:52 CEST; 6min ago
└─ ConditionFileNotEmpty=!/etc/zfs/zpool.cache was not met
Docs: man:zpool(8)

Oct 03 16:33:52 proxmox systemd[1]: Condition check resulted in Import ZFS pools by device scanning being skipped.

after command:

update-initramfs: Generating /boot/initrd.img-5.15.60-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.53-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-6-pve

proxmox reloop this command

 

[manolis11]

did you found a solution to this bug? i have just install latest version of ProxMox 8.1.3 from scratch..create a ZFS pool and then i have verify with this command systemctl status zfs-import-cache and i receive the issue you can see in the screenshot.

 

[macindy]

Also have this problem. Does anybody can give a hint? Thanks