[SOLVED] rpool import issue - recovery after power failure

[ilia987]

This morning we had some power outage issue on our server room,
all nodes recovered except one.

This node is part of (one of three) ceph cluster , (pools are set to replication 3) so the data is safe and the cluster is stable, (excluding ceph warning)

Any idea how i can fix it?

 

[Stoiko Ivanov]

* Does the node have ZFS on root? (did it get installed via PVE ISO)
* what's the output of `zpool import` in the initramfs shell
* what's the output of `lsblk`, `blkid` in the shell (not sure if both any of the are available) - else `ls -la /dev/` should also help
* check `dmesg` for potential disk-problems

I hope this helps!

 

[ilia987]

*yes from iso

zpool import:

lbsk:

ls -la /dev:
starts with

blkid:


the end of dmseg:

 

[Stoiko Ivanov]

* does `zpool import -R /rpool -N rpool` on the initramfs shell work?
* if not does `zpool import -R /rpool -N 344988999858258096` work (the numeric identifier of the pool from the screenshot)?

if yes my guess would be that scanning the disks (and all potential iSCSI devices based on the first screenshot you posted) might simply take to long and the drives are not present yet when init tries to import the pool
in that case try adding a rootdelay parameter to the kernel commandline - see:
https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Grub_boot_ZFS_problem

I hope this helps!

 

[ilia987]

* does `zpool import -R /rpool -N rpool` on the initramfs shell work?
* if not does `zpool import -R /rpool -N 344988999858258096` work (the numeric identifier of the pool from the screenshot)?

if yes my guess would be that scanning the disks (and all potential iSCSI devices based on the first screenshot you posted) might simply take to long and the drives are not present yet when init tries to import the pool
in that case try adding a rootdelay parameter to the kernel commandline - see:
https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Grub_boot_ZFS_problem

I hope this helps!

how i can scann the boot drives?

 

[Stoiko Ivanov]

ok - please try to add the root delay as described in the wiki-page.

* the import was successful - the failing mount is proably due to the non-existing /root/rpool directory
see https://github.com/openzfs/zfs/issues/6045
* the second import was unsuccessful, since the first one already imported the pool

I hope this helps!

 

[ilia987]

ok - please try to add the root delay as described in the wiki-page.

* the import was successful - the failing mount is proably due to the non-existing /root/rpool directory
see https://github.com/openzfs/zfs/issues/6045
* the second import was unsuccessful, since the first one already imported the pool

I hope this helps!

grub files does not exist,
i updated zfs pre sleep, but how i "commit" force config update for the change?
because update-initramfs does not exist

 

[Stoiko Ivanov]

yes in the initramfs neither the grub/systemd-boot config, nor update-initramfs exist:
reboot the server and hit 'e' when you're at the boot-loader - there you can set the parameter for one boot - if this fixes the issue you can set it permanently in the config once the system is booted:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot

 

[ilia987]

for some reason i thought it worked. and tried to do another reboot,
now after running zpool import i receive no pools available

 

[ilia987]

How i can get to a menu to update zfs sleep args?
"ZFS_INITRD_PRE_MOUNTROOT_SLEEP"

 

[Stoiko Ivanov]

just add the 'rootdelay=10' to the kernel commandline - that should hopefully get the system to boot

regarding the no pools available to import - that (usually) says that the pool already has been imported

 

[ilia987]

just add the 'rootdelay=10' to the kernel commandline - that should hopefully get the system to boot

regarding the no pools available to import - that (usually) says that the pool already has been imported

give the error:
cannot import 'rpool': no such pool available

 

[ilia987]

anyone got another idea? i am not willing to reinstall the server.

i tired to recover the boot via installation cd but it failed

 

[Stoiko Ivanov]

you could try to remove the duplicate root=ZFS=rpool/ROOT/pve-1 in the kernel command-line (not sure if that causes the problem though)
(additionally you could remove the quite parameter - maybe the kernel would print some helpful message pointing to the problem)

if the system boots with rootdelay=15 and it says 'cannot import 'rpool': no such pool available' -> what's the output of `zpool import`?

 

[ilia987]

i found a workaround:

after i receive the error of rpool not found. i should wait around 1-2 minutes at least and then run
zpool import -N rpool -f
after it worked i changed the zfs sleep time and now it more stable