[SOLVED] ZFS pool not mounted at startup

[DynFi User]

We have a large PBS install with two pools :

1. system pool with 2x NVMe (2x 256Go)
2. backup pool with 13x HDD (13x 14To)

The system pool is always well mounted.
But unfortunately, not the backup pool !

Upon reboot the backup pool is always left unmounted.
We have to manually mount it:

root@dc1-pbs01:~# zpool import
   pool: backup
     id: 8416596132581008822
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        backup      ONLINE
          raidz1-0  ONLINE
            sda     ONLINE
            sdb     ONLINE
            sdc     ONLINE
            sdd     ONLINE
            sde     ONLINE
            sdf     ONLINE
            sdg     ONLINE
            sdh     ONLINE
            sdi     ONLINE
            sdj     ONLINE
            sdk     ONLINE
            sdl     ONLINE
            sdm     ONLINE
            sdn     ONLINE
            sdo     ONLINE

root@dc1-pbs01:~# zpool import backup
root@dc1-pbs01:~# zpool status
  pool: backup
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        backup      ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0
            sdf     ONLINE       0     0     0
            sdg     ONLINE       0     0     0
            sdh     ONLINE       0     0     0
            sdi     ONLINE       0     0     0
            sdj     ONLINE       0     0     0
            sdk     ONLINE       0     0     0
            sdl     ONLINE       0     0     0
            sdm     ONLINE       0     0     0
            sdn     ONLINE       0     0     0
            sdo     ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
config:

        NAME                                                 STATE     READ WRITE CKSUM
        rpool                                                ONLINE       0     0     0
          mirror-0                                           ONLINE       0     0     0
            nvme-eui.00000000000000018ce38e010009f4e7-part3  ONLINE       0     0     0
            nvme-eui.00000000000000018ce38e010009f4e6-part3  ONLINE       0     0     0

errors: No known data errors

This is a real problem, because this server might be rebooted sometimes and we expect ALL pools to be correctly mounted not half of them.
And specially the backup pool.

This really looks like some kind of bug or big discrepancies to me.

Version is Backup Server 1.1-9 and under licence.

 

[DynFi User]

Any info about this one ?

I have found this thread about the same issue with PVE, but Mount Point config is not handled the same in PVE and in Proxmox Backup.
https://forum.proxmox.com/threads/zfs-pool-does-not-mount-at-boot-time.55732/

 

[DynFi User]

When we have installed the system we have configured the root pool. But since we didn't had the 3.5" disks at hand, we had to wait until all disks were received to configure / setup the second pool (backup pool).

After boot and successful install of all updates and disks, the "backup" pool with it's 13x 14TB disks was configured.

Now upon reboot the volume is not automatically mounted by PBS.

Simple question : how can I fix that ??

 

[dcsapak]

how did you crate the pool ? via gui or manually?
can you post the journal since boot? (journalctl -b)

it's probably some systemd-unit ordering issue

 

[DynFi User]

It is 18000 line long…

I think i created it through the GUI.

 

[DynFi User]

We have first installed the system on two M.2 NVMe and after used the PBS GUI in order to configure the second pool with the 3.5" HDD.

Is there a way to have a follow-up on this one ?
Because everytime we are booting we have to manually mount the pool which is really not ok.

 

[dcsapak]

as i said, it is probably a systemd unit ordering issue, but to verify i'd have to look at the journal of a boot...
at least until some failure of the zfs import or such

 

[DynFi User]

Here is the output of the DMESG

 

[dcsapak]

as i said, the journal would be important, the dmesg does not contain any info pertaining to systemd units...

 

[DynFi User]

Here is the output of journalctl -b

 

[dcsapak]

seems the journal is from a vm unrelated to pbs? at least i cannot see anything of zfs/pbs...

 

[DynFi User]

Here is the compressed file.

Please use tar -xvzf to expand.

thanks and sorry for the mistake.

 

[dcsapak]

ok, looking at the journal, it seems that the zfs-import service runs before the disks are initialized

a workaround could be to add a 'rootdelay=10' to the kernel commandline, so that the controller has time to add the disks while the system waits, in this example 10 seconds

the zfs-import fails at 12:48:52 and more disks are detected at 12:48:54, so it's only 2 seconds that may be missing, so giving the controller an additional 10 seconds could help

how you can edit the kernel commandline is written in this article:
https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_edit_kernel_cmdline

 

[DynFi User]

I already have this on the file :

root=ZFS=rpool/ROOT/pbs-1 boot=zfs

So I guess the new version should look like this :
root=ZFS=rpool/ROOT/pbs-1 boot=zfs rootdelay=10

Can you please confirm this ?
And the fact that I shall trigger : proxmox-boot-tool refresh once to generate the right files.

thanks !

 

[dcsapak]

yes exactly