[SOLVED] how do I solve a datastore missing each reboot?

[bunk3m]

Hi everyone. (newbie alert)

I have a PBS 3.3.2 server on bare metal. There are two datastores used for backup of a PVE server.

I had a couple of drives and decided to stripe two 8TB drives when I started.

I wasn't able to create the stripe using the gui, so I did it via the command line with

zpool create -f -o ashift=12 backup sdc sdd

This worked fine until I did an update of PBS in around November and the first backup failed as it couldn't "open chunk store".

After reading the forums, I found that the disks were OK just not showing up in the "zpool list" or "zpool status".

I followed directions from other posts and found the disks using

ls -lah /dev/disk/by-id/

and added it back using

zpool import -a -d /dev/disk/by-id/actual_disk_ID

The datastore was available and has worked for a few months.

I thought I fixed the problem but when I ran the next update yesterday and rebooted, I have exactly the same problem. I used the same steps to add back the disk. The second datastore on the pbs was created using the GUI and doesn't have this problem. I suspect that in doing the manual datastore creation I missed something or made a mistake but I don't know where to start to figure it out.

Could you provide some suggestions on what I have to look for to fix this properly so each reboot I don't have to manually add back the datastore?

Thanks in advance.

 

[waltar]

Just do "zpool import -d /dev/disk/by-id/" or when error with poolname backup too.
Then reboot the server to check as should be ok then.

 

[bunk3m]

Just do "zpool import -d /dev/disk/by-id/" or when error with poolname backup too.
Then reboot the server to check as should be ok then.

Thank you @waltar

I tried to do the "zpool import -d /dev/disk/by-id/" adding the diskID at the end. Then I rebooted.
When the pbs server restarted, the pool wasn't available with same error about chunk storage.

I tried with "zpool import backup" only which mounted the pool again.
When I reboot, the pool is again missing with same error.

The error continues to be:

Bad Request (400)
unable to open chunk storage at "/backup/.chunks" - No such file or directory (os error 2)

I'm assuming I didn't understand what you meant as I have to manually import the pool called "backup" at each reboot. As I mentioned earlier, the only difference between this and the other pool was that this was created with the command line and is a multi-disk stripe where the other pool was created by the gui and has only one disk in the stripe.

 

[Chris]

Hi,

I tried with "zpool import backup" only which mounted the pool again.
When I reboot, the pool is again missing with same error.

you have to make sure the pool is imported automatically at startup. This can be achieved by either setting the cachefile for the zpool e.g. by following and adapting the commands here to your need https://forum.proxmox.com/threads/z...-with-cache-file-how-to-fix.61925/post-284032
or use a systemd unit to import the pool by e.g. systemctl enable zfs-import@backup.service.

 

[waltar]

Look at your zfs services for system boot:
systemctl -a|grep zfs
You should even have exactly named this kind of link:
l /etc/systemd/system/zfs-import.target.wants/zfs-import@backup.service
lrwxrwxrwx 1 root root 39 Apr 15 2024 /etc/systemd/system/zfs-import.target.wants/zfs-import@backup.service -> /lib/systemd/system/zfs-import@.service

 

[bunk3m]

Look at your zfs services for system boot:
systemctl -a|grep zfs
You should even have exactly named this kind of link:
l /etc/systemd/system/zfs-import.target.wants/zfs-import@backup.service
lrwxrwxrwx 1 root root 39 Apr 15 2024 /etc/systemd/system/zfs-import.target.wants/zfs-import@backup.service -> /lib/systemd/system/zfs-import@.service

Thank you @Chris and @waltar .

When I run "systemctl -a|grep zfs" I see

# systemctl -a|grep zfs
...
  zfs-import-cache.service          loaded    active   exited    Import ZFS pools by cache file
  zfs-import-scan.service           loaded    inactive dead      Import ZFS pools by device scanning
  zfs-import@backup2.service        loaded    active   exited    Import ZFS pool backup2

So both the "zfs-import-scan.service is dead and there is no "zfs-import@backup.service".

And

# systemctl status zfs-import-scan
○ zfs-import-scan.service - Import ZFS pools by device scanning
     Loaded: loaded (/lib/systemd/system/zfs-import-scan.service; enabled; preset: disabled)
     Active: inactive (dead)
  Condition: start condition failed at Mon 2025-02-03 19:28:21 EST; 23h ago
             └─ ConditionFileNotEmpty=!/etc/zfs/zpool.cache was not met
       Docs: man:zpool(8)

Feb 03 19:28:21 pbs systemd[1]: zfs-import-scan.service - Import ZFS pools by device scanning was skipped because of an unmet condition >
lines 1-8/8 (END)

I created the zpool.cache & did the "update-initramfs -k all -u" but the zpool.cache wasn't there after the reboot.

I recreated the zpool.cache file again and I see it under "/etc/zfs/". It has the right time and date stamp.

I ran "update-initramfs -k all -u" again.

After the reboot, the zpool.cache file is there but the zpool "backup" didn't mount.

So finally, I tried "systemctl enable zfs-import@backup.service" and this finally worked and allowed both zpools to mount on a reboot.

Thanks for your help.