zfs pool missing / devices currently unavailable

[michelvdriel]

Hi,

After a restart of proxmox my ZFS pool is unmountable.

Serveral outputs:

zpool status

no pools available

zpool import -d /dev/disk/by-id

   pool: beastpool
     id: 9071326892089082027
  state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:

        beastpool                   ONLINE
          raidz2-0                  ONLINE
            wwn-0x50014ee2096e763a  ONLINE
            wwn-0x5000c500a5bbca7a  ONLINE
            wwn-0x5000c500a5be7966  ONLINE
            wwn-0x5000c500a5bff727  ONLINE
            wwn-0x50014ee26724a07e  ONLINE
            wwn-0x50014ee211cf6f82  ONLINE

zpool import -d /dev/disk/by-id beastpool

cannot import 'beastpool': one or more devices is currently unavailable

I have already moved the disks around to unused slots to rule out any connection issues. All containers and VMs are stopped, as well as the NFS server, so nothing is using the mountlocation at the moment. I have renamed my zpool.cache (but have not been able to set a new one, as that command also says devices are unavailable).
Not sure how to proceed. Any guidance is appreciated.

 

[michelvdriel]

Some extra info from the syslog. Attached as a file as I am unable to post it due to a constraint in the number of characters.

# zpool import beastpool

 

[apoc]

Can you check /dev/disk/by-id and verify the wwn-links are actually there as expected?
I recently was troubleshooting an issue and uninstalled miltipathd and all my devnodes changed. Maybe something happened here.

I am working with vdevs though and those did not get created due to missing dev nodes...

 

[michelvdriel]

@tburger Yes they are. I deleted them and rebooted to make sure they were recreated and this did indeed happen.

 

[apoc]

Have you checked them char by char?

 

[michelvdriel]

@tburger Yes, they are exactly the same.

# ls wwn*

wwn-0x5000c500a5bbca7a
wwn-0x5000c500a5bbca7a-part1
wwn-0x5000c500a5bbca7a-part9
wwn-0x5000c500a5be7966
wwn-0x5000c500a5be7966-part1
wwn-0x5000c500a5be7966-part2
wwn-0x5000c500a5bff727
wwn-0x5000c500a5bff727-part1
wwn-0x5000c500a5bff727-part2
wwn-0x50014ee2096e763a
wwn-0x50014ee2096e763a-part1
wwn-0x50014ee2096e763a-part9
wwn-0x50014ee211cf6f82
wwn-0x50014ee211cf6f82-part1
wwn-0x50014ee211cf6f82-part2
wwn-0x50014ee26724a07e
wwn-0x50014ee26724a07e-part1
wwn-0x50014ee26724a07e-part2
wwn-0x5002538d401f06c1
wwn-0x5002538d401f06c1-part1
wwn-0x5002538d401f06c1-part2
wwn-0x5002538d401f06c1-part3
wwn-0x5002538d401f06c1-part4
wwn-0x5002538d4257374e

 

[apoc]

wwn-0x50014ee211cf6f82-part2

I find it odd that there are some "part2" elements in your list.
All my zfs pooldisks have a "part9" as second partition. Without a doubt...
I don't know zfs enough to judge if it is looking for "part9" especially but that could be a breadcrumb you can follow.

 

[michelvdriel]

In my logs I can see it is trying to open part2. I've looked around on Google, but have not seen anything that points to a part9 being required...

 

[apoc]

It was just something I have recognised.
It wouldn't fit the message but have you checked that the mountpath directory is not there?
I don't recall the message but it took me quite a while to realize that this was a problem.

 

[michelvdriel]

The directory was there (not sure if I created this manually when I set up the system or it was created by one of the VMs after the pool went down). I have deleted it but this made no difference. I was hoping this could be fixed, but it looks to be an unfixable error.

 

[apoc]

I don't think it will change anything but its worth a try.
Check if beastpool is actually recognised via "zpool import"
Then try importing by zpool Id.
Alternatively have you tried importing it without "-d" option?

 

[michelvdriel]

The pool is recognised when I do "zpool import", gives the same output as in my first post (with the -d option). I've also tried importing without -d, it doesn't make a difference.

 

[apoc]

Which PVE and ZOL version are you actually running?
The "upgrade" message in the output of "zpool import" indicates there has been an update of ZOL - is that correct?
Have you tried downgrading or boot a Debian/Ubuntu live distro to see if you can import the pool there?

 

[michelvdriel]

zfs-0.8.5-pve1
zfs-kmod-0.8.5-pve1

There has been no recent upgrade of ZOL (and an upgrade wasn't the reason of powering down the server, it needed to be moved and thus disconnected from power for a short time). The upgrade message has been there since I moved the pool over to proxmox over a year ago. I just didn't upgrade it as the message also states it isn't required and I didn't want to risk anything going wrong.

 

[apoc]

and I didn't want to risk anything going wrong.

I see. Have been there as well but finally upgraded my pools this year.

I also don't think that this relates, but recognised the message while scrolling through this thread again.

If I were you I would really try the method of booting another system (Ubuntu 20.04 for instance) and try importing the pool there.
Through this approach you will find out if the issue relates to the pool or the system.

If it works there, reinstall proxmox. If not - hope you have a backup :/

 

[michelvdriel]

I already tried two different OSes: Ubuntu 20.04 and the latest version of FreeBSD. Under FreeBSD it prints all kinds of "can't open objset" errors in the terminal. These are the same errors as in proxmox, but here they are logged directly to the syslog. I'm giving up, all important data is backed up. The rest is a nuisance, but not the end of the world. Thanks for your suggestions!

 

[apoc]

You are very welcome.
Bummer though you need to restore...
On the other hand at least you have a backup