ZFS pool won’t import after switching from /dev/sdx to /dev/disk/by-id – mixed vdev paths

[stingray2362]

Hello everyone,

I have a ZFS1 pool made up of 10 disks. When I first created the pool, I only had 5 drives, and they formed the first vdev. At the time, I used /dev/sdx to set them up. Later on, I added another 5 drives to the pool and this time I used the /dev/disk/by-id path. So the pool ended up being a mix of both. I had no issue with this and was able to use whole pool's storage. All the disks are from the same manufacturer and have the same size (20TB).

I recently tried to fix this inconsistency by following this guide:
https://serverfault.com/questions/8...-in-a-zfs-pool-from-dev-sdx-to-dev-disk-by-id

I ran

zpool export pool-name

to export the pool. Initially, I got a "zpool is busy" message, but after stopping all my containers in Proxmox, I was able to successfully run the export command.

Next, I tried to import the pool using:

zpool import pool-name -d /dev/disk/by-id

But now the pool can't be found. The drives are all visible in /dev/disk/by-id, and by checking their SMART data, I can distinguish between the original 5 drives (the first vdev) and the 5 newer ones (the second vdev). My question is: is there a way to import the pool by explicitly specifying the disk IDs for both vdev0 and vdev1?

From my understanding, ZFS doesn’t store pool configuration based on how drives were referenced, but rather uses the metadata stored on the disks themselves. As one reddit user put it: For the record exporting a pool is akin to windows ejecting a usb drive. Can't destroy up a pool with an export command.
Have I managed to destroy the pool?

 

[Nemesiz]

Hi,

have you tried to run 'zpool import -d /dev/disk/by-id' to see what zfs sees ?

 

[stingray2362]

Hi,

have you tried to run 'zpool import -d /dev/disk/by-id' to see what zfs sees ?

Here is the output. I redacted some identifiers:

zpool import -d /dev/disk/by-id
   pool: my-pool
     id: 9362173134538127XXX
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        my-pool                                 ONLINE
          raidz1-0                                ONLINE
            ata-TOSHIBA_MG10ACA20TE_44F0A0GMXXXX  ONLINE
            ata-TOSHIBA_MG10ACA20TE_44F0A0A5XXXX  ONLINE
            ata-TOSHIBA_MG10ACA20TE_44F0A0CNXXXX  ONLINE
            ata-TOSHIBA_MG10ACA20TE_44F0A0CTXXXX  ONLINE
            ata-TOSHIBA_MG10ACA20TE_44F0A0GFXXXX  ONLINE
          raidz1-1                                ONLINE
            wwn-0x5000039d78e02XXX                ONLINE
            wwn-0x5000039d88c93XXX                ONLINE
            wwn-0x5000039d88c96XXX                ONLINE
            wwn-0x5000039d88c96XXX                ONLINE
            wwn-0x5000039d88c96XXX                ONLINE

   pool: my-pool
     id: 4156001255627227XXX
  state: UNAVAIL
status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

        my-pool                         UNAVAIL  insufficient replicas
          raidz1-0                        UNAVAIL  insufficient replicas
            wwn-0x5000039d38ca53XX-part9  UNAVAIL  corrupted data
            wwn-0x5000039d38ca49XX-part9  UNAVAIL  corrupted data
            wwn-0x5000039d38ca4bXX-part9  UNAVAIL  corrupted data
            wwn-0x5000039d38ca4bXX-part9  UNAVAIL  corrupted data
            wwn-0x5000039d38ca53XX-part9  UNAVAIL  corrupted data

Edit:
I managed to import the pool with the following command:

zpool import -d /dev/disk/by-id 9362173134538127XXX

The pool is usable again. However, I’m still a bit confused—why does zpool import show two versions of the same pool name, with one being corrupt?

In the future, if I want to add more storage, can I simply add another vdev (in my case, a RAIDZ1 with 5 identical drives), but using the drive's id?

 

[kawano-y]

Hi,

Good to hear that you were able to import the pool again.

The `UNAVAIL` entry was probably not created by the export/import operation.

`zpool import -d /dev/disk/by-id` scans the selected device paths for ZFS labels.
In your output, the two entries have the same pool name but different pool IDs,
so ZFS appears to be seeing two different sets of metadata that both refer to the
name `my-pool`.

> pool: my-pool
> id: 9362173134538127XXX
> state: ONLINE

This looks like the real/current pool, and importing it by pool ID was a good
choice.

> pool: my-pool
> id: 4156001255627227XXX
> state: UNAVAIL

The second entry may be stale ZFS metadata left on disks or partitions from an
older layout. The fact that it references `-part9` devices also suggests that
these may be old partition labels rather than the current whole-disk vdevs.

I would be careful with cleaning those labels using `zpool labelclear`, `wipefs`,
or similar tools. It would be best to first map every physical disk and confirm
which device paths belong to the currently imported pool.

For future expansion, yes, adding another 5-disk RAIDZ1 vdev using
`/dev/disk/by-id` should be fine and is generally preferable to using `/dev/sdX`.
Just double-check the disk IDs and the intended layout before running
`zpool add`, because adding a vdev is usually not something you can easily undo.