Procedure for moving two disks ZFS RAIDZ2 from internal SATA to external USB3

[guerby]

Hi,

I have a 6 disks RAIDZ2 pool, all 6 disks are currently on internal SATA.

To workaround an issue on two of the SATA ports (1) I'd like to move (until a receive a new SATA PCIe controler card) two disks to an external USB3 enclosure.

ZFS when on SATA has "ata-xxx-serialnumber" in front on each disk when doing zpool status which correspond to /dev/disk/by-id/ entry.

When moved on USB the disks become /dev/disk/by-id/usb-xxx-usbportnumber.

I can't find a sequence of ZFS command to have all 6 disks back online in the raidz2 when I move two of the disks to the external enclosure, I'm confused by the various ZFS concepts offline/online/replace/export/import and in what order these should be done.

How should I proceed ?

Notes:
- this is not a root array so I can afford to offline everything in this pool, may be zpool export mypool then zpool import mypool?
- because of (1) the array on SATA is resilvering but the asmedia issue up to now happens before the end of resilver and disks go to fault state, so the pool is in resilvering state.


(1) https://forum.proxmox.com/threads/a...-6-4-and-not-on-5-11-22-pve-7-0-branch.97743/

 

[Dunuin]

When importing a pool ZFS shouldn't care if the disk is "/dev/disk/by-id/usb-xxx-usbportnumber" or "/dev/disk/by-id/ata-xxx-serialnumber" because all the metadata for identifying disks is stored directly on the disks themself. So I guess a "zfs export" and "zfs import" should work.

 

[guerby]

@Dunuin

zpool export ssdbck1 commands succeed (no output, exit code zero) but the pool is still there after the command. zpool import says "no pools available to import". zpool import ssdbck1 says

# zpool import ssdbck1
cannot import 'ssdbck1': a pool with that name already exists
use the form 'zpool import <pool | id> <newpool>' to give it a new name

After a poweroff I moved the disks to external USB then power on again I get:

# zpool status ssdbck1
  pool: ssdbck1
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Oct 12 15:50:05 2021
    1.26T scanned at 18.2G/s, 154M issued at 2.16M/s, 11.9T total
    0B resilvered, 0.00% done, no estimated completion time
config:

    NAME                                             STATE     READ WRITE CKSUM
    ssdbck1                                          DEGRADED     0     0     0
      raidz2-0                                       DEGRADED     0     0     0
        3748762577798287549                          UNAVAIL      0     0     0  was /dev/disk/by-id/ata-xxx-part1
        11918204772067325032                         UNAVAIL      0     0     0  was /dev/disk/by-id/ata-xxx-part1
        ata-xxx  ONLINE       0     0     0
        ata-xxx  ONLINE       0     0     0
        ata-xxx ONLINE       0     0     0
        ata-xxx ONLINE       0     0     0

errors: No known data errors

The first two disks are present in /dev/disk/by-id/usb-xxx, I can read data from them, do a smartctl but ZFS doesn't seem to consider them.

Any idea on how to proceed from there?

 

[guerby]

I powered off only the external USB dock, then redid export then power on, then import and now the 2 USB disks are online with the others!

# zpool history
...
History for 'ssdbck1':
...
2021-10-12.16:19:16 zpool export ssdbck1
2021-10-12.16:19:37 zpool import -d /dev/disk/by-id/ -o cachefile=none ssdbck1

# zpool events
....
Oct 12 2021 16:19:16.042262581 sysevent.fs.zfs.pool_export
Oct 12 2021 16:19:23.210055787 sysevent.fs.zfs.config_sync
Oct 12 2021 16:19:23.498047520 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.498047520 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.498047520 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.498047520 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.502047405 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.502047405 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.522046830 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.522046830 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.526046715 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.526046715 ereport.fs.zfs.checksum
Oct 12 2021 16:19:23.706041548 sysevent.fs.zfs.history_event
Oct 12 2021 16:19:23.758040055 sysevent.fs.zfs.config_sync
Oct 12 2021 16:19:23.758040055 sysevent.fs.zfs.pool_import
Oct 12 2021 16:19:23.790039135 sysevent.fs.zfs.history_event
Oct 12 2021 16:19:26.853951364 sysevent.fs.zfs.config_sync

May be making sure the disks to be moved are offline when doing export is a requirement?

 

[Dunuin]

You need to teporarily remove the pool as a storage from PVE first (Datacenter -> Storage -> YourPool -> Remove) or PVE will autoimport the pool again within a second after export.

 

[guerby]

You need to teporarily remove the pool as a storage from PVE first (Datacenter -> Storage -> YourPool -> Remove) or PVE will autoimport the pool again within a second after export.

Thanks! Looks like someone had the issue before:

https://forum.proxmox.com/threads/prevent-zpools-from-auto-importing.30551/

And there's now a "disable" which I assume should prevent autoimport on the specified pool.