zfs loss storage when add new block device by FC-switch

[Alibek]

In linux all /dev/disk/by-* show link to block device in /dev/sd?, example:

# ls -la /dev/disk/by-path/
total 0
drwxr-xr-x 2 root root 200 Nov  2 12:58 .
drwxr-xr-x 8 root root 160 Nov  1 21:21 ..
lrwxrwxrwx 1 root root   9 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ecd8890-lun-0 -> ../../sde
lrwxrwxrwx 1 root root  10 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ecd8890-lun-0-part1 -> ../../sde1
lrwxrwxrwx 1 root root   9 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ecd8890-lun-1 -> ../../sdf
lrwxrwxrwx 1 root root  10 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ecd8890-lun-1-part1 -> ../../sdf1
lrwxrwxrwx 1 root root  10 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ecd8890-lun-1-part9 -> ../../sdf9
lrwxrwxrwx 1 root root   9 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ecd8890-lun-3 -> ../../sdg
lrwxrwxrwx 1 root root   9 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ee0b861-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root   9 Nov  2 12:59 pci-0000:05:00.0-fc-0x2100000e1ee0b861-lun-2 -> ../../sdh

# ls -la /dev/disk/by-id/ | grep scsi
lrwxrwxrwx 1 root root    9 Nov  2 12:59 scsi-3600144f0b49c14d1000059fabff00003 -> ../../sdh
lrwxrwxrwx 1 root root    9 Nov  2 12:59 scsi-3600144f0c7a5bc6100005818b76d0013 -> ../../sdg
lrwxrwxrwx 1 root root    9 Nov  2 12:59 scsi-3600144f0c7a5bc610000585d0c610018 -> ../../sde
lrwxrwxrwx 1 root root   10 Nov  2 12:59 scsi-3600144f0c7a5bc610000585d0c610018-part1 -> ../../sde1
lrwxrwxrwx 1 root root    9 Nov  2 12:59 scsi-3600144f0c7a5bc61000058d3b96d001d -> ../../sdf
lrwxrwxrwx 1 root root   10 Nov  2 12:59 scsi-3600144f0c7a5bc61000058d3b96d001d-part1 -> ../../sdf1
lrwxrwxrwx 1 root root   10 Nov  2 12:59 scsi-3600144f0c7a5bc61000058d3b96d001d-part9 -> ../../sdf9

ZFS have attached device by id

# zpool status
  pool: pool
 state: ONLINE
  scan: scrub repaired 0 in 1h39m with 0 errors on Sun Oct  8 02:03:34 2017
config:

    NAME                                      STATE     READ WRITE CKSUM
    pool                                      ONLINE       0     0     0
      scsi-3600144f0c7a5bc61000058d3b96d001d  ONLINE       0     0     0

Before export new block device from storage to hba, id have next configuration
/dev/disk/by-id/scsi-3600144f0c7a5bc61000058d3b96d001d -> ../../sdd

When added new block device by FC-switch, FC-HBA change block device names
/dev/disk/by-id/scsi-3600144f0c7a5bc61000058d3b96d001d -> ../../sdf

and ZFS can't access to device

And solve this possible only if reboot server

 

[jim.bond.9862]

I see at least one issue with your setup but there could be more.
as per your zfs config the device name shouldn't matter as zfs uses the uuid, BUT
your disk is partitioned into at least 2 partition , yet you adding the whole disk to the pool. this is wrong I think.
you need either use the partition you want in the pool or dump the whole disk layout and use the whole disk as RAW device.
how did you partition it like that and why?

lrwxrwxrwx 1 root root 9 Nov 2 12:59 scsi-3600144f0c7a5bc61000058d3b96d001d -> ../../sdf
lrwxrwxrwx 1 root root 10 Nov 2 12:59 scsi-3600144f0c7a5bc61000058d3b96d001d-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 10 Nov 2 12:59 scsi-3600144f0c7a5bc61000058d3b96d001d-part9 -> ../../sdf9

PS>> it looks like the disk was part of other ZFS boot setup. that is the only time have seen partition 9 used.

 

[Alibek]

This partitions is created automaticaly by zpool on init operation of new pool:

fdisk -l /dev/sdf

Disk /dev/sdf: 1 TiB, 1099511627776 bytes, 2147483648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6D391C8F-92B7-5D40-8951-BD49E925FA72

Device          Start        End    Sectors  Size Type
/dev/sdf1        2048 2147465215 2147463168 1024G Solaris /usr & Apple ZFS
/dev/sdf9  2147465216 2147481599      16384    8M Solaris reserved 1

That is not problem.

Problem in behavior of the system with hot-attached new storage volume, on this operation FC-switch update configuration for connected FC-HBA on servers and linux kernel of servers reorder block devices. Becouse scsi-id, wwn-id and other is symbol links block device names than on change block device name change path to device:

# stat /dev/disk/by-id/scsi-3600144f0c7a5bc61000058d3b96d001d
  File: ‘/dev/disk/by-id/scsi-3600144f0c7a5bc61000058d3b96d001d’ -> ‘../../sdf’
  Size: 9            Blocks: 0          IO Block: 4096   symbolic link
Device: 6h/6d    Inode: 536         Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-11-02 12:59:25.744018629 +0300
Modify: 2017-11-02 12:59:16.332105485 +0300
Change: 2017-11-02 12:59:16.332105485 +0300

 

[jim.bond.9862]

you are absolutely correct I checked my test setup and it is the same. sorry. jump the gun here.
I am so not used to deal with how ZFS does it things.

 

[Alibek]

I start discussion at linux-kernel@vger.kernel.org - https://marc.info/?l=linux-kernel&m=155743464501768
Please support!