ZFS defaults to /dev/disk/by-id

johnoathome · Apr 23, 2025

HI All,

I'm seeing strange behaviour with a 5 bay Orico USB external drive chassis that is preventing me from using it with ZFS. The chassis has 4 x 8tb disks installed and each disk appears in /dev - in this case as /dev/sde, /dev/sdf, /dev/sdg and /dev/sdh.

So far so good. But only 3 of the drives appear in /dev/disk/by-id, or more precisely only 3 drives appear at any one time.

So for example, on reboot, "ls -la /dev/disk/by-id | grep usb" shows device id's for sdg, sdh and sdf but not sde.

Running "fdisk /dev/sde" shows the device is actually there. If I modify it with fdisk and check again, /dev/disk/by-id now shows sde, sdg and sdh but now sdf is missing.

This goes round and around until you realise you can only see 3 drives at any one time in /dev/disk/by-id.

The issue of course is that referencing disks as /dev/sdX in a zfs pool risks random change by the OS. Because I can't use /dev/disk/by-id I've tried a few other referencing methods including by-partuuid, GPT partition labels and udev rules.

They all look great and allow the pool to be created. But on the first of second reboot, ZFS decides to map the disks over to /dev/disk/by-id and reports that one of the disks is faulted and has corrupted data. I assume this is because it can only find 3 disks in /dev/disk/by-id but I'm not entirely sure.

The worst part is that I see this behaviour using all referencing methods including /dev/sdX, so the external bay is essentially useless for ZFS.

Is there a way to stop ZFS remapping the drives to /dev/disk/by-id?

Running proxmox-ve: 8.4.0 (kernel 6.8.12-9-pve)

Thanks

Impact · Apr 23, 2025

Please share

Bash:

lsblk -e230 -o+FSTYPE,LABEL,MODEL,TRAN,VENDOR,SERIAL
ls -l /dev/disk/by-id/*
zpool status -vP
zpool status -vPL

Feel free to censor part of the serial.
Also see here: https://openzfs.github.io/openzfs-docs/Project and Community/FAQ.html#the-etc-zfs-zpool-cache-file

gfngfn256 · Apr 23, 2025

If you have something like this one, then I believe your issues are related to the that enclosure/connection itself.
Read the reviews of this product - and you will find familiarity with your issues.

I definitely wouldn't be running a ZFS pool/raid etc. off that USB enclosure.

johnoathome · Apr 23, 2025

Impact said:
Please share

Bash:

lsblk -e230 -o+FSTYPE,LABEL,MODEL,TRAN,VENDOR,SERIAL ls -l /dev/disk/ zpool status -vP zpool status -vPL

Feel free to censor part of the serial.
Also see here: https://openzfs.github.io/openzfs-docs/Project and Community/FAQ.html#the-etc-zfs-zpool-cache-file

Hi,

Code:

lsblk -e230 -o+FSTYPE,LABEL,MODEL,TRAN,VENDOR,SERIAL

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS FSTYPE     LABEL   MODEL                   TRAN   VENDOR   SERIAL
sde           8:64   0   7.3T  0 disk                                004-2U9188              usb    ST8000DM 000000123AE
sdf           8:80   0   7.3T  0 disk                                004-2CX188              usb    ST8000DM 000000123AE
sdg           8:96   0   7.3T  0 disk                                004-2M2101              usb    ST8000VN 000000123AE
sdh           8:112  0   7.3T  0 disk                                004-2M2101              usb    ST8000VN 000000123AE

Filtered to the drives in question.  No need to redact the SNs. It is the same for all drives which I suspect is paty of the problem.

Code:

ls -l /dev/disk/

drwxr-xr-x 2 root root  680 Apr 23 22:41 by-diskseq
drwxr-xr-x 2 root root 1060 Apr 23 22:41 by-id
drwxr-xr-x 2 root root  200 Apr 23 22:41 by-label
drwxr-xr-x 2 root root  360 Apr 23 13:07 by-partlabel
drwxr-xr-x 2 root root  740 Apr 23 22:41 by-partuuid
drwxr-xr-x 2 root root  760 Apr 23 22:41 by-path
drwxr-xr-x 2 root root  440 Apr 23 22:41 by-uuid

Code:

root@noc1:~# zpool status usbext -vP
  pool: usbext
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME                                                                STATE     READ WRITE CKSUM
        usbext                                                              DEGRADED     0     0     0
          raidz2-0                                                          DEGRADED     0     0     0
            /dev/sde1                                                       ONLINE       0     0     0
            /dev/disk/by-id/usb-ST8000VN_004-2M2101_000000123AE8-0:0-part1  FAULTED      0     0     0  corrupted data
            /dev/disk/by-id/usb-ST8000DM_004-2CX188_000000123AE8-0:0-part1  ONLINE       0     0     0
            /dev/disk/by-id/usb-ST8000DM_004-2U9188_000000123AE8-0:0-part1  ONLINE       0     0     0

errors: No known data errors
This is what it looks like after first reboot

Code:

root@noc1:~# zpool status usbext -vPL
  pool: usbext
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME           STATE     READ WRITE CKSUM
        usbext         DEGRADED     0     0     0
          raidz2-0     DEGRADED     0     0     0
            /dev/sde1  ONLINE       0     0     0
            /dev/sde1  FAULTED      0     0     0  corrupted data
            /dev/sdg1  ONLINE       0     0     0
            /dev/sdh1  ONLINE       0     0     0

errors: No known data errors

Interesting the -vPL switches show sde1 being referenced twice!

I'll have a read through the caching stuff, thanks.

johnoathome · Apr 23, 2025

gfngfn256 said:
If you have something like this one, then I believe your issues are related to the that enclosure/connection itself.
Read the reviews of this product - and you will find familiarity with your issues.

I definitely wouldn't be running a ZFS pool/raid etc. off that USB enclosure.

johnoathome · Apr 23, 2025

gfngfn256 said:
If you have something like this one, then I believe your issues are related to the that enclosure/connection itself.
Read the reviews of this product - and you will find familiarity with your issues.

I definitely wouldn't be running a ZFS pool/raid etc. off that USB enclosure.

Not that exact model but I'm guessing it has the same tech. I'm in touch with the manufacturer and will see what they have to say for themselves. The device is marketed as compatable with Linux but last I checked, ZFS is a part of that. I bought 2 of these for a project at more than $AUD200 each so I'll get to the bottom of it or send them back.

Impact · Apr 23, 2025

Sorry, I meant to write ls -l /dev/disk/*. Seems like it messes with the disk identifiers in some way.
The serials being all the same and the model, which I expect to show differently, looking like that is a hint.
Does smartctl -A /dev/sde work properly?

If any of the other paths are better suited you can try something like this

Bash:

zpool export usbext        
zpool import -d /dev/disk/.../ usbext

Make sure it's unused first, of course.

gfngfn256 · Apr 23, 2025

johnoathome said:
compatable with Linux but last I checked, ZFS is a part of that

ZFS & its RAID configuration will have its own overheads, so saying it is compatible with Linux probably doesn't fully guarantee ZFS smooth sailing.

johnoathome · Apr 23, 2025

Impact said:
Sorry, I meant to write ls -l /dev/disk/*. Seems like it messes with the disk identifiers in some way.
The serials being all the same and the model, which I expect to show differently, looking like that is a hint.
Does smartctl -A /dev/sde work properly?

If any of the other paths are better suited you can try something like this

Bash:

zpool export usbext zpool import -d /dev/disk/.../ usbext

Make sure it's unused first, of course.

Yes I've tried the other path identifiers and get the same result. For example:

Code:

zpool import -d /dev/disk/by-partuuid

root@noc1:~# zpool status usbext -vP
  pool: usbext
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME                                                                STATE     READ WRITE CKSUM
        usbext                                                              DEGRADED     0     0     0
          raidz2-0                                                          DEGRADED     0     0     0
            /dev/disk/by-id/usb-ST8000VN_004-2M2101_000000123AE8-0:0-part1  ONLINE       0     0     0
            16240640760338679675                                            FAULTED      0     0     0  was /dev/disk/by-id/usb-ST8000VN_004-2M2101_000000123AE8-0:0-part1
            /dev/disk/by-id/usb-ST8000DM_004-2CX188_000000123AE8-0:0-part1  ONLINE       0     0     0
            /dev/disk/by-id/usb-ST8000DM_004-2U9188_000000123AE8-0:0-part1  ONLINE       0     0     0

So even though I've imported the pool using by-partuuid, zfs is referencing by-id!

I think this is a fundamental issue with the hardware or firmware.

niko30127 · 2025-10-03T02:54:45+0200

hi , i got same issue like you and solved by inspiring me from here

here the step :
1- copy this to there: cp /lib/udev/rules.d/60-persistent-storage.rules /etc/udev/rules.d/
2- check device property an compare each usb device to find a resilient property different from themselves
example for /dev/sda

Bash:

udevadm info --name=/dev/sda --query=property

3- edit /etc/udev/rules.d/60-persistent-storage.rules by adding you specific rule for this devices

exemple i added specific rule for ID_USB_SERIAL_SHORT = "DD56419883914" wich is different from my USB device and i added ID_PART_TABLE_UUID in disk/by-id/ mapping

original code

Bash:

KERNEL=="sd*|sr*|cciss*", ENV{DEVTYPE}=="disk", ENV{ID_USB_SERIAL}=="?*", SYMLINK+="disk/by-id/usb-$env{ID_USB_SERIAL}"
KERNEL=="sd*|cciss*", ENV{DEVTYPE}=="partition", ENV{ID_USB_SERIAL}=="?*", SYMLINK+="disk/by-id/usb-$env{ID_USB_SERIAL}-part%n"

modified Code:

Bash:

KERNEL=="sd*|sr*|cciss*", ENV{DEVTYPE}=="disk", ENV{ID_USB_SERIAL}=="?*", SYMLINK+="disk/by-id/usb-$env{ID_USB_SERIAL}"
ENV{ID_USB_SERIAL_SHORT}=="DD56419883914", \
  SYMLINK+="disk/by-id/usb-$env{ID_USB_SERIAL}-$env{ID_PART_TABLE_UUID}"
KERNEL=="sd*|cciss*", ENV{DEVTYPE}=="partition", ENV{ID_USB_SERIAL}=="?*", SYMLINK+="disk/by-id/usb-$env{ID_USB_SERIAL}-part%n"
ENV{ID_USB_SERIAL_SHORT}=="DD56419883914", \
  SYMLINK+="disk/by-id/usb-$env{ID_USB_SERIAL}-$env{ID_PART_TABLE_UUID}-part%n"

then just restart your proxmox , and voila

Code:

  pool: R10
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Oct  3 00:03:33 2025
        815G / 815G scanned, 240G / 395G issued at 115M/s
        240G resilvered, 60.72% done, 00:23:05 to go
config:

        NAME                                                                               STATE     READ WRITE CKSUM
        R10                                                                                ONLINE       0     0     0
          mirror-0                                                                         ONLINE       0     0     0
            usb-JMicron_Tech_DD56419883914-0:0-43f4324e-b95f-3143-9d14-c888ab8ee75f-part1  ONLINE       0     0     0
            usb-JMicron_Tech_DD56419883914-0:0-part1                                       ONLINE       0     0     0
          mirror-1                                                                         ONLINE       0     0     0
            usb-JMicron_Tech_DD56419883914-0:0-e14497b8-7a25-7546-9c67-92fe6523971f        ONLINE       0     0     5
            usb-JMicron_Tech_DD56419883914-0:0-4ca728f7-0056-7c43-ba0e-0239af08b98c        ONLINE       0     0     0  (resilvering)

errors: Permanent errors have been detected in the following files:

Search

Search

ZFS defaults to /dev/disk/by-id

johnoathome

New Member

Impact

Renowned Member

gfngfn256

Distinguished Member

johnoathome

New Member

johnoathome

New Member

johnoathome

New Member

Impact

Renowned Member

gfngfn256

Distinguished Member

johnoathome

New Member

niko30127

New Member

We value your privacy