ZFS - Unavailable disk

[Consonant1022]

I just did a zpool replace action, on an unrelated disk using this command:

zpool replace -f nas /dev/disk/by-id/ata-ST14000NM0018-2H4101_ZHZ3VJCH /dev/disk/by-id/ata-ST14000NM000J-2TX103_ZR5BBTP8

The replace worked well, so I shut down the server to remove the bad drive and place the good drive into the correct bay. After bringing the proxmox node up, and running zpool status, I'm seeing a different error:

  pool: nas
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
    invalid.  Sufficient rplicas exist for the pool to continue
    functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: resilvered 8.12T in 22:10:01 with 0 errors on Fri May 16 08:32:55 2025
config:

    NAME                                   STATE     READ WRITE CKSUM
    nas                                    DEGRADED     0     0     0
      raidz1-0                             DEGRADED     0     0     0
        ata-ST14000NM0018-2H4101_ZHZ38XLQ  ONLINE       0     0     0
        1436561644858147629                UNAVAIL      0     0     0  was /dev/disk/by-id/ata-ST14000NM0018-2H4101_ZHZ67X6X-part1
        ata-ST14000NM000J-2TX103_ZR5BBTP8  ONLINE       0     0     0
        ata-ST14000NM0018-2H4101_ZHZ6HXSM  ONLINE       0     0     0
        ata-ST14000NM001G-2KJ103_ZTM0LSW0  ONLINE       0     0     0
        ata-ST14000NM0018-2H4101_ZHZ1D3P5  ONLINE       0     0     0

errors: No known data errors

The strange thing is when I run lsblk -S I see that disk just fine:

NAME HCTL       TYPE VENDOR   MODEL                 REV SERIAL       TRAN
sda  10:0:0:0   disk ATA      CT2000MX500SSD1      046  2404E89329B0 sas
sdb  10:0:1:0   disk ATA      CT2000MX500SSD1      046  2404E8932971 sas
sdc  10:0:2:0   disk ATA      ST14000NM000J-2TX103 SN02 ZR5BBTP8     sas
sdd  10:0:3:0   disk ATA      ST14000NM001G-2KJ103 SN04 ZTM0LSW0     sas
sde  10:0:4:0   disk ATA      ST14000NM0018-2H4101 SN04 ZHZ6HXSM     sas
sdf  10:0:5:0   disk ATA      ST14000NM0018-2H4101 SN04 ZHZ1D3P5     sas
sdg  10:0:6:0   disk ATA      ST14000NM0018-2H4101 SN04 ZHZ38XLQ     sas
sdh  10:0:7:0   disk ATA      ST14000NM0018-2H4101 SN04 ZHZ67X6X     sas

I will shut the server down and make sure I didn't bump any sata wires loose, but is there something else that went wrong here that I should check? Thanks!

 

[Consonant1022]

Shutdown the node, reset the sata cables and now it's showing the device correctly. I guess i knocked it a bit lose... Strange that I could see it with lsblk -S but the ZFS pool didn't see it.

 

[UdoB]

Shutdown the node, reset the sata cables and now it's showing the device correctly.

Great! You are a brave person to go with single redundancy with 14 TB devices.

Is it just for scratch / temporary data? If it contains actually valuable/precious data I would use RaidZ3 or maybe RaidZ2...

You know your pool is as slow as a single disk? IOPS-wise that is.

The usual recommendation is to add a "Special Device", build from some (more than one!) fast SSD/NVMe, mirrored. Its capacity can be 1% to 2% of the net data pool size. This would speed up the whole thing noticeable. Unfortunately that would only help for new data, written to the pool in the future --> it should be added before filling up the pool with data.

Caution: do not experiment with this, a Special Device CAN NOT¹ be removed once it is added.

Just saying..., of course ymmv

----
¹ I have just verified that you actually can remove a Special Device under specific circumstances: if you have set a checkpoint before and are willing to lose all later data:

Post in thread 'High I/O delay, fluctuating transfer rates'

May 18, 2025

Brutal changes like this one should always get tested in a sandbox before touching a productive environment.

This made me curious. Just to verify the actual behavior of a “checkpoint” in regard to changing the topology of a pool - especially by adding a "Special Device". I did this on a throw-away system of course:

root@pnz:~# zpool checkpoint rpool
root@pnz:~# zpool status
  pool: rpool
 state: ONLINE
checkpoint: created Sun May 18 10:08:56 2025, consumes 496K

root@pnz:~# zpool add rpool special /dev/sdb

root@pnz:~# zpool list -v...

• UdoB

• 

 

[Consonant1022]

Great! You are a brave person to go with single redundancy with 14 TB devices.

Is it just for scratch / temporary data? If it contains actually valuable/precious data I would use RaidZ3 or maybe RaidZ2...

You know your pool is as slow as a single disk? IOPS-wise that is.

The usual recommendation is to add a "Special Device", build from some (more than one!) fast SSD/NVMe, mirrored. Its capacity can be 1% to 2% of the net data pool size. This would speed up the whole thing noticeable. Unfortunately that would only help for new data, written to the pool in the future --> it should be added before filling up the pool with data.

Caution: do not experiment with this, a Special Device CAN NOT be removed once it is added.

Just saying..., of course ymmv

Yeah, it's not critical data, and it's all backed up to another Proxmox node regularly - as well as less regular offsite backups. The 'high availability' aspect of it is helpful, but not critical. I have the capability in this to add another drive, so I might do that in the future but for now, this serves my purpose.

Speed isn't too crucial with this either, but I'll look into a Special Device - interesting concept. At some point in the near future, I might rebuild this data store, when I do that, it could make sense to bring that into consideration. Thanks!