[SOLVED] ZFS I/O error

[Drag_and_Drop]

I currently have an annoying issue on my home server.

Every couple days my SSD ZFS pool failed with I/O errors

root@vmh1:/dev/disk/by-id# zpool status
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 0h8m with 0 errors on Sun Mar 12 00:32:17 2017
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdc2    ONLINE       0     0     0
            sdd2    ONLINE       0     0     0

errors: No known data errors

  pool: vmdisks_ssd_1
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-HC
  scan: scrub repaired 0 in 0h1m with 0 errors on Sun Mar 12 00:25:41 2017
config:

        NAME                                           STATE     READ WRITE CKSUM
        vmdisks_ssd_1                                  UNAVAIL      0     0     0  insufficient replicas
          ata-KINGSTON_SV300S37A120G_50026B723A03FA88  FAULTED      3     0     0  too many errors

root@vmh1:/dev/disk/by-id# ls
ata-KINGSTON_SV300S37A120G_50026B723A03FA88

But the drive is online and when I delete the pool, format the drive and recreate a pool on its healty for another couple of days.
I cant see any I/O errors in the kernel log, the SMART data are also ok and when I stress test the drive before I create a ZFS pool on it, its healthy as well (did a read write test with over 1 TB of data)

Also rm /etc/zfs/zpool.cache didn't helped

So:
1. Where can I look else why ZFS is failing?
2. is there a way to force ZFS to bring the array online?
3. Something else I could try?

My final workaround will be format the drive as btrfs or ext4 but if ZFS could do the job, I would prefer it

 

[Nemesiz]

You are using single disk for ZFS pool. Almost no protection.

Maybe your SSD is wearied out ? What smart is saying ?

Try to set copies from 1 to 2 or 3

#zfs set copies=2 vmdisks_ssd_1

It will protect you from bad blocks. Do it after you create fresh pool.

 

[Nemesiz]

Btw others file systems will not detect or protect you from data corruption.

 

[Drag_and_Drop]

root@vmh1:~# smartctl -a /dev/sde
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.4.44-1-pve] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: SandForce Driven SSDs
Device Model: KINGSTON SV300S37A120G
Serial Number: 50026B723A03FA88
LU WWN Device Id: 5 0026b7 23a03fa88
Firmware Version: 505ABBF1
User Capacity: 120,034,123,776 bytes [120 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS, ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Mar 19 14:06:16 2017 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 48) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0021) SCT Status supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 118 118 050 Pre-fail Always - 0/210880202
5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0
9 Power_On_Hours_and_Msec 0x0032 090 090 000 Old_age Always - 9596h+56m+36.500s
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 671
171 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
174 Unexpect_Power_Loss_Ct 0x0030 000 000 000 Old_age Offline - 112
177 Wear_Range_Delta 0x0000 000 000 000 Old_age Offline - 9
181 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
182 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
189 Airflow_Temperature_Cel 0x0000 024 065 000 Old_age Offline - 24 (Min/Max 10/65)
194 Temperature_Celsius 0x0022 024 065 000 Old_age Always - 24 (Min/Max 10/65)
195 ECC_Uncorr_Error_Count 0x001c 120 120 000 Old_age Offline - 0/210880202
196 Reallocated_Event_Count 0x0033 100 100 003 Pre-fail Always - 0
201 Unc_Soft_Read_Err_Rate 0x001c 120 120 000 Old_age Offline - 0/210880202
204 Soft_ECC_Correct_Rate 0x001c 120 120 000 Old_age Offline - 0/210880202
230 Life_Curve_Status 0x0013 100 100 000 Pre-fail Always - 100
231 SSD_Life_Left 0x0013 100 100 010 Pre-fail Always - 0
233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 25566
234 SandForce_Internal 0x0032 000 000 000 Old_age Always - 10216
241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 10216
242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 9862

SMART Error Log not supported

SMART Self-test Log not supported

SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Drag_and_Drop]

root@vmh1:~# dmesg |grep sde
[ 20.075665] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 21.058876] sd 0:1:0:0: [sdb] Synchronizing SCSI cache
[ 21.098298] sd 0:1:0:0: Attached scsi generic sg1 type 0
[ 21.099149] sd 0:1:0:0: [sde] 234441648 512-byte logical blocks: (120 GB/112 GiB)
[ 21.148124] sd 0:1:0:0: [sde] Write Protect is off
[ 21.148128] sd 0:1:0:0: [sde] Mode Sense: 46 00 10 08
[ 21.168117] sd 0:1:0:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 21.239242] sde: sde1 sde9
[ 21.318131] sd 0:1:0:0: [sde] Attached SCSI removable disk
[ 335.635348] WARNING: Pool 'vmdisks_ssd_1' has encountered an uncorrectable I/O failure and has been suspended.

so everything looks normal, except that ZFS is blocking my pool. And as I said, all other file systems are working normally. without any I/O errors or data lose

 

[Drag_and_Drop]

Ok thats random...

After an long SMART test:

smartctl -t long /dev/sde

where no errors are found, I'm able to clear the errors with:

zpool clear vmdisks_ssd_1

after that ZFS starts to scrub the pool and found 2 errors on metadata (and repaired them)

root@vmh1:~# zpool status -v
pool: vmdisks_ssd_1
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: scrub in progress since Sun Mar 19 18:01:31 2017
17.5G scanned out of 34.0G at 238M/s, 0h1m to go
0 repaired, 51.29% done
config:

NAME STATE READ WRITE CKSUM
vmdisks_ssd_1 ONLINE 0 0 0
ata-KINGSTON_SV300S37A120G_50026B723A03FA88 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

<metadata>:<0x0>
<metadata>:<0x20>

Now everything looks healthy and I can boot my VM again. Very random...

If anyone knows why it's working now, fell free to share your answer :-D
I think ZFS is somehow relaying on SMART data and there is maybe a bug or feature that you have to test your SSD sometimes

 

[Nemesiz]

ZFS keeps few copies of metadata in the pool. If it will repeats try to change cable or the port.

 

[Drag_and_Drop]

ZFS keeps few copies of metadata in the pool. If it will repeats try to change cable or the port.

But do you know why ZFS blocked the I/O while the drive was healthy and recognized by the OS? I/O errors are bad enough, but when you can't clear them, because ZFS is blocking the drive, its double bad...
I didn't do anything else than a long SMART test and it worked...

 

[Nemesiz]

ZFS is not blocking I/O it just reported the problem. ZFS stops sending data to application on error.

The problem can be in a lot of places like time out I/O from SSD, silence corruption and so on. Try to look at /var/log/syslog maybe you can find some information related to I/O errors.

 

[Drag_and_Drop]

For those who are interested:
I got finally the root cause! My Adaptec 72405 (24 port SAS controller), which I got used from Ebay, was set manually to: SGPIO backplane mode. I switched it to "Automatic" mode (Default) and now the backplane runs with the IBPI protocol. Now my pools are healthy for over 4 Weeks.
Sometimes it could be that easy