smart and rpool errors

[sidoni]

Hi,

I got some S.M.A.R.T. errors by email from my proxmox:

The following warning/error was logged by the smartd daemon:

Device: /dev/sdd [SAT], 1 Offline uncorrectable sectors

Device info:
Hitachi HUA723020ALA641, S/N:YGGNVL1A, WWN:5-000cca-224c97b35, FW:MK7OA840, 2.00 TB

So I ran smartctl -a:

smartctl -a /dev/sdd
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.35-2-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Hitachi Ultrastar 7K3000
Device Model: Hitachi HUA723020ALA641
Serial Number: YGGNVL1A
LU WWN Device Id: 5 000cca 224c97b35
Firmware Version: MK7OA840
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jan 30 23:55:17 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (20448) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 341) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 133 133 054 Pre-fail Offline - 90
3 Spin_Up_Time 0x0007 100 100 024 Pre-fail Always - 500
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 6
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 18
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 120 120 020 Pre-fail Offline - 32
9 Power_On_Hours 0x0012 098 098 000 Old_age Always - 14906
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 6
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 441
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 441
194 Temperature_Celsius 0x0002 214 214 000 Old_age Always - 28 (Min/Max 20/42)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 18
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 6
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 34 -
# 2 Short offline Completed without error 00% 9314 -
# 3 Short offline Completed without error 00% 7226 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Should I replace this 'sdd' harddisk? If yes, what is the easiest way to do it?

Last but not least, I ran zpool status:

zpool status
pool: BCK_VM
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 01:06:45 with 0 errors on Sun Jan 8 01:30:46 2023
config:

NAME STATE READ WRITE CKSUM
BCK_VM ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 0B in 00:02:46 with 0 errors on Sun Apr 10 00:26:49 2022
config:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-KINGSTON_SV300S37A480G_50026B775B085DBA-part3 DEGRADED 0 0 0 too many errors
ata-KINGSTON_SV300S37A480G_50026B775B085DCA-part3 ONLINE 0 0 0

errors: No known data errors

Damned, it seems there is another disk (ssd this time) dying!

Should I replace it too? If yes, what is the easiest way to do it?

Thanks in advance for your kind help

 

[sidoni]

Hi,
Is there any additional information you would need to be able to help me? :- )

 

[tuxis]

If you like your data, the answer to 'should I replace my disk is pretty simple'; Yes.

As for your second question, see https://pve.proxmox.com/wiki/ZFS_on_Linux and look for "changing a failed device". Make sure you have backups before you start playing around with ZFS for the first time.