SSD Failing?

mendoncart · Apr 23, 2023

Hey guys! I need some help.

I have a proxmox install, currently using 2 x 2tb SSD. When installing proxmox I've divided each SSD in half (1tb+1tb), and I've used half of each and installed proxmox using a raidz1. The other half (2x1tb) I passed to truenas and create a raid1 vdev.

In summary I have:

SSD SDA:

1tb - raidz1: proxmox install
1tb - raid1: truenas pool.

SDA - OK???

SSD SDE:

1tb - raidz1: proxmox install
1tb - raid1: truenas pool.

SDE - The culprit? 26 pendind sectors...

My problem is, now when I'm making a backup of a vm or trying to restore a backup, or mostly anything that uses more the ssd, the whole proxmox and all vms crashes.

My question: since I don't know exactly where to look or how to solve this, is there any test that I can run to isolate the problem and be sure it's the ssd fault? The SSD seller have a 3 year warranty, and they asked me to format the driver. I don't think this will solve the pending sectors... Other question would be if can I replace one of the partitions used in raidz1 with a 1tb ssd. If it's possible, I'll replace this 2 x 2tb with 2 x 1tb ssd nvme.

Dunuin · Apr 23, 2023

I don't get how you partitioned your disks. Splitting two 2TB in half should give 4x 1TB partitions and a raidz1 requires at least 3 partitions and mirror at least 2 partitions.
What is the output of lsblk and zpool list?

mendoncart · Apr 24, 2023

Sorry it took so long to reply... I was outside my house, and apparently the scheduled backup ran and proxmox crashed... Now I'm at home and available to respond...

Dunuin said:
I don't get how you partitioned your disks. Splitting two 2TB in half should give 4x 1TB partitions and a raidz1 requires at least 3 partitions and mirror at least 2 partitions.

Now that I'm thinking about it, you are right. I didn't setup a raidz1, my mistake, it's just a raid1/mirror in proxmox, and the 2 x 1tb left over, I passed to Truenas and made another raid1...

What is the output of lsblk and zpool list?

Here are the outputs of the commands

Code:

lsblk

Code:

zpool list

Additional outputs:

I'm now trying to run a smartctl -t long to see what results will I get. I hope this test doesn't crash proxmox as well...

If anyone can suggest anything, please.

mendoncart · Apr 24, 2023

I just run the long smartctl test.
First I tried to run the short test, then the long one.
Apparently its reporting Interrupted (host reset), what does this mean? Is it related?

This is the complete command with feedback:

Code:

root@pve:~# smartctl -t long /dev/sde
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.104-1-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 10 minutes for test to complete.
Test will complete after Sun Apr 23 22:24:45 2023 -03
Use smartctl -X to abort test.
root@pve:~# smartctl -a /dev/sde
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.104-1-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     SSD 2TB
Serial Number:    AA000000000000000873
Firmware Version: T0317A0
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Apr 23 22:26:41 2023 -03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  41) The self-test routine was interrupted
                                        by the host with a hard or soft reset.
Total time to complete Offline
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
                                        entering power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       26
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       14552
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       115
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       0
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       43
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       18
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       391427
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       3475
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       553
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       869
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1500
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       43
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       26
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       77
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       45
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       99719280
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       26
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       0
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       43
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       4097792
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       1483455
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       35655331

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 1

ATA Error Count: 0
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 00 00 00 00 00 00

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d0 01 00 4f c2 00 08      00:00:00.000  SMART READ DATA
  b0 d1 01 01 4f c2 00 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 da 00 00 4f c2 00 08      00:00:00.000  SMART RETURN STATUS
  b0 d5 01 00 4f c2 00 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 00 08      00:00:00.000  SMART READ LOG

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Interrupted (host reset)      90%     14552         -
# 2  Short offline       Completed without error       00%     14552         -
# 3  Short offline       Interrupted (host reset)      90%     14551         -
# 4  Extended offline    Interrupted (host reset)      90%     12257         -
# 5  Short offline       Completed without error       00%        14         -

Selective Self-tests/Logging not supported

leesteken · Apr 24, 2023

mendoncart said:
I just run the long smartctl test.
First I tried to run the short test, then the long one.
Apparently its reporting Interrupted (host reset), what does this mean? Is it related?

The test can take a long time and it appears that you restarted the system before it was finished. Maybe try again and wait longer?

mendoncart said:
=== START OF INFORMATION SECTION ===
Device Model: SSD 2TB
Serial Number: AA000000000000000873
Firmware Version: T0317A0

This does not look like a reputable brand. What kind of flash memory does it use? QLC will slow down so much during writes that ZFS will mark it as an error. If both drives are like this, then Proxmox will lose both parts of the mirror after ZFS decided it has seen too many write errors (and it cannot log this because it has no drives anymore).

mendoncart said:
Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
00 00 00 00 00 00 00

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
b0 d0 01 00 4f c2 00 08 00:00:00.000 SMART READ DATA
b0 d1 01 01 4f c2 00 08 00:00:00.000 SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
b0 da 00 00 4f c2 00 08 00:00:00.000 SMART RETURN STATUS
b0 d5 01 00 4f c2 00 08 00:00:00.000 SMART READ LOG
b0 d5 01 01 4f c2 00 08 00:00:00.000 SMART READ LOG

Looks like the drive itself also noticed it is having read errors.

mendoncart · Apr 24, 2023

leesteken said:
The test can take a long time and it appears that you restarted the system before it was finished. Maybe try again and wait longer?

No, I didn't restarted the system. In my research I've read that this interruption may be caused for some other process using the drive, so proxmox abort the test. I paused all my vms and ran the tests again, and It was sucessful in both ssds. I'll post results bellow, due to character limit...

leesteken said:
This does not look like a reputable brand.

Well, you're right, it isn't. But it was working fine. Now it's crashing whenever I ran a backup, or try to restore a backup. This didn't happen before.

leesteken said:
QLC will slow down so much during writes that ZFS will mark it as an error. If both drives are like this, then Proxmox will lose both parts of the mirror after ZFS decided it has seen too many write errors (and it cannot log this because it has no drives anymore).

Zpool appears to be healthy... SMART tests are "OK"... I need someway to isolate the problem...

I've recently upgraded Truenas. Truenas uses the other half of the ssds in a raid1 configuration.
Is it possible that this is causing some conflicts now?
Is there a way to get some logs when the proxmox crashes?

mendoncart · Apr 24, 2023

Long smart test completed,

SDA:

Code:

root@pve:~# smartctl -a /dev/sda

smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.104-1-pve] (local build)

Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org



=== START OF INFORMATION SECTION ===

Device Model:     SSD 2TB

Serial Number:    AA000000000000000746

Firmware Version: S1128B0

User Capacity:    2,000,398,934,016 bytes [2.00 TB]

Sector Size:      512 bytes logical/physical

Rotation Rate:    Solid State Device

Form Factor:      2.5 inches

TRIM Command:     Available

Device is:        Not in smartctl database [for details use: -P showall]

ATA Version is:   ACS-3 T13/2161-D revision 4

SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)

Local Time is:    Mon Apr 24 10:47:11 2023 -03

SMART support is: Available - device has SMART capability.

SMART support is: Enabled



=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED



General SMART Values:

Offline data collection status:  (0x02) Offline data collection activity

                                        was completed without error.

                                        Auto Offline Data Collection: Disabled.

Self-test execution status:      (   0) The previous self-test routine completed

                                        without error or no self-test has ever

                                        been run.

Total time to complete Offline

data collection:                (  120) seconds.

Offline data collection

capabilities:                    (0x11) SMART execute Offline immediate.

                                        No Auto Offline data collection support.

                                        Suspend Offline collection upon new

                                        command.

                                        No Offline surface scan supported.

                                        Self-test supported.

                                        No Conveyance Self-test supported.

                                        No Selective Self-test supported.

SMART capabilities:            (0x0002) Does not save SMART data before

                                        entering power-saving mode.

                                        Supports SMART auto save timer.

Error logging capability:        (0x01) Error logging supported.

                                        General Purpose Logging supported.

Short self-test routine

recommended polling time:        (   2) minutes.

Extended self-test routine

recommended polling time:        (  10) minutes.



SMART Attributes Data Structure revision number: 1

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0

  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       11126

 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       118

160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       0

161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       100

163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       15

164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       334327

165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1721

166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       318

167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       464

168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1500

169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       70

175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0

176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0

177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0

178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       0

181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0

182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0

192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       84

194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       40

195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       4007348

196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       0

199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       2

232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       100

241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       2256121

242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       754311

245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       23830196



SMART Error Log Version: 1

Warning: ATA error count 0 inconsistent with error log pointer 3



ATA Error Count: 0

        CR = Command Register [HEX]

        FR = Features Register [HEX]

        SC = Sector Count Register [HEX]

        SN = Sector Number Register [HEX]

        CL = Cylinder Low Register [HEX]

        CH = Cylinder High Register [HEX]

        DH = Device/Head Register [HEX]

        DC = Device Command Register [HEX]

        ER = Error register [HEX]

        ST = Status register [HEX]

Powered_Up_Time is measured from power on, and printed as

DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,

SS=sec, and sss=millisec. It "wraps" after 49.710 days.



Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)

  When the command that caused the error occurred, the device was active or idle.



  After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  00 00 00 00 00 00 00



  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  b0 d0 01 00 4f c2 00 08      00:00:00.000  SMART READ DATA

  b0 d1 01 01 4f c2 00 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]

  b0 da 00 00 4f c2 00 08      00:00:00.000  SMART RETURN STATUS

  b0 d5 01 00 4f c2 00 08      00:00:00.000  SMART READ LOG

  b0 d5 01 01 4f c2 00 08      00:00:00.000  SMART READ LOG



SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Extended offline    Completed without error       00%     11125         -

# 2  Extended offline    Interrupted (host reset)      90%     11125         -

# 3  Extended offline    Interrupted (host reset)      90%     11114         -

# 4  Short offline       Completed without error       00%     11114         -



Selective Self-tests/Logging not supported

SDE:

Code:

root@pve:~# smartctl -a /dev/sde

smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.104-1-pve] (local build)

Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org



=== START OF INFORMATION SECTION ===

Device Model:     SSD 2TB

Serial Number:    AA000000000000000873

Firmware Version: T0317A0

User Capacity:    2,000,398,934,016 bytes [2.00 TB]

Sector Size:      512 bytes logical/physical

Rotation Rate:    Solid State Device

Form Factor:      2.5 inches

TRIM Command:     Available

Device is:        Not in smartctl database [for details use: -P showall]

ATA Version is:   ACS-3 T13/2161-D revision 4

SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)

Local Time is:    Mon Apr 24 10:48:24 2023 -03

SMART support is: Available - device has SMART capability.

SMART support is: Enabled



=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED



General SMART Values:

Offline data collection status:  (0x02) Offline data collection activity

                                        was completed without error.

                                        Auto Offline Data Collection: Disabled.

Self-test execution status:      (   0) The previous self-test routine completed

                                        without error or no self-test has ever

                                        been run.

Total time to complete Offline

data collection:                (  120) seconds.

Offline data collection

capabilities:                    (0x11) SMART execute Offline immediate.

                                        No Auto Offline data collection support.

                                        Suspend Offline collection upon new

                                        command.

                                        No Offline surface scan supported.

                                        Self-test supported.

                                        No Conveyance Self-test supported.

                                        No Selective Self-test supported.

SMART capabilities:            (0x0002) Does not save SMART data before

                                        entering power-saving mode.

                                        Supports SMART auto save timer.

Error logging capability:        (0x01) Error logging supported.

                                        General Purpose Logging supported.

Short self-test routine

recommended polling time:        (   2) minutes.

Extended self-test routine

recommended polling time:        (  10) minutes.



SMART Attributes Data Structure revision number: 1

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0

  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       26

  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       14564

 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       115

160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       0

161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       43

163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       18

164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       391720

165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       3475

166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       553

167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       870

168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1500

169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       42

175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0

176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0

177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0

178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       26

181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0

182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0

192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       77

194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       45

195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       99719280

196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       26

198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       0

199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       0

232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       43

241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       4098527

242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       1483636

245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       35691907



SMART Error Log Version: 1

Warning: ATA error count 0 inconsistent with error log pointer 1



ATA Error Count: 0

        CR = Command Register [HEX]

        FR = Features Register [HEX]

        SC = Sector Count Register [HEX]

        SN = Sector Number Register [HEX]

        CL = Cylinder Low Register [HEX]

        CH = Cylinder High Register [HEX]

        DH = Device/Head Register [HEX]

        DC = Device Command Register [HEX]

        ER = Error register [HEX]

        ST = Status register [HEX]

Powered_Up_Time is measured from power on, and printed as

DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,

SS=sec, and sss=millisec. It "wraps" after 49.710 days.



Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)

  When the command that caused the error occurred, the device was active or idle.



  After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  00 00 00 00 00 00 00



  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  b0 d0 01 00 4f c2 00 08      00:00:00.000  SMART READ DATA

  b0 d1 01 01 4f c2 00 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]

  b0 da 00 00 4f c2 00 08      00:00:00.000  SMART RETURN STATUS

  b0 d5 01 00 4f c2 00 08      00:00:00.000  SMART READ LOG

  b0 d5 01 01 4f c2 00 08      00:00:00.000  SMART READ LOG



SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Extended offline    Completed without error       00%     14563         -

# 2  Extended offline    Interrupted (host reset)      90%     14552         -

# 3  Short offline       Completed without error       00%     14552         -

# 4  Short offline       Interrupted (host reset)      90%     14551         -

# 5  Extended offline    Interrupted (host reset)      90%     12257         -

# 6  Short offline       Completed without error       00%        14         -



Selective Self-tests/Logging not supported

mendoncart · Apr 24, 2023

This is trying to restore an backup.
When the progress hits around 1%&, the connection goes offline, proxmox web manager, proxmox ssh, vms services... All stop responding and becomes inaccessible.

Syslog right until desconnects:

Code:

Apr 24 15:07:01 pve pvedaemon[1993135]: <root@pam> starting task UPID:pve:0020A3C7:0060E815:6446C545:qmrestore:102:root@pam:
Apr 24 15:07:02 pve kernel: [63511.469866] debugfs: Directory 'zd16' with parent 'block' already present!
Apr 24 15:07:26 pve pveproxy[2055883]: worker exit
Apr 24 15:07:26 pve pveproxy[4714]: worker 2055883 finished
Apr 24 15:07:26 pve pveproxy[4714]: starting 1 worker(s)
Apr 24 15:07:26 pve pveproxy[4714]: worker 2140710 started
Apr 24 15:07:58 pve pvedaemon[2092056]: <root@pam> successful auth for user 'root@pam'
Apr 24 15:08:56 pve postfix/qmgr[3341]: 5F362208AD: from=<root@pve.rtmhouse>, size=944, nrcpt=1 (queue active)
Apr 24 15:08:56 pve postfix/qmgr[3341]: 5DBDA20CA4: from=<root@pve.rtmhouse>, size=944, nrcpt=1 (queue active)
Apr 24 15:08:56 pve postfix/qmgr[3341]: 58D4D208BD: from=<root@pve.rtmhouse>, size=944, nrcpt=1 (queue active)
Apr 24 15:08:56 pve postfix/qmgr[3341]: 502142093A: from=<root@pve.rtmhouse>, size=944, nrcpt=1 (queue active)
Apr 24 15:08:56 pve postfix/qmgr[3341]: 16FD221D43: from=<>, size=2966, nrcpt=1 (queue active)
Apr 24 15:08:56 pve postfix/qmgr[3341]: 4A44A2078C: from=<root@pve.rtmhouse>, size=773, nrcpt=1 (queue active)
Apr 24 15:08:56 pve local[2147535]: fatal: execvp /usr/bin/pvemailforward: No such file or directory
Apr 24 15:08:56 pve local[2147537]: fatal: execvp /usr/bin/pvemailforward: No such file or directory
Apr 24 15:08:56 pve postfix/local[2147533]: 5F362208AD: to=<root@pve.rtmhouse>, orig_to=<root>, relay=local, delay=141731, delays=141730/0.02/0/0.49, dsn=4.3.0, status=deferred (temporary failure. Command output: local: fatal: execvp /usr/bin/pvemailforward: No such file or directory )
Apr 24 15:08:56 pve local[2147566]: fatal: execvp /usr/bin/pvemailforward: No such file or directory
Apr 24 15:08:56 pve postfix/local[2147534]: 5DBDA20CA4: to=<root@pve.rtmhouse>, orig_to=<root>, relay=local, delay=147131, delays=147130/0.01/0/0.6, dsn=4.3.0, status=deferred (temporary failure. Command output: local: fatal: execvp /usr/bin/pvemailforward: No such file or directory )
Apr 24 15:08:57 pve local[2147573]: fatal: execvp /usr/bin/pvemailforward: No such file or directory
Apr 24 15:08:57 pve postfix/local[2147533]: 58D4D208BD: to=<root@pve.rtmhouse>, orig_to=<root>, relay=local, delay=205629, delays=205628/0.49/0/0.53, dsn=4.3.0, status=deferred (temporary failure. Command output: local: fatal: execvp /usr/bin/pvemailforward: No such file or directory )
Apr 24 15:08:57 pve local[2147584]: fatal: execvp /usr/bin/pvemailforward: No such file or directory
Apr 24 15:08:57 pve postfix/local[2147534]: 502142093A: to=<root@pve.rtmhouse>, orig_to=<root>, relay=local, delay=94031, delays=94030/0.61/0/0.58, dsn=4.3.0, status=deferred (temporary failure. Command output: local: fatal: execvp /usr/bin/pvemailforward: No such file or directory )
Apr 24 15:08:57 pve local[2147601]: fatal: execvp /usr/bin/pvemailforward: No such file or directory
Apr 24 15:08:57 pve postfix/local[2147533]: 16FD221D43: to=<root@pve.rtmhouse>, relay=local, delay=13202, delays=13201/1/0/0.4, dsn=4.3.0, status=deferred (temporary failure. Command output: local: fatal: execvp /usr/bin/pvemailforward: No such file or directory )
Apr 24 15:08:57 pve postfix/local[2147534]: 4A44A2078C: to=<root@pve.rtmhouse>, orig_to=<root>, relay=local, delay=63481, delays=63480/1.2/0/0.23, dsn=4.3.0, status=deferred (temporary failure. Command output: local: fatal: execvp /usr/bin/pvemailforward: No such file or directory )
Apr 24 15:09:52 pve kernel: [63681.265424] INFO: task txg_sync:923 blocked for more than 120 seconds.
Apr 24 15:09:52 pve kernel: [63681.267126]       Tainted: P           O      5.15.104-1-pve #1
Apr 24 15:09:52 pve kernel: [63681.268805] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 24 15:09:52 pve kernel: [63681.270457] task:txg_sync        state:D stack:    0 pid:  923 ppid:     2 flags:0x00004000
Apr 24 15:09:52 pve kernel: [63681.272072] Call Trace:
Apr 24 15:09:52 pve kernel: [63681.273645]  <TASK>
Apr 24 15:09:52 pve kernel: [63681.275179]  __schedule+0x34e/0x1740
Apr 24 15:09:52 pve kernel: [63681.276688]  ? lock_timer_base+0x3b/0xd0
Apr 24 15:09:52 pve kernel: [63681.278173]  ? __mod_timer+0x271/0x440
Apr 24 15:09:52 pve kernel: [63681.279631]  schedule+0x69/0x110
Apr 24 15:09:52 pve kernel: [63681.281041]  schedule_timeout+0x87/0x140
Apr 24 15:09:52 pve kernel: [63681.282425]  ? __bpf_trace_tick_stop+0x20/0x20
Apr 24 15:09:52 pve kernel: [63681.283789]  io_schedule_timeout+0x51/0x80
Apr 24 15:09:52 pve kernel: [63681.285176]  __cv_timedwait_common+0x135/0x170 [spl]
Apr 24 15:09:52 pve kernel: [63681.286535]  ? wait_woken+0x70/0x70
Apr 24 15:09:52 pve kernel: [63681.287871]  __cv_timedwait_io+0x19/0x20 [spl]
Apr 24 15:09:52 pve kernel: [63681.289201]  zio_wait+0x137/0x300 [zfs]
Apr 24 15:09:52 pve kernel: [63681.290665]  ? __cond_resched+0x1a/0x50
Apr 24 15:09:52 pve kernel: [63681.291983]  dsl_pool_sync+0xcc/0x4f0 [zfs]
Apr 24 15:09:52 pve kernel: [63681.293415]  ? spa_suspend_async_destroy+0x60/0x60 [zfs]
Apr 24 15:09:52 pve kernel: [63681.294811]  ? add_timer+0x20/0x30
Apr 24 15:09:52 pve kernel: [63681.296067]  spa_sync+0x55a/0x1020 [zfs]
Apr 24 15:09:52 pve kernel: [63681.297408]  ? spa_txg_history_init_io+0x10a/0x120 [zfs]
Apr 24 15:09:52 pve kernel: [63681.298722]  txg_sync_thread+0x278/0x400 [zfs]
Apr 24 15:09:52 pve kernel: [63681.300004]  ? txg_init+0x2c0/0x2c0 [zfs]
Apr 24 15:09:52 pve kernel: [63681.301256]  thread_generic_wrapper+0x64/0x80 [spl]
Apr 24 15:09:52 pve kernel: [63681.302394]  ? __thread_exit+0x20/0x20 [spl]
Apr 24 15:09:52 pve kernel: [63681.303501]  kthread+0x12a/0x150
Apr 24 15:09:52 pve kernel: [63681.304571]  ? set_kthread_struct+0x50/0x50
Apr 24 15:09:52 pve kernel: [63681.305615]  ret_from_fork+0x22/0x30
Apr 24 15:09:52 pve kernel: [63681.306652]  </TASK>
Apr 24 15:10:37 pve pvestatd[3877]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - got timeout
Apr 24 15:10:38 pve pvestatd[3877]: status update time (8.109 seconds)

Session was closed

This is the monitor connected to the server:

About 20 minutes later, proxmox web manager, ssh, and other vms came back online.
What could it be?

leesteken · Apr 24, 2023

Looks like the drives are slowing down because of the writes and blocking I/O of other processes. Please search the forum about QLC drives.

DST73 · Oct 28, 2023

Did you have any luck? Did you find a ssd to replace?

Search

Search

SSD Failing?

mendoncart

New Member

Dunuin

Distinguished Member

mendoncart

New Member

mendoncart

New Member

leesteken

Distinguished Member

mendoncart

New Member

mendoncart

New Member

mendoncart

New Member

leesteken

Distinguished Member

DST73

Member