Two of my nodes, both of which are storing the OS and VMs on two-year-old 500GB Samsung EVO 870, have begun failing every action that requires reading of an entire virtual disk. The console reports only "Error: migration aborted" or similar, with the output log reporting "dd: error reading '/dev/pve/vm-100-disk-0': Input/output error"
The journal for an example failed migration attempt looks like this:
Based on some previous posts in this forum, this seems to indicate an issue with the disk itself. This is an SSD with only 1% wearout, so this gives me significant cause for concern. The SMART readout in the Proxmox console indicates that the disk passed, but the output of smartctl -a tells a different story:
This leads me to believe that there are issues with reading the disk but, paradoxically, all of the VMs and containers on these nodes appear to be working without issue.
With all this in mind, my questions for the community are this:
1 - does anyone know a method to migrate or clone a VM from a disk with the issues that mine are reporting? Thus far I have tried Proxmox's own backup, Proxmox Backup Server, and Veeam. All fail after reading approximately 6% of the VM disks.
2 - has anyone experienced a similar issue and successfully managed to repair the disk/PVE instance without needing to move or rebuild/restore the VMs from an earlier backup?
The journal for an example failed migration attempt looks like this:
Code:
EXT4-fs warning (device dm-6): ext4_end_bio:344: I/O error 10 writing to inode 148212 starting block 765955)
Buffer I/O error on device dm-6, logical block 765955
ata2.00: Enabling discard_zeroes_data
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: irq_stat 0x40000001
ata2.00: failed command: READ DMA
ata2.00: cmd c8/00:80:80:4a:27/00:00:00:00:00/ed tag 22 dma 65536 in
res 51/40:00:c8:4a:27/00:00:00:00:00/ed Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: supports DRM functions and may not be fully accessible
ata2.00: supports DRM functions and may not be fully accessible
ata2.00: configured for UDMA/133
sd 1:0:0:0: [sda] tag#22 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
sd 1:0:0:0: [sda] tag#22 Sense Key : Medium Error [current]
sd 1:0:0:0: [sda] tag#22 Add. Sense: Unrecovered read error - auto reallocate failed
sd 1:0:0:0: [sda] tag#22 CDB: Read(10) 28 00 0d 27 4a 80 00 00 80 00
blk_update_request: I/O error, dev sda, sector 220678856 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
ata2: EH completeBased on some previous posts in this forum, this seems to indicate an issue with the disk itself. This is an SSD with only 1% wearout, so this gives me significant cause for concern. The SMART readout in the Proxmox console indicates that the disk passed, but the output of smartctl -a tells a different story:
Code:
=== START OF INFORMATION SECTION ===
Device Model: Samsung SSD 870 EVO 500GB
Serial Number: S62ANJ0RA21514A
LU WWN Device Id: 5 002538 f31a1fd49
Firmware Version: SVT01B6Q
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Fri Dec 15 13:26:53 2023 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 85) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 059 059 010 Pre-fail Always - 224
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 16868
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 28
177 Wear_Leveling_Count 0x0013 099 099 000 Pre-fail Always - 21
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 059 059 010 Pre-fail Always - 224
181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0
183 Runtime_Bad_Block 0x0013 059 059 010 Pre-fail Always - 224
187 Reported_Uncorrect 0x0032 092 092 000 Old_age Always - 80299
190 Airflow_Temperature_Cel 0x0032 060 045 000 Old_age Always - 40
195 Hardware_ECC_Recovered 0x001a 199 199 000 Old_age Always - 80299
199 UDMA_CRC_Error_Count 0x003e 099 099 000 Old_age Always - 9
235 Unknown_Attribute 0x0012 099 099 000 Old_age Always - 19
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 18701120240
SMART Error Log Version: 1
Warning: ATA error count 65535 inconsistent with error log pointer 4
ATA Error Count: 65535 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 65535 occurred at disk power-on lifetime: 16868 hours (702 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 80 80 4a 27 ed Error: UNC 128 sectors at LBA = 0x0d274a80 = 220678784
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 80 80 4a 27 ed 15 2d+04:32:52.715 READ DMA
ca 00 08 f8 2c aa ed 15 2d+04:32:52.715 WRITE DMA
ca 00 08 88 2d aa ed 15 2d+04:32:52.715 WRITE DMA
ca 00 40 20 d7 aa ed 15 2d+04:32:52.715 WRITE DMA
ca 00 05 08 19 ec ed 15 2d+04:32:52.715 WRITE DMA
Error 65534 occurred at disk power-on lifetime: 16868 hours (702 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 80 80 4a 27 ed Error: UNC 128 sectors at LBA = 0x0d274a80 = 220678784
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 80 80 4a 27 ed 15 2d+04:31:52.762 READ DMA
ca 00 80 98 c0 fe ed 15 2d+04:31:52.762 WRITE DMA
ca 00 08 58 e0 8c ed 15 2d+04:31:52.762 WRITE DMA
ca 00 40 18 e0 8c ed 15 2d+04:31:52.762 WRITE DMA
ca 00 08 70 ac 1a ed 15 2d+04:31:52.762 WRITE DMA
Error 65533 occurred at disk power-on lifetime: 16868 hours (702 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 80 80 4a 27 ed Error: UNC 128 sectors at LBA = 0x0d274a80 = 220678784
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 80 80 4a 27 ed 15 2d+04:30:52.707 READ DMA
ca 00 08 38 5c 24 ed 15 2d+04:30:52.707 WRITE DMA
ca 00 08 58 10 d0 ed 15 2d+04:30:52.707 WRITE DMA
ca 00 08 d8 4e 94 ed 15 2d+04:30:52.707 WRITE DMA
ca 00 08 e0 21 94 ed 15 2d+04:30:52.707 WRITE DMA
Error 65532 occurred at disk power-on lifetime: 16868 hours (702 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 80 80 4a 27 ed Error: UNC 128 sectors at LBA = 0x0d274a80 = 220678784
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 80 80 4a 27 ed 15 2d+04:29:52.754 READ DMA
ca 00 18 c0 1e 94 ed 15 2d+04:29:52.754 WRITE DMA
ca 00 18 a0 14 08 ed 15 2d+04:29:52.754 WRITE DMA
ca 00 18 60 d0 b0 ed 15 2d+04:29:52.754 WRITE DMA
ca 00 08 10 2a 95 ed 15 2d+04:29:52.754 WRITE DMA
Error 65531 occurred at disk power-on lifetime: 16868 hours (702 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 80 80 4a 27 ed Error: UNC 128 sectors at LBA = 0x0d274a80 = 220678784
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 80 80 4a 27 ed 15 2d+04:28:52.697 READ DMA
ca 00 80 98 bf fe ed 15 2d+04:28:52.697 WRITE DMA
ca 00 08 b8 a3 1a ed 15 2d+04:28:52.697 WRITE DMA
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 16624 20208144
# 2 Short offline Completed: read failure 60% 16624 20208144
# 3 Short offline Completed: read failure 60% 16577 20208144
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
256 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.This leads me to believe that there are issues with reading the disk but, paradoxically, all of the VMs and containers on these nodes appear to be working without issue.
With all this in mind, my questions for the community are this:
1 - does anyone know a method to migrate or clone a VM from a disk with the issues that mine are reporting? Thus far I have tried Proxmox's own backup, Proxmox Backup Server, and Veeam. All fail after reading approximately 6% of the VM disks.
2 - has anyone experienced a similar issue and successfully managed to repair the disk/PVE instance without needing to move or rebuild/restore the VMs from an earlier backup?