Crash Opinion

[Smash_Override]

Soooooooooooooooo what are the chances that an overheated external USB HDD would cause a crash? Below is the error message that was on my command line console (not sure what that screen is called?) and the logs that go with it. The drive is a cheap 12 TB desktop drive that I had laying on it's side. It was very hot to the touch. I powered it down and turned it upright to cool off, then rebooted the machine. SmartCTL was showing 113 worst value but not sure if that is accurate. RAW VALUE show 50 (Min/Max 18/63). The last line in the logs is a REALLY hot number for a spinny rust SATA (usb External) drive according to the interwebs searches. The crash happens every couple of days and it appears to be happening just before 4 AM. I have a backup of all things scheduled once a week at 1AM and that doesn't take very long (fast and good), so I don't suspicion that being an issue. The logs don't show anything crazy happening at crash time anyways. This is just my home lab running plex on a Fedora VM so no biggy. Just my learning machine.

Thoughts?

PVE login: [131760.947960] ext-fs error (device dm-1) in ext4_convert_unwritten_io_end_vec:4892: IO failure
[131760.948077] EXT4-fs (dm-1): Remounting filesystem read-only
[131760.948674] EXT4-fs (dm-1): ext4_do_writepages: jbd2: 989 pages, ino 788973; err -5
[131760.948683] EXT4-fs (dm-1): ext4_do_writepages: jbd2: 9223372036854775744 pages, ino 788973; err -5

Last line of log: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 107 to 109


Last 24 hours of logs and CLI error attached.

 

[l.leahu-vladucu]

Hello Smash_Override! Can you please post the output of the command smartctl -A /dev/sdb ? Can you please do the same for your main system drive?

I'm not sure if the reported temperature is in degrees Celsius, though.

What are you using the external drive for? I would not expect your whole system to freeze / restart unless you have other issues as well, e.g. with your main disk.

 

[Maximiliano]

Hello,

Smartd attributes are just raw values reported by the disk in the 0..255 range, they do not necessarily report the temperature in any particular unit or even whether lower values are better. These values are vendor/device specific. There are normalized values (in the 0..100 range, where lower is better) that might be more helpful though.

It is possible to run into issues if the temperature is too high, but with an external USB disks it could be many other things, e.g. a bad cable or a bad controller.

 

[Smash_Override]

Hello Smash_Override! Can you please post the output of the command smartctl -A /dev/sdb ? Can you please do the same for your main system drive?

I'm not sure if the reported temperature is in degrees Celsius, though.

What are you using the external drive for? I would not expect your whole system to freeze / restart unless you have other issues as well, e.g. with your main disk.

Reports attached as requested. Apologies for the delayed response. As I am sure you can tell this is just a home lab that I am learning on. I have left the sdb USB drive turned off for the last couple of days seeing show the problem persisted after turn the drive upright. So far so good.

Thoughts?

 

[l.leahu-vladucu]

Thanks for the logs! The NVMe drive where you installed PVE doesn't show any temperature-related warnings.

An IO failure can happen due to multiple reasons, including bad cables and connections, or a bad drive. However, since you mentioned that it's an M.2 drive (probably directly attached to the motherboard?), the only thing you can do is to try another M.2 slot on your motherboard (if possible) to see if the issues go away. If not, you might want to try to do some read/write tests to your NVMe SSD, because the drive itself might have issues.

My guess at this point is that it's most probably the drive, but let us know what you find out!

 

[Smash_Override]

I'll give it a couple of weeks and come back to this post unless it dies on me before then. If the problem goes away, then we know it's directly associated with the USB connection. The cable looks fine, but obviously that's just superficial. I'm going to put my bet on overheating external drive, but I'll take a temp reading of the NVME stick with an IR temp reader also.

Whats the temp of an M.2 supposed to run at?

 

[Smash_Override]

I'll give it a couple of weeks and come back to this post unless it dies on me before then. If the problem goes away, then we know it's directly associated with the USB connection. The cable looks fine, but obviously that's just superficial. I'm going to put my bet on overheating external drive, but I'll take a temp reading of the NVME stick with an IR temp reader also.

Whats the temp of an M.2 supposed to run at anyways?

 

[Smash_Override]

Welp, it crashed again. The USB was turned off but still plugged in, but there goes that idea. I took a thermal reading of the M.2 chip and it was at 109 F at the hottest point closest to the socket. Attached is an image of the error message.

Thoughts?

 

[l.leahu-vladucu]

The temperature is fine, so there's nothing to be worried about.

Some things you can try:

1. If available, try a different M.2 slot on your motherboard.
2. Try updating the firmware of the NVMe drive (if available / possible).
3. Try installing the latest BIOS update (although I somehow doubt this will fix your issues, but it's probably worth trying before replacing the whole disk).
4. Try using a different disk. While I hope that some of the previous tests will fix your issues, I'm afraid that you might have to replace your drive entirely due to it being faulty.

 

[Smash_Override]

Thanks for the suggestions. I suspect and fear my M.2 is faulty also. I will do all that you suggested.

Something else that I am going to do is run GRC's SpinRight 6.1 on it. I listen to a podcast called "Security Now" where they go over all kinds of cyber news and events around the world. Steve Gibson (host) writes all kinds of software and is known for Spin Right. A tool originally designed to scan and recover bad sectors on spinning rust. He told a story about his wife's Windows laptop that started dragging ass on boot, so for grins he ran spin right on it. Not expecting it to really work seeings how spin right was designed for spinning rust, it worked. I found a FAQ about it on his site https://www.grc.com/sr/faq.htm. See below.

During the development of v6.1, many of SpinRite's testers discovered something quite unexpected: The front regions of SSDs were performing much more slowly than the other regions of those drives. A well-known problem with solid state storage known as “read disturb” provides the answer. The front of drives contain the operating system files which are infrequently written after being initially stored. Over time with repeated reading, the well-known problem of read disturbance can cause a dramatic reduction in read performance as SSDs controllers struggle to determine the proper value of “softened” data bits. The solution for this is simple: A SpinRite level 3 scan of an SSD will read and re-write an SSD's media to restore its stored bits to their original and proper values.

 

[Smash_Override]

So the damn thing gets about an hour into the scan and then just reboots itself. I am assuming it's a crash since I haven't been able to watch it happen. I guess it's just time for anew SSD. Thanks for the help.