HDD failures if used intensively

A

AOM

New Member
Feb 7, 2022
8
0
1
40
Hi,
I have a Proxmox instance installed that I use at home for storage(Truenass VM), and a couple other VMs running.
I have several sata HDD and SSD used some directly on the Proxmox, others mapped to the Truenass VM.
All discs experience randomly RW I/O errors when used intensively. This happens if I use torrent file download at high speed (~8-10MBps) or if I play from local Plex instance at above 1080p resolutions.
Once the a drive fails the only fix is to reboot the Proxmox instance to restore access to that drive.
Until now I tried the following:
- replacing the HDD and SSDs > no change, the new ones are failing two
- adding a sata pcie controller and moving the disks to this controller instead of the main-board sata ports > no change, drives still failing.
- replacing the PSU with a more powerful PSU > no change drives still failing
- replacing the mainboard > no change, drives still failing
I think I have eliminated all possible hardware failures so I belive the hypervisor software is the problem.

pool: rpool
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub in progress since Fri Apr 7 11:36:31 2023
269G scanned at 380M/s, 136G issued at 191M/s, 453G total
0B repaired, 29.93% done, 00:28:19 to go
config:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-Samsung_SSD_870_QVO_1TB_S5SVNF0NB05808R-part3 ONLINE 0 0 0
ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T409531B-part3 FAULTED 6 0 0 too many errors

Attached are the sata failures seen from Proxmox dmesg.
 

Attachments

  • SATA Failures.txt
    55.8 KB · Views: 3
You've not mentioned how much RAM you have; more than 32GB, I hope? [ZFS on ZFS, you probably need 64GB.]
Also, the SATA cables, given the amount of data you are moving, any noise will cause you trouble. You might want to fit high-quality well-shielded cabling.

In this case, I wonder if adding a ZFS ZIL would help. It's something I'm in the process of doing with my home server, which is currently a bit ill, mainly due to poor RAM. I am, however, using EXOS 8TB drives in a mirror for storage in that situation. The array also faults out, in terms of a ZFS kernel fault, when placed under excessive load.

You could get a small Enterprise class SSD, such as a Samsung PM893, to address it. It'd undoubtedly increase the lifespan of your QLC drives.
https://www.servethehome.com/what-is-the-zfs-zil-slog-and-what-makes-a-good-one/

Note: These are suggestions and only cover angles you don't seem to have mentioned yet. And that log reads like a horror story; double-check those SATA cables.
 
I forgot to mention that I replaced the sata cables already two times. What is strange is that the failures are not isolated to a certain disk/cable/sata port. The disks randomly became unreachable and their status remain failed until the hole physical system is powercicled.

The system memory is 32GB but it never tops. Usage it tops 28GB but in general is stable at 24GB. Memory is used is ECC.

I don't think the usage is extreme. Any system should be able to handle 10MBPS read/write with a mirror of two SSD or a raid-Z1of 3 hdds.
Both the mirror used in proxmox and the raid-z1 mapped under the trueNAS vm experience the same I/O errors caused when the disk is not reachable.

I did not mention that at some point I powered some of the drives from a second PSU to try to balance the load from the system PSU. The drives continued to fail on both PSUs.
I also balanced the drives between SATA ports on the mainboard and SATA ports on the PCIe controller.

I am also monitoring the system power usage and the idle is @65W while on high usage it tops at 75W. So nothing unusual here.

I think he only common point of failure remaining is the Proxmox kernel.

[187551.851784] ata12: hard resetting link
[187552.167432] ata12: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[187557.276971] ata12.00: qc timeout (cmd 0xec)
[187557.277750] ata12.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[187557.278341] ata12.00: revalidation failed (errno=-5)
[187557.278932] ata12: hard resetting link
[187557.595169] ata12: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[187567.776696] ata12.00: qc timeout (cmd 0xec)
[187567.777537] ata12.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[187567.778259] ata12.00: revalidation failed (errno=-5)
[187567.778812] ata12: limiting SATA link speed to 3.0 Gbps
[187567.779373] ata12: hard resetting link
[187573.136543] ata12: link is slow to respond, please be patient (ready=0)
[187577.820406] ata12: COMRESET failed (errno=-16)
[187577.821274] ata12: hard resetting link
 
That's very interesting, so the SATA cables can be discounted. The ports can't be it, as you're using a different MB, and the RAM is ECC.
Let's do a deeper dive here, as sometimes it can be strange things, so could you list the parts in use? I'm thinking down to the model, like AMD 1700X, ASUS STRYX B450M ITX-MINI etc.

A silly question, has the CPU stayed the same across builds, and have you tried running Memtest? [Just in case.]

On the kernel front, there are also the 6.2 ones available, you could always install it for a whirl to see if you hit the same problem.

Motherboard BIOS fully up-to-date? [Some of that newer firmware does address electrical/signalling stability]
 
AOM said:
Any system should be able to handle 10MBPS read/write with a mirror of two SSD or a raid-Z1of 3 hdds.
Click to expand...
Depends on the workload. I have seen SMR HDDs drop down to a few kb/s. I think you are overrating QLC/SMR performance and underestimating the write amplification of CoW and sync writes.
Example: There are Samsung QVOs that will drop to 40MB/s once the SLC and DRAM caches are filled up. Using ZFS I've seen write amplification here of up to factor 62. If you combine both you get 40 MB/s / 62 = 645 KB/s.
 
Nuke Bloodaxe said:
That's very interesting, so the SATA cables can be discounted. The ports can't be it, as you're using a different MB, and the RAM is ECC.
Let's do a deeper dive here, as sometimes it can be strange things, so could you list the parts in use? I'm thinking down to the model, like AMD 1700X, ASUS STRYX B450M ITX-MINI etc.

A silly question, has the CPU stayed the same across builds, and have you tried running Memtest? [Just in case.]

On the kernel front, there are also the 6.2 ones available, you could always install it for a whirl to see if you hit the same problem.

Motherboard BIOS fully up-to-date? [Some of that newer firmware does address electrical/signalling stability]
Click to expand...
MB is a Gigabyte Aorus B550 elite V2, CPU is AMD Ryzen 5 PRO 5650G, I have run the Memtest and resulted ok.
I have tried all versions of MB official bios hoping that will solve the issue but it didn't. Now it uses latest bios version.

CPU remained the same but I can't find a possible link to indicate the CPU is the issue. A faulty CPU should have other indicators as well. Other errors should be present.
 
Dunuin said:
Depends on the workload. I have seen SMR HDDs drop down to a few kb/s. I think you are overrating QLC/SMR performance and underestimating the write amplification of CoW and sync writes.
Example: There are Samsung QVOs that will drop to 40MB/s once the SLC and DRAM caches are filled up. Using ZFS I've seen write amplification here of up to factor 62. If you combine both you get 40 MB/s / 62 = 645 KB/s.
Click to expand...
But the result of a slow drive would be to have the HDD be completely removed from the system until next reboot? Also the intensive R/W is done on the 3xSegate pool not on the proxmox pool, yet the SSDs from the Proxmox pool are failing too.
The phisycal 3x Segate HDD pool are mapped directly under the TrueNAS VM.
 
AOM said:
But the result of a slow drive would be to have the HDD be completely removed from the system until next reboot?
Click to expand...
No, ZFS will try to read/write to the disk and if that disks can't answer in time, because it is busy, ZFS will timeout and count that as a read/write error. Too much of those read/write errors and the pool will switch to degraded state. So the disks don't have to be damaged, they just can be too slow.
 
Dunuin said:
No, ZFS will try to read/write to the disk and if that disks can't answer in time, because it is busy, ZFS will timeout and count that as a read/write error. Too much of those read/write errors and the pool will switch to degraded state. So the disks don't have to be damaged, they just can be too slow.
Click to expand...
Would the enterprise SSD I suggested earlier to act as a ZFS ZIL buy AOM's array enough time to handle the write amplification in this situation? [I have my own considerations, and his setup is somewhat similar in some ways.]
 
A slog (ZIL on dedicated disk) will only help with sync writes. Won't help anything with async writes or any reads. So only useful in case you got a lot of sync writes.
 
So practically what you are saying is that Proxmox cannot be used for a small home use hypervisor running NAS or PLEX because ZFS cannot handle enduser HDD and SSD drives? It sounds strange because many use it for this purpose.
 
Nuke Bloodaxe said:
You've not mentioned how much RAM you have; more than 32GB, I hope? [ZFS on ZFS, you probably need 64GB.]
Also, the SATA cables, given the amount of data you are moving, any noise will cause you trouble. You might want to fit high-quality well-shielded cabling.

In this case, I wonder if adding a ZFS ZIL would help. It's something I'm in the process of doing with my home server, which is currently a bit ill, mainly due to poor RAM. I am, however, using EXOS 8TB drives in a mirror for storage in that situation. The array also faults out, in terms of a ZFS kernel fault, when placed under excessive load.

You could get a small Enterprise class SSD, such as a Samsung PM893, to address it. It'd undoubtedly increase the lifespan of your QLC drives.
https://www.servethehome.com/what-is-the-zfs-zil-slog-and-what-makes-a-good-one/

Note: These are suggestions and only cover angles you don't seem to have mentioned yet. And that log reads like a horror story; double-check those SATA cables.
Click to expand...
Can you please recommend a high quality well-shielded sata cable? I only found esata cables to be shielded. If quality means having locking clips I already have that type.
 
ZFS is made for the datacenter to be run on proper hardware. If you don't want to use proper hardware (a lot of ECC RAM, enterprise SSDs, UPS, LSI HBA card... ) it would be better to use some other raid/filesystem that is less demanding.
And yes, you can use ZFS with nearly anything, just don't expect good performance or long living SSDs. ZFS for example killed 4 consumer SSDs in my home server last year.
So you get what you pay for when buying SSDs.
 
Last edited:
Enterprise SSD: ZFS works great but you won't hear from those people because there are no problems.
MLC/TLC SSD: ZFS works fine but some people complain that they wear out quickly. Depends on the I/O work-load.
CMR HDD: ZFS works fine but some people complain that it is slow, but that is to be expected unless you run many in a SAN.
SMR HDD: ZFS fails because those drives are too slow, and people complain that the manufacturer did not label them clearly enough and nobody warned them.
QLC SSD: ZFS fails because people assume that are better than (CMR) HDD. They are so disappointed that they won't believe they wasted their money.

I did find some EXOS 8TB which were labeled as not suitable for NAS but I don't know if yours are the same 7E8, 5E8.
 
  • Like
Reactions: Dunuin
leesteken said:
Enterprise SSD: ZFS works great but you won't hear from those people because there are no problems.
MLC/TLC SSD: ZFS works fine but some people complain that they wear out quickly. Depends on the I/O work-load.
CMR HDD: ZFS works fine but some people complain that it is slow, but that is to be expected unless you run many in a SAN.
SMR HDD: ZFS fails because those drives are too slow, and people complain that the manufacturer did not label them clearly enough and nobody warned them.
QLC SSD: ZFS fails because people assume that are better than (CMR) HDD. They are so disappointed that they won't believe they wasted their money.

I did find some EXOS 8TB which were labeled as not suitable for NAS but I don't know if yours are the same 7E8, 5E8.
Click to expand...
This got me thinking my end, I do have 7E8 units here, but after careful checking on Seagate's list I found they are of the CMR variety. However, I also discovered something I had completely overlooked, which was to upgrade the Firmware of the HDDs themselves. I've gone from SN02 to SN04.
[I completely forgot about the HDD firmware, it's normally more of an SSD thing.] I'll be running some tests. [@AOM, you've probably upgraded all of the SSD drive firmware? If you like living life dangerously, you could also look at the HDD firmware.]

Also, the RAM in this unit has now been replaced and jumped up to 64GB, it took a fair amount of checking with the speeds [green screen etc, not the Ram's fault] until I hit stable at 2400. [@AOM , if you, like me, have extensive experience encountering "voodoo" faults, you could always check to see if downclocking the memory speed will produce an improvement, as you have an AMD system, but carefully and only as a last test.]

@AOM In terms of the SATA cables, I was thinking of the cables included with enterprise-level RAID cards, which tend to be foil-shielded like this:
Foil-shielded cable example
You can always get the foil tape separately and wrap your cables. Less noise is always a good thing.
 
Last edited:
So what would be a recommended setup of my enduser drives? Two 1T SSD used in Proxmox hypervisor for storing my VMs and 3x 8TB HDD in a virtualized TrueNAS for data storage(docs, multimedia, etc) and backups.
Right now the 2xSSD are mirrored and the 3X8TB HDD are raidz1-0.

TrueNas only supports ZFS.
Proxmox does not recommend software RAID.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back