ZFS data slowly getting corrupt

M

mayrjohannes

New Member
Jan 24, 2024
11
0
1
Hi all,
I run pve on a Lexxar SSD NM620 with zfs. I know this is not a enterprise SSD, but I only use it in my homelab on my HPE Proliant ML30 Server. The RAM is EEC protected.
A second zfs pool holds my important data (2 redundand HDD's).
A while ago I noticed that some old data on the Lexxar zfs pool is corrupt (the data in the main pool and in all snapshots). It was only 3 or 4 files. Now this weekend a few more files are added to this list. It is all very old data, being there from the time of installing pve.
Is this the first sign of the SSD slowly passing away? The SSD is only a year old.

I also noticed, that if i run to many LXC Containers (more than 10), the iodelay is getting very high. A few weeks ago, I moved some of the LXC's to the HDD Pool and the iodelay is now back to a few percent. Maybe this is also related?

Thx
 
I would check your SMART data. Go to -> proxmox VE GUI - > Host -> Disks and check if it still says passed by the S.M.A.R.T column.

It might also be bit rot but for just 1 year it seems to be really early to consider bit rot as a reason.

Also check your ZFS pool.
proxmox VE GUI - > Host -> ZFS and check the pool health.
 
If you are doing zfs and you want self-healing scrubs, you need at least a mirror... And you want everything on a UPS with NUT installed so you're getting clean power + shutdowns
 
Sorry for the delay, I was on a longer trip.
Here is my smart data for the disk:
INI: 
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        31 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    7%
Data Units Read:                    53,791,789 [27.5 TB]
Data Units Written:                 44,184,917 [22.6 TB]
Host Read Commands:                 302,951,146
Host Write Commands:                760,682,269
Controller Busy Time:               56,311
Power Cycles:                       68
Power On Hours:                     7,347
Unsafe Shutdowns:                   37
Media and Data Integrity Errors:    22
Error Information Log Entries:      3
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               31 Celsius
Temperature Sensor 2:               27 Celsius

The Lexxar pool only consists of one Lexar SSD NM620 2TB (M.2)

I'm not sure if I understand this rigth, but I do not so much want a self healing scrubs, I would like to have a disk that does not destroy data after less than a year.

Also I noticed that the IO delay is quite high and thus also the load average. A speed test within a LXC container is much slower than on the pve host itself. I already had this a while ago, but there it was due to a container that was very close to its quota and that slowed down zfs.
I increased the quota and all was working again. Then after 2 months again, but nothing close to any quota. I then moved parts of the subvolumes to the HDD data pool and problem solved again. Now it starts again to have very high IO delays of 25% and more.

Could this be related? Is the Lexar SSD NM620 a bad choice and replacing it, will solve all my issues?

Or was it just a bad idea to use ZFS for the pve host's root partition?

Thx Johannes
 
I just realize that even 'zpool status' lists "too many slow I/Os"
Code: 
pool: rpool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 01:08:23 with 15 errors on Sun Nov 10 01:32:29 2024
config:

        NAME                                                 STATE     READ WRITE CKSUM
        rpool                                                DEGRADED     0     0     0
          nvme-Lexar_SSD_NM620_2TB_NLE308R007449P110N-part3  DEGRADED     0     0     0  too many slow I/Os

till now, all corrupted files only appear in one dataset. Lucky me, this is a dataset I do not car about at all, but anyhow, it is like 1 or 2 more files per month. My fear it that one day, it starts with another dataset that is more important.
No worry, I have regular backups to Proxmox Backup Server and to external HDD.
 
did you run zpool scrub rpool and zpool clear rpool ?
And add 1 oder more SSD SATA3 to your rpool.
And after that 2x more SSD SATA3 as rpool, think of the boot partition too, then you can remove the nvme m.2!
 
You must log in or register to reply here.
Top Bottom