Linux ECC RAM memory error analysis and faulty DIMM module detection

mipsH

Well-Known Member
I have created Linux RAM memory error analysis script to analize the ECC memory errors.
The script provides the following functionality:
  • Memory Error Detection – Reads EDAC counters for Corrected Errors (CE) and Uncorrected Errors (UE).
  • Memory Channel Identification – Identifies the processor socket and memory channel where errors occur.
  • DIMM Module Identification – Uses EDAC DIMM index information and physical memory slot mapping to identify the affected DIMM module.
  • Physical Module Verification – Compares the results with DMI/SMBIOS information retrieved using dmidecode.
  • Potentially Faulty DIMM Reporting – Displays the exact physical module designation, for example PROC 2 DIMM 1 (P2-A), including its memory capacity and Part Number.
Script is tested on HP Blade server ProLiant BL460c Gen8 with faulty ECC DIMM module.
Script is located on Github: https://github.com/hhorvat/scripts/blob/main/dimm-diagnose.sh

Final result (short version) may look like:
================================================
POTENTIAL FAULTY DIMMs
================================================
EDAC LABEL: CPU_SrcID#1_Ha#0_Chan#1_DIMM#0
Socket: 1
Processor: PROC 2
Channel: 1
DIMM index: 0
Corrected errors: 106
Uncorrected errors: 0

POTENTIAL DIMM:
PROC 2 DIMM 1 (P2-A)
DMI status: INSTALLED
DIMM size: 16 GB
Part Number: 672612-081
SEVERITY: CORRECTED ERRORS
================================================



Feel free to test it.
 
Last edited:
Nice solution to a problem that seems only Debian-based system to have. On RHEL I'm using for decades mcelog, which does that for me already via dmesg:

Code:
[227225.048231] mce: [Hardware Error]: Machine check events logged
[227225.048300] EDAC skx MC0: HANDLING MCE MEMORY ERROR
[227225.048323] EDAC skx MC0: CPU 0: Machine Check Event: 0 Bank 7: 8c00004001010090
[227225.048351] EDAC skx MC0: TSC 0
[227225.048365] EDAC skx MC0: ADDR 1c4598f6c0
[227225.048383] EDAC skx MC0: MISC 200000c027e01086
[227225.048396] EDAC skx MC0: PROCESSOR 0:50654 TIME 1592803731 SOCKET 0 APIC 0
[227225.048429] EDAC MC0: 1 CE memory read error on CPU_SrcID#0_MC#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x1c4598f offset:0x6c0 grain:32 syndrome:0x0 -  err_code:0101:0090 socket:0 imc:0 rank:0 bg:1 ba:3 row:121d2 col:3e8)

I did not find any similar package or program in Debian, Hmm .. have to investigate this.

Have you seen the error in the ipmi sel? That is normally also logged there.
 
  • Like
Reactions: Johannes S
Nice solution to a problem that seems only Debian-based system to have. On RHEL I'm using for decades mcelog, which does that for me already via dmesg:

Code:
[227225.048231] mce: [Hardware Error]: Machine check events logged
[227225.048300] EDAC skx MC0: HANDLING MCE MEMORY ERROR
[227225.048323] EDAC skx MC0: CPU 0: Machine Check Event: 0 Bank 7: 8c00004001010090
[227225.048351] EDAC skx MC0: TSC 0
[227225.048365] EDAC skx MC0: ADDR 1c4598f6c0
[227225.048383] EDAC skx MC0: MISC 200000c027e01086
[227225.048396] EDAC skx MC0: PROCESSOR 0:50654 TIME 1592803731 SOCKET 0 APIC 0
[227225.048429] EDAC MC0: 1 CE memory read error on CPU_SrcID#0_MC#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x1c4598f offset:0x6c0 grain:32 syndrome:0x0 -  err_code:0101:0090 socket:0 imc:0 rank:0 bg:1 ba:3 row:121d2 col:3e8)

I did not find any similar package or program in Debian, Hmm .. have to investigate this.

Have you seen the error in the ipmi sel? That is normally also logged there.
It is not more about the tool but the easy way to pinpoint to the right DIMM module, especially when the server is in remote site.
So for me the important information is something like (output of the script):

POTENTIAL DIMM:
PROC 2 DIMM 1 (P2-A)
DMI status: INSTALLED
DIMM size: 16 GB
Part Number: 672612-081
SEVERITY: CORRECTED ERRORS


P.S. This version of the script has a limited HW specific mappings to the slot names inside the Motherboard, but will be modular for more vendors (HP, Dell, ...) and specific servers/motherboards.
 
Last edited:
This version of the script has a limited HW specific mappings to the slot names inside the Motherboard, but will be modular for more vendors (HP, Dell, ...) and specific servers/motherboards.
Yes, I saw that. My older HP models have a map of all the modules on their info panel and a red led to signal what DIMM is errornous. I've never seen this on newer models. Most have the DIMM layout printed on the chassis lit and need to find it yourself. I haven't looked in a recent epyc case with the many, many DIMMS, must be even harder to localize.
 
On Debian, rasdaemon is the usual counterpart to mcelog: it records EDAC/MCE events and can report labels like CPU_SrcID#0_MC#0_Chan#0_DIMM#1. It will not translate those into chassis silkscreen names like P2-A, which is where your mapping adds real value. For a quick cross-check, edac-util and the sysfs EDAC counters are also available, but rasdaemon is probably the package to investigate first.