Node randomly reboots

[daros]

Hello,

We got an 6 node cluster, 1 node randomly reboots (twice this week) after working for months/years correctly.
i could not find anything in the logs that can point me to an reason what happens.

Specs:
48 x Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz (2 Sockets)
288gb ram

Proxmox
Linux 5.4.114-1-pve #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200)
pve-manager/6.4-8/185e14db

Logs are attached, syslog and messages log.

Can someone help me? Or point me in the direction to look for?
I already disabled the PBS the node could not find.

 

[mira]

Please provide the syslog from one of the other cluster nodes as well.

 

[daros]

Hello Mira, see attached.

 

[mira]

So the node just didn't respond anymore and ~4 minutes later was back up.
Looks like a hard reset, rather than a simple reboot. Do you have additional logging/IPMI logs?

 

[daros]

Hello,

I just looked in IPMI en found this:

 35 | 2021/07/13 08:30:23 | Memory
    | Assertion:Uncorrectable ECC / other uncorrectable memory error @DIMME1(CPU1)
 36 | 2021/07/13 08:31:04 | BIOS OEM (Memory Error)
    | Assertion:(runtime) Failing DIMM: DIMM location (P1-DIMME1)
 37 | 2021/07/13 08:31:21 | BIOS OEM (Memory Error)
    | Assertion:(runtime) Failing DIMM: DIMM location (P1-DIMME1)
 38 | 2021/07/13 08:31:32 | BIOS OEM (Memory Error)
    | Assertion:(runtime) Failing DIMM: DIMM location (P1-DIMME1)
 39 | 2021/07/13 08:33:36 | Processor
    | Assertion:CATERR has occurred
 40 | 2021/07/13 08:36:34 | BIOS OEM (Memory Error)
    | Assertion:Failing DIMM: DIMM location (Correctable memory component found) (P1-DIMME1)
 41 | 2021/07/15 05:55:49 | Memory
    | Assertion:Uncorrectable ECC / other uncorrectable memory error @DIMME1(CPU1)
 42 | 2021/07/15 05:56:29 | BIOS OEM (Memory Error)
    | Assertion:(runtime) Failing DIMM: DIMM location (P1-DIMME1)
 43 | 2021/07/15 05:56:46 | BIOS OEM (Memory Error)
    | Assertion:(runtime) Failing DIMM: DIMM location (P1-DIMME1)
 44 | 2021/07/15 05:56:57 | BIOS OEM (Memory Error)
    | Assertion:(runtime) Failing DIMM: DIMM location (P1-DIMME1)

Also when i look in proxmox found out the total ram is changed.

 

[mira]

35 | 2021/07/13 08:30:23 | Memory
    | Assertion:Uncorrectable ECC / other uncorrectable memory error @DIMME1(CPU1)
 36 | 2021/07/13 08:31:04 | BIOS OEM (Memory Error)
    | Assertion:(runtime) Failing DIMM: DIMM location (P1-DIMME1)

So it seems to be an issue with your memory.