[SOLVED] [HELP] PVE can not detect all the ram (128G RAM only detects 64GB)

[pw0rld]

Hi, everyone. I got an incomprehensible problem in using PVE.
My server has installed 8*16 128G RAM, I am confident that the RAM is not broken and that it is correctly inserted. Because both the bios and the BMC can recognize 128G RAM. However, PvE cannot identify 128GB RAM and can only detect 64GB RAM. I'm not sure what went wrong.

Here is some information about my server.

root@pve:~# cat /proc/meminfo
MemTotal:       65580648 kB
MemFree:        34197828 kB

root@pve:~# lshw -c memory
  *-firmware           
       description: BIOS
       vendor: American Megatrends Inc.
       physical id: 1
       version: 06.00.01
       date: 10/09/2021
       size: 64KiB
       capacity: 32MiB
       capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppynec int13floppytoshiba int13floppy360 int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int14serial int17printer int10video usb biosbootspecification uefi
  *-memory
       description: System Memory
       physical id: 57
       slot: System board or motherboard
       size: 128GiB
       capacity: 12TiB
       capabilities: ecc
       configuration: errordetection=ecc
     *-bank:0
          description: DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
          product:
          vendor: Hynix
          physical id: 0
          serial:
          slot: CPU0_C0D0
          size: 8GiB
          width: 64 bits
          clock: 3200MHz (0.3ns)

And here is the system initialize information.

And PvE


Here is dmesg information, it only load 64G RAM

root@pve:~# dmesg | grep -i memory
[    0.017314] ACPI: Reserving FACP table memory at [mem 0x69558000-0x69558113]
[    0.017316] ACPI: Reserving DSDT table memory at [mem 0x6951d000-0x69556d17]
[    0.017318] ACPI: Reserving FACS table memory at [mem 0x699f9000-0x699f903f]
[    0.017319] ACPI: Reserving SPMI table memory at [mem 0x69557000-0x69557040]
[    0.017320] ACPI: Reserving FIDT table memory at [mem 0x6951c000-0x6951c09b]
[    0.017321] ACPI: Reserving SSDT table memory at [mem 0x6955a000-0x6955a747]
[    0.017322] ACPI: Reserving EINJ table memory at [mem 0x69559000-0x6955914f]
[    0.017323] ACPI: Reserving ERST table memory at [mem 0x6951b000-0x6951b22f]
[    0.017324] ACPI: Reserving BERT table memory at [mem 0x6951a000-0x6951a02f]
[    0.017325] ACPI: Reserving SSDT table memory at [mem 0x69519000-0x69519744]
[    0.017326] ACPI: Reserving MCFG table memory at [mem 0x69518000-0x6951803b]
[    0.017327] ACPI: Reserving BDAT table memory at [mem 0x69517000-0x6951702f]
[    0.017328] ACPI: Reserving HMAT table memory at [mem 0x69516000-0x6951617f]
[    0.017329] ACPI: Reserving HPET table memory at [mem 0x69515000-0x69515037]
[    0.017330] ACPI: Reserving MIGT table memory at [mem 0x69514000-0x6951403f]
[    0.017331] ACPI: Reserving MSCT table memory at [mem 0x69513000-0x69513063]
[    0.017332] ACPI: Reserving MUDT table memory at [mem 0x69512000-0x6951202f]
[    0.017333] ACPI: Reserving WDDT table memory at [mem 0x69511000-0x6951103f]
[    0.017334] ACPI: Reserving APIC table memory at [mem 0x6950f000-0x6950f35d]
[    0.017336] ACPI: Reserving SLIT table memory at [mem 0x6950e000-0x6950e42b]
[    0.017337] ACPI: Reserving SRAT table memory at [mem 0x6950a000-0x6950d22f]
[    0.017338] ACPI: Reserving OEM4 table memory at [mem 0x69446000-0x69509d60]
[    0.017339] ACPI: Reserving OEM1 table memory at [mem 0x693b7000-0x69445788]
[    0.017340] ACPI: Reserving SSDT table memory at [mem 0x6937b000-0x693b62a4]
[    0.017341] ACPI: Reserving BDAT table memory at [mem 0x6937a000-0x6937a02f]
[    0.017342] ACPI: Reserving HEST table memory at [mem 0x69379000-0x6937913b]
[    0.017343] ACPI: Reserving DMAR table memory at [mem 0x69378000-0x6937826f]
[    0.017344] ACPI: Reserving SPCR table memory at [mem 0x69377000-0x6937704f]
[    0.017346] ACPI: Reserving SSDT table memory at [mem 0x69372000-0x6937615f]
[    0.017347] ACPI: Reserving WSMT table memory at [mem 0x69510000-0x69510027]
[    0.017348] ACPI: Reserving FPDT table memory at [mem 0x69371000-0x69371043]
[    0.017947] Early memory node ranges
[    0.142397] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.142400] PM: hibernation: Registered nosave memory: [mem 0x0003e000-0x0003efff]
[    0.142402] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.142405] PM: hibernation: Registered nosave memory: [mem 0x61046000-0x61046fff]
[    0.142407] PM: hibernation: Registered nosave memory: [mem 0x6715b000-0x6905afff]
[    0.142408] PM: hibernation: Registered nosave memory: [mem 0x6905b000-0x6955afff]
[    0.142409] PM: hibernation: Registered nosave memory: [mem 0x6955b000-0x699fafff]
[    0.142410] PM: hibernation: Registered nosave memory: [mem 0x699fb000-0x6bfaefff]
[    0.142411] PM: hibernation: Registered nosave memory: [mem 0x6bfaf000-0x6c1fefff]
[    0.142413] PM: hibernation: Registered nosave memory: [mem 0x6f800000-0x8fffffff]
[    0.142414] PM: hibernation: Registered nosave memory: [mem 0x90000000-0xfcffffff]
[    0.142415] PM: hibernation: Registered nosave memory: [mem 0xfd000000-0xfe7fffff]
[    0.142416] PM: hibernation: Registered nosave memory: [mem 0xfe800000-0xfed1ffff]
[    0.142417] PM: hibernation: Registered nosave memory: [mem 0xfed20000-0xfed44fff]
[    0.142418] PM: hibernation: Registered nosave memory: [mem 0xfed45000-0xfeffffff]
[    0.142419] PM: hibernation: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.142421] PM: hibernation: Registered nosave memory: [mem 0x800000000-0xfffffffff]
[    0.142423] PM: hibernation: Registered nosave memory: [mem 0x1800000000-0x1fffffffff]
[    0.319591] Memory: 65359996K/66755560K available (16393K kernel code, 4345K rwdata, 10180K rodata, 2884K init, 4904K bss, 1395360K reserved, 0K cma-reserved)
[    0.402884] Freeing SMP alternatives memory: 40K
[    0.880359] x86/mm: Memory block size: 2048MB
[    2.140011] acpi/hmat: HMAT: Memory (0x0 length 0x80000000) Flags:0003 Processor Domain:0 Memory Domain:0
[    2.140014] acpi/hmat: HMAT: Memory (0x100000000 length 0xf80000000) Flags:0003 Processor Domain:0 Memory Domain:0
[    2.140016] acpi/hmat: HMAT: Memory (0x1080000000 length 0x1000000000) Flags:0003 Processor Domain:1 Memory Domain:1
[    2.694708] Freeing initrd memory: 61680K
[    2.741322] Freeing unused decrypted memory: 2036K
[    2.741942] Freeing unused kernel image (initmem) memory: 2884K
[    3.064403] Freeing unused kernel image (text/rodata gap) memory: 2036K
[    3.064466] Freeing unused kernel image (rodata/data gap) memory: 60K
[    3.246606] i2c i2c-0: 16/32 memory slots populated (from DMI)
[    3.246608] i2c i2c-0: Systems with more than 4 memory slots not supported yet, not instantiating SPD
[   42.208822] EDAC i10nm: No hbm memory

And my server is INSPUR NF5280M6, and the CPU is 2 Intel(R) 5318Y.
My PvE version is 7.2-3.

It would be great if someone could help.

Thanks.
Pw0rld.

 

[fabian]

please post a full boot log..

 

[pw0rld]

please post a full boot log..

Sorry, here is all the dmesg logs.

 

[fabian]

your bios only reports

0x0000000000000000-0x000000000003dfff: 0 GB
0x000000000003f000-0x000000000009ffff: 0 GB
0x0000000000100000-0x000000006715afff: 1 GB
0x000000006c1ff000-0x000000006f7fffff: 0 GB
0x0000000100000000-0x00000007ffffffff: 27 GB
0x0000001000000000-0x00000017ffffffff: 31 GB
0x0000002000000000-0x000000207fffffff: 1 GB

(from the output of bios-e820) as usable memory..

what does numactl --hardware report? what about dmidecode --type 17?

 

[pw0rld]

your bios only reports

0x0000000000000000-0x000000000003dfff: 0 GB
0x000000000003f000-0x000000000009ffff: 0 GB
0x0000000000100000-0x000000006715afff: 1 GB
0x000000006c1ff000-0x000000006f7fffff: 0 GB
0x0000000100000000-0x00000007ffffffff: 27 GB
0x0000001000000000-0x00000017ffffffff: 31 GB
0x0000002000000000-0x000000207fffffff: 1 GB

(from the output of bios-e820) as usable memory..

what does numactl --hardware report? what about dmidecode --type 17?

Thanks, Fabian. Thanks for your help.

Here is numactl report:

root@pve:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
node 0 size: 31838 MB
node 0 free: 7567 MB
node 1 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 1 size: 32204 MB
node 1 free: 15182 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

And here is dmidecode report as the attached files:

 

[fabian]

I'd check your mainboard manual if there is anything special about the dimm slots and/or corresponding bios settings

 

[pw0rld]

I'd check your mainboard manual if there is anything special about the dimm slots and/or corresponding bios settings

Thanks, fabian. Following your suggestion, I double-checked the bios settings.
And I get the answer to my problem. The first time I checked that the memory capacity is 128GB, it met my expectations. But I forget to verify the CPU settings. On this server, I do some research using the Intel SGX. I opened the Intel SGX settings and selected 32G as the SGX PRMRR size. I have two CPUs, and a total of 64GB of memory will be pre-planned to join the SGX EPC. That explains why my server only detected 64GB.


The PRMRR size is then adjusted to 8G. The memory size has been increased to 109G. (128 - 8 * 2 = 112 ≈ 109G)



Thank you, Fabian. Thank you for your assistance with this issue.
Best regards,
Pw0rld
.