Proxmox not booting after mem upgrade, but windows does on same hardware

[sroot]

I have proxmox running for a long time on a Fujistu Esprimo D7010 i5-10500 CPU with 32GB ram in 2 x 16GB sticks.

I realised that the memory was now too low for things I was doing so I bought another identical 32GB ram in 2 x 16GB stick and have now filled all 4 of the slots.

When booting there is 1 beep, a pause, then continuous rapid beeping. OK, I think it's a bad ram stick, or bad slot/motherboard...
So i tried the 2 new sticks on their own, it boots fine.
I tried the sticks in the blue pair of slots instead of the black pair of slots, it boots fine.
I tried a stick from each set - at 32GB it boots fine.
So i tried the backup hardware I keep on hand, an identical D7010 i5-10500 with it's original SSD and windows 11... and it surprised me and booted fine with 64GB ram

OK, so I moved my 2TB NVme in place of the on board SSD + the Add on card for the mirrored 2TB NVMe + the dual port network card - and it beeps and doesn't boot
(so it's not the motherboard or the ram, and the Add on Card NVMe and network card work fine with 32GB ram could they really upset things having 64GB?)

So I updated the motherboard firmware (via the windows 11 tool from fujitsu), used all 64GB in 4 slots fordoing it, and windows still starts fine,
but proxmox... still doesn't boot if I use all 4 sticks.

Does anyone have any ideas on where I can fault find further please?

 

[GabrielLando]

I don't know if I understand right, but could you run Windows 11 on the proxmox box with all 64GB?
If you put the proxmox drives, can you see the POST video? (The bios/manufacturer logo)

If you try to replace the machines (moving proxmox drives to the backup system), does the system boot?

 

[UdoB]

Look for "memtest86+", boot it and let it run over night. It will stress-test the memory...

(I am unsure if it is included in the normal PVE-installation media, but https://memtest.org/ might help...)

 

[sroot]

I don't know if I understand right, but could you run Windows 11 on the proxmox box with all 64GB?
If you put the proxmox drives, can you see the POST video? (The bios/manufacturer logo)

If you try to replace the machines (moving proxmox drives to the backup system), does the system boot?

Yes, on both boxes windows nvme boots with 64GB
But on both boxes with Proxmox nvmes and 64Gb it beeps once and goes not reach POST (no display).
Proxmox does boot with any of the sticks in 32GB

 

[sroot]

Look for "memtest86+", boot it and let it run over night. It will stress-test the memory...

(I am unsure if it is included in the normal PVE-installation media, but https://memtest.org/ might help...)

Good idea.
Now I appreciate deciding to buy 2 identical PCs, I can test the memory without losing access to proxmox for the day!

 

[leesteken]

Yes, on both boxes windows nvme boots with 64GB
But on both boxes with Proxmox nvmes and 64Gb it beeps once and goes not reach POST (no display).

If your system does not POST then it cannot be a software configuration (or Proxmox) issues. Maybe the NVMe drive that you use for Proxmox is broken? What does one beep mean according to the motherboard manual/support?

 

[sroot]

Thanks for the suggestions,

If your system does not POST then it cannot be a software configuration (or Proxmox) issues.

This was my original thought, until I put in the original windows NVMe and it boots fine with 64GB

Maybe the NVMe drive that you use for Proxmox is broken?

But proxmox boots with 32GB RAM.
Proxmox is on 2 x 1TB NMVe mirrored, one in the motherboard and another in a PCIe Add in card.
Maybe I've stumbled on an edge case related to the add in card and 64GB ram.

What does one beep mean according to the motherboard manual/support?

Its very unhelpful. The PC manual doesn't mention beeps at all. For other mainboards I found the Fujistu "Technical Description" and from this and other google results it seems 1 beep is RAM related. that makes sense I think, use 2 sticks it's fine, 4 sticks and it is not.

One potential hint; the manual says memory up to 2933mhz. I have 3200mhz. I assumed this would just work at a lower speed and 3200mhz is also listed on compatible memory sites like https://www.kingston.com/unitedking...jitsu-esprimo-d7010-f-d3823-a2?speed=3200mt/s. and of course it worked with windows.... so I'm lost.

I'm going to try memtest on all the sticks first, then I'll install proxmox on the backup hardware from scratch on a spare nvme.
If that works I guess I'll try a new nvme add in card and try a mirrored proxmox install.

If I find a cause I will post back!

Thanks for the suggestions everyone.

 

[sroot]

Solved!

It was the Add in Card for NVMe, 'SMBus conflict' means you sometimes (motherboard dependant) can't use all memory slots if you have that card.

I found this post that confirmed it once I found a magic combination of google search phrases: https://www.supermicro.com/support/faqs/faq.cfm?faq=26714

the AOC-SLG3-2M2 NVMe add in card can take 2 NVMe drives, although I only use one. It can do bifurcation, meaning each NVMe disk get's the full 4 lanes of this 8 lane card OR it shares the 4 lanes without bifurcation. Originally it was in a supermicro server that did that but I downsized to an old desktop as I didn't need the power.

The motherboard I have doesn't do bifurcation but it didn't matter as I only have one NVMe.

Buying a new PCIe to NVMe Card (single) solved the problem.

From my first post "and the Add on Card NVMe and network card work fine with 32GB ram could they really upset things having 64GB?"
Why yes, apparently they really can upset things