Host hard crashes, PVE 8.1.4

[logikbarn]

I might consider doing that, I saw someone post on another forum to reduce the 4800Mhz to 4400 in BIOS. It seems to be running fine with pfSense directly on it though (no Proxmox). I am currently at 30hrs and running, wondering if that could still be the issue if now it seems to be ok.

 

[sammyke007]

I might consider doing that, I saw someone post on another forum to reduce the 4800Mhz to 4400 in BIOS. It seems to be running fine with pfSense directly on it though (no Proxmox). I am currently at 30hrs and running, wondering if that could still be the issue if now it seems to be ok.

View attachment 82575

For me, reducing the RAM speed wasn't an option, even after flashing a modded/unlocked BIOS. Samsung DDR5 4800mhz running perfectly fine now.

 

[logikbarn]

For me, reducing the RAM speed wasn't an option, even after flashing a modded/unlocked BIOS. Samsung DDR5 4800mhz running perfectly fine now.

I didn't check if it was an option either but am willing to give this a shot thanks for the tip! I Just ordered a new stick of Samsung 32GB DDR5-4800, should come in late next week. Keeping this on pfSense for now. It hasn't rebooted yet so with FreeBSD it seems to be fine and if all else fails I may just keep this setup since it works at least. I still have the old NVME with Proxmox installed so I'll just swap those out once the new stick is in.

I did see there was an update to the Intel microcode package coming https://tracker.debian.org/pkg/intel-microcode. Could take a month or so to make it to stable judging by previous releases. I'd be curious to try this as well if the stick doesn't work, the system would eventually update to this anyway since non-free-firmware repo is enabled. I did notice previously that by switching the NIC type to the virtualized Intel 1000E it didn't have issues, but I only ran it this way for a short period of time as speed tests proved the Intel 1000E driver ran at 300Mbps compared to VirtIO of 900-1000Mbps (1GBps Port). So not conclusive yet but enough to raise suspicions.

FWIW I have another Proxmox host with much better hardware. I ran pfSense there for months before migrating to this without issue. So I don't think this configuration or VM related. When I ran the TopTon unit without pfSense for a few days and a Debian VM with stress-ng and fio to test hardware, it also didn't reboot for the 24-30hr period it ran. So the issue seems to be with running pfSense particularly. I've seen other posts in other forums talk about the NIC (i-226v) being the issue, potentially also related to FreeBSD running on Debian and drivers. This is now making some sense for me at least considering the no reboots with just pfSense so far but what do I know. Going into the weeds now but just some food for thought I guess.

In the end, the last 2 remaining attempts for me looks like will be:

• Replacing the memory stick to Samsung from Crucial
• Updating to the latest intel-microcode

Will update as soon as I have something here. The memory stick update will come in likely sometime next week or after. The microcode updates will take some time until Debian releases it.

Out of curiosity what can you share about your model sammyke007?

 

[sammyke007]

I've tried the Intel microcode fixes too (on OPNsense) but that didn't help in my case.

My model is this one:
N100 "X2E" from Topton (AliExpress):



Working RAM:

Motherboard BK-1264NP-4L v1.4

 

[logikbarn]

I've tried the Intel microcode fixes too (on OPNsense) but that didn't help in my case.

My model is this one:
N100 "X2E" from Topton (AliExpress):

View attachment 82598
View attachment 82599
Working RAM:
View attachment 82600
Motherboard BK-1264NP-4L v1.4

Thanks for that, this is very similar if not the exact unit I almost went for but hearing a lot of people complain about overheating I went with the one that had the fan. I just tried looking up the MoBo with

dmidecode -t 2

but came up empty. Once the stick is in I'll look for that model # again, curious to see if we're on the same version.

You mentioned you had updated/unlocked BIOS? Is there a set of instructions you followed to unlock/update? I ordered a 32GB and at least according to a few places should be working fine.

 

[sammyke007]

I think the Crucial sometimes works and sometimes it doesn't... That's probably why the seller only advices to use SK Hynix or Samsung DDR5...
I've attached an USB powered 120mm fan to my system.
Uses +- 13W all together and temp. idles around 28° celcius.

 

[logikbarn]

I think the Crucial sometimes works and sometimes it doesn't... That's probably why the seller only advices to use SK Hynix or Samsung DDR5...
I've attached an USB powered 120mm fan to my system.
Uses +- 13W all together and temp. idles around 28° celcius.

Gotcha good to know! I did go back to my messages with the seller last night once more and I thought he had said that but didn't find anything so I must have seen that somewhere on the listing maybe.

Thanks for all the help, very curious to see how this pans out with the Samsung memory. It is not the 16GB but considering the seller said it supports 32GB hoping that won't be an issue. I'll update as soon as I have it installed.

 

[logikbarn]

Well, had pfSense running on it (no Proxmox) for 7 days straight without issue. No overheating, no hiccups just ran fine. The Samsung RAM module arrived yesterday and so I swapped out the RAM and NVME to the one that had Proxmox installed with pfSense as VM. It ran fine until exactly 11:15PM so ~6hrs from the moment of install give or take. It was late in the night so I went to bed. By 6:30AM it had rebooted another 3-4x which seems higher than usual. The RAM I purchased is the exact family you had but 32GB vs. 16GB.

I just put back the NVME with just pfSense (no Proxmox) and will let it run like this again. I'm going to wait until the latest microcode updates are available for Debian and will apply those to test as well but not a lot of hope. If I can confirm that this runs fine on other OS like Windows then I'll sell it on eBay and buy a Netgate appliance or something. Lesson learned, DO NOT buy these TopTon products.

Edit:

FWIW even with just pfSense installed I did run some memory intensive stress tests on it so that the usage would increase. I saw spikes in RAM usage similar to that on Proxmox. I was just curious on whether it was memory related still or something else. The unit did not experience any reboots during these tests.

 

[Zdolny]

I have BKHD G30W N100 I226
I tried many tricks, settings... Proxmox 8.3.5 - all time crash on this BKHD...

I have few some other PCs with N150 - everything works smooth.
Based on post - I decided to check...

Move from one of my PC with N150:

Memory Device
Array Handle: 0x003B
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: SODIMM
Set: None
Locator: Controller0-ChannelA-DIMM0
Bank Locator: BANK 0
Type: DDR5
Type Detail: Synchronous
Speed: 4800 MT/s
Manufacturer: Corsair
Serial Number: 00000000
Asset Tag: 9876543210
Part Number: CMSX64GX5M2A4800C40
Rank: 2
Configured Memory Speed: 4800 MT/s
Minimum Voltage: 1.1 V
Maximum Voltage: 1.1 V
Configured Voltage: 1.1 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 3, Hex 0x9E
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None

with my N100:

Memory Device
Array Handle: 0x0027
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: SODIMM
Set: None
Locator: Controller0-ChannelA-DIMM0
Bank Locator: BANK 0
Type: DDR5
Type Detail: Synchronous
Speed: 4800 MT/s
Manufacturer: Crucial Technology
Serial Number: E8E45FE2
Asset Tag: 9876543210
Part Number: CT32G48C40S5.C16A1
Rank: 2
Configured Memory Speed: 4800 MT/s
Minimum Voltage: 1.1 V
Maximum Voltage: 1.1 V
Configured Voltage: 1.1 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 6, Hex 0x9B
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

After moving Corsair DDR5 to N100 - all looks working perfectly fine now.
I put Crucial back to my N150 PC - and also it is working fine.

Simple swapping memory between 2 PCs resolve problem

 

[jiveman]

For me, reducing the RAM speed wasn't an option, even after flashing a modded/unlocked BIOS. Samsung DDR5 4800mhz running perfectly fine now.

What bios did you flash? I have a topton w/8505 having freezing issues. Wondering if this would work for me. I've never flashed aftermarket bios. Any docs you can point to?

 

[marcus01]

Hello everyone, my Proxmox started crashing at irregular intervals after I installed a 4x NVMe expansion board and a different BIOS version. The seller on Amazon provided me with the new BIOS, but it didn't work right away. After some research, I found the right BIOS for me. Since then, I haven't experienced any crashes.

Server: HUNSN RJ35 - Intel 12th Gen Core I3 N305
RAM: Crucial DDR5 RAM 32GB 4800MHz
1x NVMe for OS: Samsung 990 EVO NVMe M.2 SSD 1 TB
4x NVMe for Datastorage: Crucial P3 Plus SSD 2TB M.2 PCIe Gen4 NVMe

The exact device name is printed on a small sticker on the motherboard. The following link contains the BIOS for versions V1 and V2 of the device. This differentiates also whether you are working with or without an NVMe expansion board:
BIOS tested for V2 5x NVMe on CW-AL-4L
I don't know who uploaded these files but they worked for me, so please be careful.

The CWWK website also links to the following page to download the BIOS:
Link from CWWK (link appears to be currently unavailable)