[SOLVED] Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler

[VGE]

Hi guys,

This my first post so I hope im doing it correct.

The last couple weeks Ive been testing Proxmox on my old Intel I7 2600 and got it working perfectly no problems etc.. All this to use Proxmox on my low power machine which was running Ubuntu-server but was idle for 90% of its resources. To learn and use the system more I wanted to start with Proxmox. It will be used to have a 24/7 Ubuntu VM running and for distro testing so it doesn't need a lot of power.

The hardware of the low-power machine is:
CPU - ASROCK J4105M (Intel J4105)
RAM - 2x 4GB

PCIE cards
intel 1gbit NIC
delock 70137 (2x SATA expansion, no raid configured etc)

SSD's
2x Kingston A400 120GB
1x Samsung 850 EVO 500GB

I installed Proxmox on the two 120GB SSD's using the ZFS Mirror and configured the 500GB to be single ZFS disk for the VM's. The mobo has only two SATA ports so thats why I added the delock PCIE-card using only 1 of the two SATA ports for now.

Install goes smooth and i can boot op Proxmox with no problem. Added the non-licensed version and update/dist-upgrade. To test stuff out I created two VM's Ubuntu and PopOS and they both start up and install like normally would.

I started doing other stuff and came back to a locked Proxmox which was not accessible trough ssh or the web GUI. Fortunately had a monitor connected and saw this message: Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler

Picture:


After a lot of research I found it could be a BIOS or microcode problem with Intel CPU. So I checked my bios and it was already on the latest, so I updated the microcode adding:

deb http://ftp.us.debian.org/debian/ stretch main contrib non-free deb http://security.debian.org/ stretch/updates main contrib non-free

to:

/etc/apt/sources.list

and ran a update and distro-upgrade.

Unfortunately it didn't help and i'm still getting the same message with a system which is not accessible. Only solution is hard-stop with holding the power-button.

Any tips or tricks I can do to fix my problems?

Let me know if you need more info.

 

[aaron]

Hmm, tricky problem.

Have you reset the BIOS settings to defaults and then enabled the important ones? (VT-x, VT-d..)

You could try to disable power save states in the BIOS is possible.

You could also run a memory test to rule it out as the problem.

 

[VGE]

Hmm, tricky problem.

Have you reset the BIOS settings to defaults and then enabled the important ones? (VT-x, VT-d..)

You could try to disable power save states in the BIOS is possible.

You could also run a memory test to rule it out as the problem.

Hi Aaaron,

Thanks for the quick reply!

VT-x and -d are both on. I'm now trying to run with powestates off and first do a mem test but when I plug in the Proxmox usb I can select the memtest but nothing happens... I get a black popup screen with no infromation and the system is unusable. Could this be my problem?

 

[aaron]

The included memtest in the Proxmox ISO is a bit finicky :/. Creating a dedicated memtest86 usb flash drive should work much better.

 

[VGE]

The included memtest in the Proxmox ISO is a bit finicky :/. Creating a dedicated memtest86 usb flash drive should work much better.

Hi Aaron,

First pass of the Memtest86 usb has 0 error count. Will let it run for the rest of the day to be sure. I disabled all powersaving options in bios. Any other tips?

 

[aaron]

You could also try to install the newer and not yet default kernel 5.4. Maybe that helps as well.

apt update && apt install pve-kernel-5.4

Other than that I would check the /var/log/syslog* and /var/log/kern.log* to see if there are any more details available that might lead you to the root cause.

 

[VGE]

You could also try to install the newer and not yet default kernel 5.4. Maybe that helps as well.

apt update && apt install pve-kernel-5.4

Other than that I would check the /var/log/syslog* and /var/log/kern.log* to see if there are any more details available that might lead you to the root cause.

I tried to look at the syslog and kernlog but its a lot of information which I dont understand. It did crash again but the last message was: usb 1-7.1 reset high-speed USB device number 5 using xhci_hcd also the IO wait was nearly maxed out.

Any tips on how to read the log? Because it freezes up as soon as it needs to do work.

EDIT 6-may: im zeroing it down to the PCIE adapter + SSD. I kept the system running for 24 hours with no VMs or tasks running without a issue but as soon as I try to start a VM or make a backup of a VM it freezes and crashes. I have VMs running on the 500GB SSD. This for me would suggest that its somewhere in that part that's failing.

Could it be the controller not handling ZFS? Will try to work with other filesystem.

 

[VGE]

So I think I fixed it by buying a better pcie card. I exchanged the Delock 70137 for the newer Delock 89395 which on its product page says it supports SSD. After installing this card and reformatting the SSD everything works perfectly.

This is message is to thank you for your help and maby help others in the future!

 

[aaron]

So I think I fixed it by buying a better pcie card. I exchanged the Delock 70137 for the newer Delock 89395 which on its product page says it supports SSD. After installing this card and reformatting the SSD everything works perfectly.

This is message is to thank you for your help and maby help others in the future!

Thank you for the feedback! Unfortunately Delock doesn't mention which chipset they use for the 70137 :-/

Could you please mark the thread as solved by editing the first post and selecting the "Solved" prefix next to the title?

 

[VGE]

Thank you for the feedback! Unfortunately Delock doesn't mention which chipset they use for the 70137 :-/

Could you please mark the thread as solved by editing the first post and selecting the "Solved" prefix next to the title?

Done