PVE keeps rebooting - Beelink U59

[mzac23]

Hi, new to the forum and new to proxmox here but longtime VMWare user.

I purchased 2 Beelink U59 Celeron computers to replace my aging 12 year old VMware server and installed Proxmox on them. I was able to migrate over all my VMs but I keep running into issues with both of them.

I keep getting random reboots of VMs, VMs hanging and also Proxmox rebooting itself. I can't find anything in the kernel logs or syslogs that would indicate why they crashed and rebooted.

The computers have:
- https://www.amazon.ca/gp/product/B09J4D6TMG
- Intel N5095 Celeron CPU
- 16gb DDR4 2933 MHz Memory
- 512gb M.2 SSD
- Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
- AMI Bios
- 12v 3A Power Supply (came with it)

I have tried tweaking some settings in the BIOS on one of them such as disabling Turbo mode and so far so good, not sure if this is really the culprit.

I read some posts suggesting to try using memtest86 however I get a black screen when trying to boot with it.

Does anyone have any suggestions for me to try? It is getting really annoying having my VMs and hypervisors reboot multiple times per day...

root@pve1:/var/log# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           156
Model name:                      Intel(R) Celeron(R) N5095 @ 2.00GHz
Stepping:                        0
CPU MHz:                         2000.000
CPU max MHz:                     2000.0000
CPU min MHz:                     800.0000
BogoMIPS:                        3993.60
Virtualization:                  VT-x
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        1.5 MiB
L3 cache:                        4 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Vulnerable: No microcode
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss
                                 ht tm pbe syscall nx rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpui
                                 d aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 s
                                 se4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault epb cat_l2 cdp
                                 _l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep e
                                 rms rdt_a rdseed smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm
                                 arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req umip waitpkg gfni rdpid movdiri movdir64b md_cl
                                 ear flush_l1d arch_capabilities

root@pve1:/var/log# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-3-pve: 5.15.35-6
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-5
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[mzac23]

Even after disabling turbo mode my system rebooted at 4am this morning. Nothing in the logs at all...

 

[Neobin]

I purchased 2 Beelink U59

• Did you test both with PVE? Do both behave the same?
• Did you test the power supply from the other one on the problematic one?
• Is the bios up-to-date?
• Did you run an Memtest already?
• Did you test with another linux distribution? And stress test all components there?
• Did you test with an Windows installation? And stress test all components there?

At least this is what I would start with to narrow down, if there is a (PVE-)specific or an general problem with the device.

 

[ness1602]

IF there is a black screen with memtest, try different memory, memtest should always at least start, sometimes freeze right after.

 

[Neobin]

IF there is a black screen with memtest, try different memory, memtest should always at least start, sometimes freeze right after.

I read some posts suggesting to try using memtest86 however I get a black screen when trying to boot with it.

Missed that one.
If the PVE-integrated memtest does not work, you can also try with an "standalone" one created on an USB-stick by yourself.
Maybe also try altering between UEFI- and legacy-boot for the memtest(s).

 

[mzac23]

Hi, thanks for the help. I was finally able to get Memtest86+ working with the daily build, seems there have been some UEFI issues.

That said:

- Both my Beelink pve machines are experiencing the same issue
- Both have the same kind of power supply, I might try and find a bigger 12v power supply (more amps) with the right connector
- BIOS is up to date, no new versions supplied by Beelink yet
- Memtest ran and passed
- Did not test another OS yet

All that said, I also wonder about the VMs and if they could be causing the issue. I am running kubernetes on Ubuntu VMs and they are running Metallb which advertises additional IP addresses on my L2 network. I doubt this would cause any issues but thought it would be worth mentioning.

I have now disabled turbo on my sencond Beelink as the first one seems to be more stable since I disabled it. Will let it run a few more days and see how it goes.

 

[SecretMineDE]

I also had issues trying to run Proxmox on the Beelink U59 a few months ago. VMs kept crashing after random time intervals (sometimes 5 minutes, sometimes 1-2 days). I have also tested swapping drives or using a 2.5 inch SSD, same problem.

In the end I sent the thing back. Seems to be an issue with the unit or some weird configuration from the manufacturer.

 

[dbri]

I'm having the same issue on this device, did you found how to fix it?

 

[SecretMineDE]

I tried many things like changing the SSD, changing memory, changing power and frequency related cpu settings in the bios (I dont know what exactly) - nothing solved my problems.
As I said, I sent the device back to the seller.

 

[mzac23]

I wrote to Beelink, they said I have the latest bios and to try and re-install windows to see if it makes things better.... seriously....

I'm pretty sure we can find a solution to this issue. What is strange is that the VMs and the host rebooting seem to have nothing to do with the amount of load the system is under. I can sometime be transcoding with Plex with no issues and it doesn't reboot. Then when it is doing nothing, it just reboots.

I'm testing one of the boxes without running any VMs on it and so far it has been stable for many days without rebooting. Seems only when running VMs that it reboots.

 

[SecretMineDE]

Well, in my case the unit itself didnt reboot, but VMs crashed at random with a kernel panic (and I found nothing interesing in the logs). Also independed of their load, time ranged between a few minutes and a few days. Keep in mind, I tested the Beelink only for a few days and sent it back as soon as it was clear that my VMs were not the issue.

 

[mzac23]

Well, in my case the unit itself didnt reboot, but VMs crashed at random with a kernel panic (and I found nothing interesing in the logs). Also independed of their load, time ranged between a few minutes and a few days. Keep in mind, I tested the Beelink only for a few days and sent it back as soon as it was clear that my VMs were not the issue.

I have a feeling that somehow it is the VMs causing the issue. If I run the system without any VMs running it stays very stable. But as soon as I try and run any VMs it goes nuts.

I'm giving up on trying proxmox on this beelink for now, I will run straight Ubuntu with K3S as part of my cluster and see if it is stable.

 

[mzac23]

I've been running straight Ubuntu now for 5 days without any reboots. I guess proxmox and VMs is the problem..

 

[stevekdavis]

@mzac23 did you find a resolution to this? I got a U59 and put proxmox on and and am seeing the same random VM lockups. I may cut my losses and send it back if you’re still seeing the same issue.

 

[mzac23]

@mzac23 did you find a resolution to this? I got a U59 and put proxmox on and and am seeing the same random VM lockups. I may cut my losses and send it back if you’re still seeing the same issue.

Nope.. I gave up on Proxmox running on these and just run straight Ubuntu now running K3S for my kubernetes cluster.

 

[toonvanvr]

I was considering buying an Odroid H3+ with N6005 or N5105 and I've seen mentions of intel C-state issues. I don't know the details, but your symptoms sound similar. No clear solution, but disabling C-states or changing kernel version seems to have helped for some.