Random system freeze on PN51-E1

[jeppe]

Hi @mira

I am running 5.15.83-1-pve #1 SMP PVE 5.15.83-1. Now running solid for 26 days

Yes I have the logs:

 ~ $ ls -l /var/log/apt/history.log*
-rw-r--r-- 1 root root    0 Feb  1 00:00 /var/log/apt/history.log
-rw-r--r-- 1 root root  499 Apr 19  2022 /var/log/apt/history.log.10.gz
-rw-r--r-- 1 root root 1834 Mar 29  2022 /var/log/apt/history.log.11.gz
-rw-r--r-- 1 root root  833 Jan 24 15:25 /var/log/apt/history.log.1.gz
-rw-r--r-- 1 root root  168 Jan  3 12:06 /var/log/apt/history.log.2.gz
-rw-r--r-- 1 root root  923 Dec  3 11:17 /var/log/apt/history.log.3.gz
-rw-r--r-- 1 root root  982 Oct 23 11:46 /var/log/apt/history.log.4.gz
-rw-r--r-- 1 root root  580 Sep  7  2022 /var/log/apt/history.log.5.gz
-rw-r--r-- 1 root root  522 Aug  8  2022 /var/log/apt/history.log.6.gz
-rw-r--r-- 1 root root  841 Jul 15  2022 /var/log/apt/history.log.7.gz
-rw-r--r-- 1 root root  479 Jun  6  2022 /var/log/apt/history.log.8.gz
-rw-r--r-- 1 root root  634 May 14  2022 /var/log/apt/history.log.9.gz

...and looking at the newest file I honestly get a bit uncertain about how long ago I did the most recent kernel upgrade; I can see my current kernel was installed Jan 15th, I installed 6.1 on Jan 17th and I removed the 6.x kernel and upgraded on Jan 24th - but the latest update was without kernel updates:

Start-Date: 2023-01-15  17:28:40
Commandline: apt dist-upgrade
Requested-By: jeppe (1000)
Install: pve-kernel-5.15.83-1-pve:amd64 (5.15.83-1, automatic)
Upgrade: libnftables1:amd64 (0.9.8-3.1, 0.9.8-3.1+deb11u1), pve-firmware:amd64 (3.5-6, 3.6-2), zfs-zed:amd64 (2.1.6-pve1, 2.1.7-pve2), libtasn1-6:amd64 (4.16.0-2, 4.16.0-2+deb11u1), zfs-initramfs:amd64 (2.1.6-pve1, 2.1.7-pve2), spl:amd64 (2.1.6-pve1, 2.1.7-pve2), libnvpair3linux:amd64 (2.1.6-pve1, 2.1.7-pve2), libproxmox-acme-perl:amd64 (1.4.2, 1.4.3), grub-pc-bin:amd64 (2.06-3~deb11u4, 2.06-3~deb11u5), libuutil3linux:amd64 (2.1.6-pve1, 2.1.7-pve2), libzpool5linux:amd64 (2.1.6-pve1, 2.1.7-pve2), proxmox-backup-file-restore:amd64 (2.3.1-1, 2.3.2-1), qemu-server:amd64 (7.3-1, 7.3-2), libpve-access-control:amd64 (7.2-5, 7.3-1), libproxmox-acme-plugins:amd64 (1.4.2, 1.4.3), base-files:amd64 (11.1+deb11u5, 11.1+deb11u6), proxmox-backup-client:amd64 (2.3.1-1, 2.3.2-1), distro-info-data:amd64 (0.51+deb11u2, 0.51+deb11u3), grub-efi-amd64-bin:amd64 (2.06-3~deb11u4, 2.06-3~deb11u5), grub2-common:amd64 (2.06-3~deb11u4, 2.06-3~deb11u5), pve-manager:amd64 (7.3-3, 7.3-4), nano:amd64 (5.4-2+deb11u1, 5.4-2+deb11u2), grub-common:amd64 (2.06-3~deb11u4, 2.06-3~deb11u5), pve-kernel-5.15:amd64 (7.2-14, 7.3-1), libzfs4linux:amd64 (2.1.6-pve1, 2.1.7-pve2), libksba8:amd64 (1.5.0-3+deb11u1, 1.5.0-3+deb11u2), libvirglrenderer1:amd64 (0.8.2-5, 0.8.2-5+deb11u1), grub-pc:amd64 (2.06-3~deb11u4, 2.06-3~deb11u5), pve-kernel-helper:amd64 (7.2-14, 7.3-1), zfsutils-linux:amd64 (2.1.6-pve1, 2.1.7-pve2), postfix:amd64 (3.5.13-0+deb11u1, 3.5.17-0+deb11u1), nftables:amd64 (0.9.8-3.1, 0.9.8-3.1+deb11u1)
End-Date: 2023-01-15  17:29:51

Start-Date: 2023-01-17  19:22:17
Commandline: apt install pve-kernel-6.1
Requested-By: jeppe (1000)
Install: pve-kernel-6.1:amd64 (7.3-2), pve-kernel-6.1.2-1-pve:amd64 (6.1.2-1, automatic)
End-Date: 2023-01-17  19:22:42

Start-Date: 2023-01-24  15:20:01
Commandline: apt remove pve-kernel-6.1
Requested-By: jeppe (1000)
Remove: pve-kernel-6.1:amd64 (7.3-2)
End-Date: 2023-01-24  15:20:01

Start-Date: 2023-01-24  15:21:53
Commandline: apt remove pve-kernel-6.1.2-1-pve
Requested-By: jeppe (1000)
Remove: pve-kernel-6.1.2-1-pve:amd64 (6.1.2-1)
End-Date: 2023-01-24  15:22:00

Start-Date: 2023-01-24  15:25:01
Commandline: apt dist-upgrade
Requested-By: jeppe (1000)
Upgrade: zfs-zed:amd64 (2.1.7-pve2, 2.1.7-pve3), zfs-initramfs:amd64 (2.1.7-pve2, 2.1.7-pve3), spl:amd64 (2.1.7-pve2, 2.1.7-pve3), libnvpair3linux:amd64 (2.1.7-pve2, 2.1.7-pve3), libpve-cluster-api-perl:amd64 (7.3-1, 7.3-2), libuutil3linux:amd64 (2.1.7-pve2, 2.1.7-pve3), libzpool5linux:amd64 (2.1.7-pve2, 2.1.7-pve3), pve-cluster:amd64 (7.3-1, 7.3-2), sudo:amd64 (1.9.5p2-3, 1.9.5p2-3+deb11u1), libzfs4linux:amd64 (2.1.7-pve2, 2.1.7-pve3), pve-kernel-helper:amd64 (7.3-1, 7.3-2), zfsutils-linux:amd64 (2.1.7-pve2, 2.1.7-pve3), libpve-cluster-perl:amd64 (7.3-1, 7.3-2)
End-Date: 2023-01-24  15:25:28

This is the last update in the history.log.1.gz - the history.log file is empty.

/Jeppe

 

[cpc0307]

Maybe I should give it a go again

I've got no luck on my PN51-E1.
Still failing with 5.15.85-1 after a few hours.

 

[keldbroe]

Same problem here.
It still crashes.

 

[jeppe]

Mine is still running - but I got the below concerning message on the console when I looked today. Proxmox and all virtual servers survived and are now on day 33 wrt. uptime! Any clues what these messages mean, @mira? Would be cool for the Proxmox "community" of PN51 users if we got to the bottom of this

 

[cpc0307]

I've also tried the latest 6.2 kernel
Which does not improve the situation

I wonder what's the difference between E1 and S1
Maybe that's why @jeppe 's machine works without any hitches?

 

[hahell]

I have the PN51-E1 and has been testing the 6.1.14-1 kernel the last 2 week. I managed to squeeze the uptime to over 9 days before it crashed without any traces in the syslog. Currently running on the 6.1.15-1 kernel.

This problem started for me when I did an update around the last week of October last year (been running dead stable before that) and has been an constant pain in the a... since. I really hope we can find a solution since I love using Proxmox and doesn't want to switch to another platform.

 

[mira]

Mine is still running - but I got the below concerning message on the console when I looked today. Proxmox and all virtual servers survived and are now on day 33 wrt. uptime! Any clues what these messages mean, @mira? Would be cool for the Proxmox "community" of PN51 users if we got to the bottom of this

View attachment 48081

Looks like some tasks are hanging, could even be because of a process in D state.

Look through the output of ps auxwf for any of those.

How is the I/O wait?

 

[mira]

I've also tried the latest 6.2 kernel
Which does not improve the situation

I wonder what's the difference between E1 and S1
Maybe that's why @jeppe 's machine works without any hitches?

Which exact model do you have? @jeppe mentioned theirs already in the thread, but I couldn't find any exact model for one of those E1s.

 

[hahell]

Which exact model do you have? @jeppe mentioned theirs already in the thread, but I couldn't find any exact model for one of those E1s.

I know that you are not asking me, but I'll answer anyway
My exact model number is as follows:
ASUS PN51-BB757MDE1

 

[cpc0307]

Which exact model do you have? @jeppe mentioned theirs already in the thread, but I couldn't find any exact model for one of those E1s.

Hi @mira
I'm using PN51-E1-57UBPAA

 

[jeppe]

Looks like some tasks are hanging, could even be because of a process in D state.

Look through the output of ps auxwf for any of those.

How is the I/O wait?

My model is a PN51-BB757MDS1.

The host has no processes in state D, and the IO wait is at zero (it’s a home server, not very busy, but still crashed within a few days until a month ago).

The messages were some days ago and haven’t repeated.

I am now on 35 days uptime. Maybe it’s time for an apt update/distupgrade soon…

/jeppe

 

[j0nas]

I have the exact same problem!

But for me it started for some months ago, before that it ran smoothly for months on Proxmox 7.1 if I remember correctly. I even tried to downgrade to that version without any success.

My network is flooded every time my machine hangs, and a hard reboot is the only thing that works. Or pull the CAT cable which instantly fixes the network issues for every device! The keyboard is unresponsive and the screen will be reverted to the login screen (bash) when connected to a monitor. I even experienced that the machine would not register NVME or SSD drive in BIOS at one time. I have tried with two different NVME drives, and two different SSD drives. I have not changed the RAM which is non-EEC and I have not tested them with memtest86+.

BIOS version is 0505
MODEL is PN51-E1
RAM Kingston DDR4 64GB

edit:
I upgraded to linux kernel 6.2 yesterday. Still problems, either it reboots, or it hangs on the login screen and I have to manually reboot the machine. Maybe I have to give ut Proxmox for now and try some other distros without using VMs, such as Kubernetes or docker...

 

[keldbroe]

I have the exact same problem!

But for me it started for some months ago, before that it ran smoothly for months on Proxmox 7.1 if I remember correctly. I even tried to downgrade to that version without any success.

My network is flooded every time my machine hangs, and a hard reboot is the only thing that works. Or pull the CAT cable which instantly fixes the network issues for every device! The keyboard is unresponsive and the screen will be reverted to the login screen (bash) when connected to a monitor. I even experienced that the machine would not register NVME or SSD drive in BIOS at one time. I have tried with two different NVME drives, and two different SSD drives. I have not changed the RAM which is non-EEC and I have not tested them with memtest86+.

BIOS version is 0505
MODEL is PN51-E1
RAM Kingston DDR4 64GB

120% same history here... First it was fine and then the network flooding.

I even tried with a USB-C NIC - it flooded the network even after I turned off the computer (power was still on the USB port) - only after unplugging the NIC the network recovered.

 

[j0nas]

Ok, so I installed ubuntu server without Proxmox and I startet the same services as with the proxmox VMs and LXC; Kubernetes. And behold, it did not take a long time before the entire machine froze, I was able to capture the screen, but I do not know what it means and if its relatable to what's happening with proxmox.

Sorry for the potato quality

 

[kamau]

For me, it's the same game. I landed here via a Google search. So I am not alone, although I have not installed proxmox, but Arch Linux. So this is how you end up saving energy with a mini PC....

After many freezes and even more frustration, I discovered another problem:
7za a /.../file.7z /directory/with/very/many/files/ -t7z -mx=9 -ms=off
Information: The directory and its subdirectories are 270GB in size.
After some time an error occurs, the best was so far at 20%.

I would be interested if any of you can reproduce this. My hope is that the freeze and the 7zip error have the same cause, so you don't have to wait so long to check the success of a change.

 

[Doomi]

Just tried with the new beta of 8.0, but still have the same issue as before. :-/

 

[keldbroe]

So frustrating.
I've installed Windows 11 and have more than 15 days of runtime currently.

 

[hahell]

So frustrating.
I've installed Windows 11 and have more than 15 days of runtime currently.

Totally agree! I had to buy an Intel NUC (no issues whatsoever) because these random crashes was driving me mad. Currently the Asus PN51 sits on the shelf, useless!!

 

[cpc0307]

So frustrating.
I've installed Windows 11 and have more than 15 days of runtime currently.

Also switched to Win10 with Hyper-V enabled. With a 2-month uptime right now, zero crashes/freeze.

 

[albertpruis]

Totally agree! I had to buy an Intel NUC (no issues whatsoever) because these random crashes was driving me mad. Currently the Asus PN51 sits on the shelf, useless!!

If I may ask, which Intel NUC did you go with? I understand no issues at all with that Intel NUC? I am considering buying one myself so that's why I am asking.