[0x08000800]: an uncorrected error caused a data fabric sync flood event

zyghom said:
I solved this issue with kernel parameters:
Code: 
processor.max_cstate=1

apparently this problem started with PVE kernel update (not sure which one, probably latest as I have never had this problem before)

my machine:
UM790 Pro Ryzen 9 7940HS
16 x AMD Ryzen 9 7940HS w/ Radeon 780M Graphics (1 Socket)
Click to expand...
it work with my p620 workstation, but standby power consumption has increased by 30W.
 
Howdy folks,

Just registered to post here, since I've been actively troubleshooting this issue for a few weeks and finally reached stability. Trying to provide as much info as possible so Google can find this thread for others.

Hardware
Bosgame P4 (Ryzen 7 5825U), BIOS EffiZen 1.06 (tested with 1.04 as well), PVE 9 with kernel 7.0.14-25-pve, ZFS mirror on 2× NVMe.

Issue
Random resets with Previous system reset reason [0x08000800]: an uncorrected error caused a data fabric sync flood event. They started within a few eeks of each other, then daily and then eventually to every few minutes.

What I ruled out:

  • RAM: two different 32 GB kits, each module tested alone, and Memtest86+ passes. Both kits also run fine in another machine.
  • Power brick, cooling, BIOS update and NVMe drive: none of these made a difference.
What I found:
  • It only crashes when the system is idle. Under constant load (I moved my containers to another box and created a stress-ng script with a continuous VAAPI decode component) it ran stable for days at a time.
  • With the amd-pstate-epp driver, the governor is the trigger. On performance it’s stable. Switching to powersave on a running system reproduces the sync flood within minutes. I tested this with each RAM module on its own, and both behaved the same.
  • The governor setting alone wasn’t enough on a fresh install. I also needed the two parameters I’d added during earlier testing.
The kernel parameters that are stable for me (24h+ idle, now running real workloads), in /etc/kernel/cmdline:

cpufreq.default_governor=performance processor.max_cstate=1 nvme_core.default_ps_max_latency_us=0

then proxmox-boot-tool refresh.

Comments:
  • This is a workaround, not a root-cause fix. It costs a few watts at idle (3 to 5 from what my UPS measured).
  • I haven’t tested which of the last two parameters is actually needed. processor.max_cstate=1 alone, with powersave, did not help.
  • To check your own system: cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver and cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor. If you have a cron job or tool setting powersave, that may be your trigger too.
Hopefully this helps somebody out there.

Cheers!
 
I have 2 machines:
Code: 
 cat /proc/cpuinfo | grep 'model name'
model name    : AMD Ryzen 9 7940HS w/ Radeon 780M Graphics

and:
Code: 
cat /proc/cpuinfo | grep 'model name'
model name    : AMD Ryzen 7 7735HS with Radeon Graphics

and on both it is:
Code: 
 cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance

and no more issues
I don't have "idle" status, I am running on both with 20+ VMs and LXC - nothing is idle there anymore ;-)
 
zyghom said:
I have 2 machines:
Code: 
 cat /proc/cpuinfo | grep 'model name'
model name    : AMD Ryzen 9 7940HS w/ Radeon 780M Graphics

and:
Code: 
cat /proc/cpuinfo | grep 'model name'
model name    : AMD Ryzen 7 7735HS with Radeon Graphics

and on both it is:
Code: 
 cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance

and no more issues
I don't have "idle" status, I am running on both with 20+ VMs and LXC - nothing is idle there anymore ;-)
Click to expand...
After a doing a fresh install, my default scaling governor was powersave, not performance.

This has been a pretty wild ride for me, and I was able to pinpoint exactly the "idle" issue with the synthetic stress tests. The moment I stopped them, it would crash without couple minutes.

Glad you're not experiencing the issue anymore!
 
flood_sync said:
After a doing a fresh install, my default scaling governor was powersave, not performance.

This has been a pretty wild ride for me, and I was able to pinpoint exactly the "idle" issue with the synthetic stress tests. The moment I stopped them, it would crash without couple minutes.

Glad you're not experiencing the issue anymore!
Click to expand...
are you sure it does not follow up what is in the BIOS?
I remember in BIOS I set to performance as well
 
flood_sync said:
Not sure, to be completely honest with you. Don't remember what the BIOS settings were, all changes I made were in software.
Click to expand...
maybe worth checking
I was today relocating servers and had to reboot them, and went to BIOS and just remember I saw it there
both machines are minisforum, 1st is um790 pro and 2nd is um773 lite
both have very limited BIOS
 
Started having this problem after upgrading to Kernel 7. I've tried a lot of the fixes posted here and here are some of my observations and details. I'm on a Minisforum N5 NAS, very limited bios options as others have stated and is on the latest BIOS version.

Code: 
cat /proc/cpuinfo | grep 'model name'
model name      : AMD Ryzen 7 255 w/ Radeon 780M Graphics

Code: 
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance

Code: 
cat /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet processor.max_cstate=1 pcie_aspm=off nvme_core.default_ps_max_latency_us=0 amdgpu.ppfeaturemask=0xfff73fff pcie_ports=native"

In the latest I have tried max_cstate=2 and adding additional lines as the problem persisted. As FloodSync has stated, it only crashes in Idle is largely true for me as well. If after a fresh reboot and I do not login to any of my linux VM's or reconnect drive mappings, essentially just let the host idle I've seen uptime up to three days. However if I reconnect shares on the VMs or a single VM and let it idle, it will crash in as little as 8 hours. I don't want to reinstall my host and continue to face the same issue so for now I've rolled back to Kernel 6.17 and am going to observe and remove the added lines from my Grub to see how things go.

Has anyone else rolled back to Kernel 6 and seen the problem go away?
 
You must log in or register to reply here.
Top Bottom
Back