Hard system freezes (no kernel panic) under multi-GPU AI workload on Z690 (RTX 3090 + RTX 3060)

Feb 21, 2023
49
2
13
Stockholm, Sweden
Hi everyone,

I am troubleshooting recurring hard system freezes on my Proxmox VE node during heavy local AI workloads (Ollama/LLM inference) and would appreciate any insights into potential kernel, BIOS, or PCIe/chipset bottlenecks.

Symptoms:
  • Under sustained multi-GPU AI load, the host completely locks up in a "zombie state".
  • Fans and LEDs stay on, but the system is completely unresponsive over SSH, WebUI, and local console.
  • Nothing is written to journalctl at the moment of the crash—no Kernel Panic, no OOM killer, no PCIe AER errors. The log simply cuts off until the next boot.
  • A normal reboot is impossible; I have to hold the power button for 4+ seconds and wait a few seconds before powering it back on.

Hardware Specifications:
  • Motherboard: ASUS Prime Z690-P D4 (no physical PCIe bifurcation; top slot is CPU-bound x16, secondary slots are routed through the Z690 chipset/DMI 4.0)
  • CPU: Intel Core i5-12600K
  • RAM: 128 GB DDR4 (4x32 GB)
  • GPU 0 (Top PCIe 5.0 x16 slot): Gainward RTX 3090 24GB
  • GPU 1 (Chipset PCIe slot): ASUS TUF RTX 3060 12GB
  • Storage: 3x NVMe SSDs
  • PSU: Corsair RM1000x (1000W)
  • OS/Kernel: Proxmox VE (Kernel 6.14.11-9-pve), NVIDIA Driver 590.48.01 / CUDA 13.1

Troubleshooting steps already taken:
  1. Disabled C-states & ASPM: Added intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=off to GRUB. Verified active, yet the system still froze under load.
  2. Power Limiting: Capped the RTX 3090 to 250W via a systemd service (nvidia-smi -pl 250) to mitigate transient power spikes.
  3. Current Isolation Test: I have now shut down the LXC/service utilizing the secondary RTX 3060 (leaving GPU 1 idle at ~6W) to see if offloading the Z690 chipset/DMI bus and reducing motherboard PCIe slot power draw stabilizes the node.

Questions for the community:
  • Has anyone experienced similar silent bus/chipset deadlocks on consumer Z690 boards when pushing heavy DMA traffic through chipset-routed PCIe slots alongside 3x NVMe drives?
  • Are there specific BIOS settings (e.g., forcing PCIe Link Speed to Gen3, Above 4G Decoding, Resizable BAR, or PCH voltage/ASPM tweaks) or kernel parameters (like nvme_core.default_ps_max_latency_us=0 or IOMMU flags) that could prevent the DMI/PCIe bus from locking up?
  • Or am I simply hitting a hard electrical/architectural limit of the ASUS Prime Z690-P D4 with this configuration (4x32GB RAM + dual GPUs + 3x NVMe)?

Thanks in advance for any advice!
 
while I can't say exactly what the reason is, maybe some more troubleshooting could help:

i'd try to open an ssh session and monitor the log, cpu usage, temps, etc. and try to trigger it. sometimes there can be a hint written out to the journal that does not reach the disk, but since you also tried the local monitor i'd say that won't show anything new.

you didn't write explicitly: did any of the troubleshooting steps show any difference?

if not, checking the temps (also for the chipset for instance) can make sense

aside from that, i have often seen very weird behavior when the power supply is faulty or "incompatible" with the system (i have seen certain cpu/psu combos not simply not wanting to work together properly)