Hi everyone,
I am troubleshooting recurring hard system freezes on my Proxmox VE node during heavy local AI workloads (Ollama/LLM inference) and would appreciate any insights into potential kernel, BIOS, or PCIe/chipset bottlenecks.
Symptoms:
Hardware Specifications:
Troubleshooting steps already taken:
Questions for the community:
Thanks in advance for any advice!
I am troubleshooting recurring hard system freezes on my Proxmox VE node during heavy local AI workloads (Ollama/LLM inference) and would appreciate any insights into potential kernel, BIOS, or PCIe/chipset bottlenecks.
Symptoms:
- Under sustained multi-GPU AI load, the host completely locks up in a "zombie state".
- Fans and LEDs stay on, but the system is completely unresponsive over SSH, WebUI, and local console.
- Nothing is written to
journalctlat the moment of the crash—no Kernel Panic, no OOM killer, no PCIe AER errors. The log simply cuts off until the next boot. - A normal reboot is impossible; I have to hold the power button for 4+ seconds and wait a few seconds before powering it back on.
Hardware Specifications:
- Motherboard: ASUS Prime Z690-P D4 (no physical PCIe bifurcation; top slot is CPU-bound x16, secondary slots are routed through the Z690 chipset/DMI 4.0)
- CPU: Intel Core i5-12600K
- RAM: 128 GB DDR4 (4x32 GB)
- GPU 0 (Top PCIe 5.0 x16 slot): Gainward RTX 3090 24GB
- GPU 1 (Chipset PCIe slot): ASUS TUF RTX 3060 12GB
- Storage: 3x NVMe SSDs
- PSU: Corsair RM1000x (1000W)
- OS/Kernel: Proxmox VE (Kernel 6.14.11-9-pve), NVIDIA Driver 590.48.01 / CUDA 13.1
Troubleshooting steps already taken:
- Disabled C-states & ASPM: Added
intel_idle.max_cstate=1 processor.max_cstate=1 pcie_aspm=offto GRUB. Verified active, yet the system still froze under load. - Power Limiting: Capped the RTX 3090 to 250W via a systemd service (
nvidia-smi -pl 250) to mitigate transient power spikes. - Current Isolation Test: I have now shut down the LXC/service utilizing the secondary RTX 3060 (leaving GPU 1 idle at ~6W) to see if offloading the Z690 chipset/DMI bus and reducing motherboard PCIe slot power draw stabilizes the node.
Questions for the community:
- Has anyone experienced similar silent bus/chipset deadlocks on consumer Z690 boards when pushing heavy DMA traffic through chipset-routed PCIe slots alongside 3x NVMe drives?
- Are there specific BIOS settings (e.g., forcing PCIe Link Speed to Gen3, Above 4G Decoding, Resizable BAR, or PCH voltage/ASPM tweaks) or kernel parameters (like
nvme_core.default_ps_max_latency_us=0or IOMMU flags) that could prevent the DMI/PCIe bus from locking up? - Or am I simply hitting a hard electrical/architectural limit of the ASUS Prime Z690-P D4 with this configuration (4x32GB RAM + dual GPUs + 3x NVMe)?
Thanks in advance for any advice!