PVE 9.1/9.2 installer + kernel 7.0 fail on Intel H97 SATA — 9.0-1 and kernel 6.14 work!

[DerPaddy]

Preface:
I just registered to share my almost week long suffering chasing imaginary hardware issues and the very simpel software bug/limitation that caused it.
A few weeks ago I used Claude.ai to cobble together a comprehensive manual to get Proxmox and Home Assistant running on a HP ProDesk 600 G2 MT Intel Core i5-6500. This went great and worked for a few hours until the PSU died and I switched to a completely new system of used hardware from ebay. (None of the ProDesk components are standard issue!)
Then the nightmare started that lead me to throw out a (from what I know now was a perfectly good) Mushkin Helix M.2 SSD in frustration, before getting the same error with a completely different SSD.

The following was written up by Claude so I don't get PTSD going through all the troubleshooting chats I had with it:




TL;DR: On an Intel 9-series (H97) board, the 9.1/9.2 installer fails at "make system bootable" (unable to install initramfs / read-only fs), and once you get onto pve-kernel 7.0.14-5-pve you get constant ICRC ABRT / SATA link resets under small bursty writes. pve-kernel 6.14.8 and 6.14.11 are completely stable on the exact same hardware. Install via the older 9.0-1 ISO, upgrade, then pin the 6.14 kernel.

---

Background / why I trusted the hardware first

I previously ran Proxmox on an HP ProDesk 600 G2 with zero storage issues for several hours and was starting to set up Home Assistant — until its PSU died. That's the only reason I moved to the H97 build. Because the ProDesk had been so reliable, I did not initially trust the used H97 hardware and spent a lot of time chasing a hardware fault (see below) before the real cause turned out to be software.

My symptoms match this Reddit thread almost exactly: https://www.reddit.com/r/Proxmox/comments/1rbop49/installation_fails_please_help/ — same class of board, same failure.

---

Hardware used across all attempts

• Board (final): ASRock H97M Pro4 — Intel H97, LGA1150
• Board (previous, swapped out): ASUS H97M-PLUS — same H97 chipset
• CPU: Intel i7-4790S (Haswell)
• RAM: 32 GB DDR3-1600 (4×8 GB Corsair Vengeance), XMP off
• System SSD: Crucial BX500 240 GB (SATA) → /dev/sda
• VM SSD: Samsung 850 EVO 500 GB (SATA) → /dev/sdb
• PSU: ENDORFY Vero L6 550 W
• 2× different SATA data cables, different batches
• Previous host (no issues until PSU death): HP ProDesk 600 G2 MT

---

What went wrong, in order

1. Current 9.2 ISO on the H97 board: the board wouldn't even list the USB stick as a UEFI boot entry (Legacy USBentry only, no UEFI: entry), regardless of CSM setting. Stick written with Etcher, checksum verified, tried multiple ports.
2. When I forced it through, install failed at the end with either unable to install initramfs or a read-only filesystem error — never completed.
3. I swapped the whole motherboard (ASUS → ASRock). Same failure. (In hindsight: same chipset, so this proved nothing.)

The hardware wild goose chase (all came back clean):

• Memtest86+ — 2 full passes, all 4 modules, no errors
• BX500 SMART — Reallocated / Pending / Uncorrectable all 0, 100% lifetime
• dd 20 GB raw write to the BX500 — clean, no kernel errors
• 2 different SATA cables (different batches) — no change
• SATA port swap — no change
• SATA Aggressive LPM (already disabled), CPU C-states (disabled C6/C7, then also C3) — no change
• libata.force=noncq and noncq,nolpm at install time — no change

None of it helped, because none of it was the cause.

---

What actually fixed the install

Installing from the older proxmox-ve_9.0-1.iso (good luck finding it online!) worked first try — and notably, 9.0-1 did present a proper UEFI boot entry on the same board where 9.1/9.2 did not. After install, apt update && apt full-upgrade to current (9.2.4) completed with no errors. So the install-time failure is specific to the 9.1/9.2 installer environment, not to running the resulting system.

Multiple others report the same "9.0-1 works, 9.1/9.2 don't" in forum thread 180437.

---

The runtime bug — cleanly isolated to a kernel regression

After upgrading, I started getting storage errors on the running system under small bursty writes (a simple apt install was enough — but a sustained 8-thread dd stress test ran clean, which is what threw me off). Typical log:

ata2.00: error: { ICRC ABRT }
ata2: hard resetting link
sd 1:0:0:0: [sda] Add. Sense: Scsi parity error
I/O error, dev sda, ... op WRITE
EXT4-fs (dm-1): failed to convert unwritten extents ... potential data loss!

…ending in ext4 going emergency_ro.

I have both older kernels still installed, so I tested the same hardware, same trigger (apt install), across three kernels:

Kernel	Result
6.14.8-2-pve (the 9.0-1 installer kernel)	clean
6.14.11-9-pve	clean
7.0.14-5-pve (current)	fails reproducibly

So this is a regression somewhere between 6.14.11 and 7.0.14 in the libata/AHCI path for the Intel 9-series controller — not a hardware fault. Every single ICRC/parity/link-reset event in my logs happened only on 7.0.14.

What's odd: the kernel already knows this drive is quirky. It applies the built-in quirk on every boot:

ata1.00: Model 'CT240BX500SSD1' ... applying quirks: nolpm
ata1.00: LPM support broken, forcing max_power

That nolpm quirk for the exact model CT240BX500SSD1 has been in libata-core.c since May 2024. So the quirk is present and applied on 7.0.14 too — yet 7.0.14 still fails, while 6.14.x with the same quirk is fine. Whatever regressed, it's not the quirk being missing. (That last point is my interpretation — I haven't bisected the kernel changelog between the two versions.)

---

Workaround (working, stable)

1. Install from proxmox-ve_9.0-1.iso
2. apt full-upgrade to current
3. Pin the good kernel:

proxmox-boot-tool kernel pin 6.14.11-9-pve
proxmox-boot-tool refresh
reboot

Stable since! New kernels can still install; 6.14.11 stays the boot kernel until unpinned.

---

Questions for anyone who knows the internals

• Is the 9.1/9.2 installer failure and the 7.0.14 runtime failure the same root cause, just triggered in different places?
• Anyone with an Intel 8/9-series board who can confirm 7.0.x vs 6.14.x on bursty SATA writes?
• Is this worth a bugzilla report, and does anyone have the changelog diff between 6.14.11 and 7.0.14 for the ahci/libata path?

 

[DerPaddy]

#########
Update — narrowed down: it's not 7.0-only. 6.17 fails too; 6.14 is solid under sustained load.
#########

Wanted to at least try the latest possible 6.x release, hoping to last a bit longer till EOL, but no success.
Again, the following is a summary done by Claude:




My first post pinned this on kernel 7.0. That was premature — a few clean apt runs prove nothing against an intermittent, partly time-dependent fault (errors sometimes fire seconds after activity, while idle). So I retested properly under sustained load.

Test method (same hardware, each kernel selected via proxmox-boot-tool kernel pin <ver> --next-boot): one terminal running a continuous loop of write+read work, a second watching dmesg.

# terminal 1 — load loop
i=0; while true; do i=$((i+1)); echo "[$(date +%H:%M:%S)] run $i"; \
apt reinstall -y htop tree mtr-tiny >/dev/null 2>&1; \
find /usr -type f -exec cat {} + >/dev/null 2>&1; sleep 5; done

# terminal 2 — watch for errors
watch -n 10 "dmesg -T | grep -iE 'ICRC|parity|I/O error|hard resetting' | tail -20"

Results:

Kernel	Under sustained read+write load
6.14.11-9-pve	stable — 450+ iterations, >1 h, zero errors
6.17.13-18-pve	fails within ~1–2 min
7.0.14-5-pve	fails, same pattern

So the regression starts at 6.17, not 7.0 — and 6.17.13-18 is a mature point release, not an early-6.17 fluke. New this round: the first-ever op 0x0:(READ) I/O error (every prior failure was WRITE-only), which appeared under 6.17 after only a few apt reinstalls:

Kernel: 6.17.13-18-pve

ata2: hard resetting link
I/O error, dev sda, sector 18984880 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
ata2.00: error: { ICRC ABRT }
ata2.00: error: { ICRC ABRT }
[... dozens of repeats ...]
ata2: hard resetting link
sd 1:0:0:0: [sda] tag#0 Add. Sense: Scsi parity error
I/O error, dev sda, sector 78467072 op 0x1:(WRITE) flags 0x4000 phys_seg 64 prio class 2
sd 1:0:0:0: [sda] tag#1 Add. Sense: Scsi parity error
I/O error, dev sda, sector 78475264 op 0x1:(WRITE) flags 0x0 phys_seg 64 prio class 2
EXT4-fs warning (device dm-3): ext4_end_bio:368: I/O error 10 writing to inode 1836418 starting block 7448576)
Buffer I/O error on device dm-3, logical block 7448576
Buffer I/O error on device dm-3, logical block 7448577
[... more buffer I/O errors ...]
ata2.00: error: { ICRC ABRT }
ata2: hard resetting link
[... pattern repeats until load stops ...]


Two things I'd still like input on:

• Anyone on Intel 8/9-series (Lynx Point PCH) seeing ICRC ABRT + link resets appear at 6.17 but not 6.14? Known ahci/libata change in that window?

Staying on pinned 6.14.11 for now (stopgap, not a fix — 6.14 is nearing EOL and I can't move to 6.17 because that's where it breaks).

 

[BigWeb]

I can confirm a very similar and reproducible problem on another H97/Haswell system, but I found a narrower workaround that allows kernel 7.0 to run cleanly.

Hardware:
- ASRock H97 Anniversary, BIOS P1.70
- Intel Core i7-4770
- 32 GB DDR3-1600, 4 x 8 GB
- Intel 9 Series SATA AHCI controller at 00:1f.2
- Additional ASM1062 SATA controller
- Several ZFS pools, including a large HDD pool

The system was upgraded from PVE 8 to Debian 13.6 / PVE 9.2.6 with kernel 7.0.14-8-pve.

Without a workaround, kernel 7.0 reproduced the failure twice. The logs first showed Intel VT-d/DMAR translation and PTE access faults involving the Intel AHCI controller. These were followed by SATA/EXT4 I/O errors and, during one boot, a ZFS module load failure:

Failed to insert module 'zfs': Key was rejected by service

The second reproduction was performed with every VM and container stopped, so guest workload was not the cause. The same unchanged hardware was clean when booted with kernel 6.8.12-39-pve.

Booting kernel 7.0 with:

intel_iommu=sp_off

completely resolved the problem on this machine.

Validation performed:
- two clean kernel 7.0 boots with intel_iommu=sp_off
- no DMAR, PTE, ICRC, FPDMA, ATA exception or I/O errors
- 20 repeated SHA256 reads of the ZFS kernel module were identical
- ZFS module signer was valid and Secure Boot was disabled
- controlled 128 MiB non-compressible write + fsync and direct-read test
- all ZFS pools and Proxmox storages online
- all five production VMs/containers started and tested
- the existing UDMA CRC counter remained unchanged, with no new errors

Unlike intel_iommu=off, the sp_off option does not disable Intel IOMMU completely. It only disables IOMMU superpage support.

This strongly suggests an Intel IOMMU superpage mapping regression affecting the SATA/AHCI DMA path, rather than a bad SATA cable, SSD, RAM, or a purely libata-related problem.

I have kept kernel 6.8.12-39-pve installed as a recovery option. I did not test kernel 6.14 on this machine, so I cannot provide an additional 6.14 confirmation, but kernel 7.0.14-8-pve with intel_iommu=sp_off is currently stable under the full production workload.

 

[drpepper285]

I have a Haswell System with Z97 Chipset and had not the same but similar issues with SATA controllers and especially with IOMMU functionality. I a few months ago I was able to figure out that my issue was the IOMMU superpages implementation on that specific hardware (same as @BigWeb s recommendation). Since April my system is running rock stable without any issues and after several updates. See here for more details, maybe that helps:


https://forum.proxmox.com/threads/f...proxmox-9-with-8-14-kernel.170165/post-846290