HPE MR416i-o & HPE MR408i-o Gen11/10

[Patrik_V]

Hi
I tried this way ....

Subject: [BUG] megaraid_sas: PRP list overflow on NVMe drives with kernel >= 7.0
(BLK_DEF_MAX_SECTORS_CAP raised to 4M)

To: linux-scsi@vger.kernel.org
Cc: megaraidlinux.pdl@broadcom.com
Cc: Kashyap Desai <kashyap.desai@broadcom.com>
Cc: Sumit Saxena <sumit.saxena@broadcom.com>
Cc: Chandrakanth patil <chandrakanth.patil@broadcom.com>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-block@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>

Hello,

We are seeing a reproducible kernel oops in megasas_make_prp_nvme() on
NVMe drives attached behind a MegaRAID SAS39xx controller. The crash
appears after upgrading from 6.8 to 7.0 and is triggered by any large
sequential write.

The root cause looks like an interaction between two independent
changes, neither of which is wrong on its own:

1. BLK_DEF_MAX_SECTORS_CAP was raised from 2560 sectors (1280 KiB)
to SZ_4M >> SECTOR_SHIFT (8192 sectors, 4096 KiB), and moved from
include/linux/blkdev.h to block/blk.h.

2. megasas_make_prp_nvme() builds the NVMe PRP list into a single
page (cmd->sg_frame, instance->max_chain_frame_sz) with no bound
check against the number of entries it emits.

With max_hw_sectors reported as 4096 KiB by the controller firmware,
max_sectors is now also 4096 KiB, which requires 1024 PRP entries.
The buffer holds 512.


HARDWARE
========

Controller : Broadcom MegaRAID 12GSAS/PCIe Secure SAS39xx [1000:10e2]
(HPE MR416i, HPE ProLiant DL380 Gen11, BIOS 2.84)
Drives : 5x HPE MO003200KXPAA, 3.2 TB NVMe, FW HPS7, 512 B LBA
presented as [0:2:1:0] .. [0:2:5:0], vendor string "NVMe"
Layout : per drive - 1007K BIOS boot, 1 GiB vfat ESP, rest ZFS
(Proxmox VE, rpool mirror)


KERNEL VERSIONS
===============

OK : 6.8.12-34-pve megaraid_sas 07.727.03.00-rc1
BAD : 7.0.14-4-pve megaraid_sas 07.734.00.00-rc1
BAD : 7.0.14-11-pve megaraid_sas 07.734.00.00-rc1

(Proxmox VE kernels, Ubuntu-derived. The relevant code paths are
unmodified upstream.)


QUEUE LIMITS - THE ACTUAL DIFFERENCE
====================================

Identical hardware, same node, only the kernel differs:

attribute (sysfs) 6.8.12-34 7.0.14-11
--------------------- ----------- -----------
max_hw_sectors_kb 4096 4096
max_sectors_kb 1280 4096 <-- 3.2x increase
optimal_io_size 0 0
minimum_io_size 512 512
virt_boundary_mask 4095 4095
nomerges 2 0

optimal_io_size and minimum_io_size are both 0/512, so the io_opt and
io_min branches added to blk_validate_limits() are not involved. The
increase comes purely from the raised default cap:

/* v6.8, include/linux/blkdev.h */
#define BLK_DEF_MAX_SECTORS_CAP 2560u

/* v7.1, block/blk.h */
#define BLK_DEF_MAX_SECTORS_CAP (SZ_4M >> SECTOR_SHIFT)

lim->max_sectors = min(max_hw_sectors, BLK_DEF_MAX_SECTORS_CAP);
= min(4096 KiB, 4096 KiB) = 4096 KiB

Separately, QUEUE_FLAG_NOMERGES is no longer set by the driver
(nomerges 2 -> 0). This is consistent with the atomic queue limits
conversion but removes a second constraint on request size.


OOPS
====

Full trace from 7.0.14-4-pve. Note the driver's own KERN_NOTICE
immediately preceding the fault:

sd 0:2:1:0: [sda] tag#281 page boundary ptr_sgl: 0x00000000f5092361
BUG: unable to handle page fault for address: ff7ff5ac80b0e000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 100000067 P4D 100d30067 PUD 100d31067 PMD 103a39067 PTE 0
Oops: Oops: 0002 [#1] SMP NOPTI
CPU: 38 UID: 0 PID: 12 Comm: kworker/u256:0 Tainted: P S O 7.0.14-4-pve #1
Hardware name: HPE ProLiant DL380 Gen11/ProLiant DL380 Gen11, BIOS 2.84
Workqueue: writeback wb_workfn (flush-8:0)
RIP: 0010:megasas_build_and_issue_cmd_fusion+0xe88/0x1850 [megaraid_sas]
RSP: 0018:ff7ff5ac800c72f8 EFLAGS: 00010206
RAX: 00000000fe9e7000 RBX: ff307b1c028d3948 RCX: ff7ff5ac80b0e000
RDX: ff7ff5ac80b0e008 RSI: ff307b1c028d3810 RDI: 0000000000000000
RBP: ff7ff5ac800c73c8 R08: 0000000000000200 R09: 0000000000001000
R10: 0000000000000fff R11: 0000000000001000 R12: 00000000001ff000
R13: 0000000000200000 R14: 00000000e1200000 R15: ff7ff5ac80b0e008
Call Trace:
scsi_queue_rq+0x82e/0xcf0
blk_mq_dispatch_rq_list+0x136/0x7c0
__blk_mq_sched_dispatch_requests+0x415/0x620
blk_mq_sched_dispatch_requests+0x2d/0x80
blk_mq_run_hw_queue+0x2c3/0x330
blk_mq_dispatch_list+0x16a/0x4d0
blk_mq_flush_plug_list+0x62/0x1d0
blk_add_rq_to_plug+0xfc/0x1e0
blk_mq_submit_bio+0x695/0x8c0
__submit_bio+0xaf/0x250
submit_bio_noacct_nocheck+0x29d/0x370
submit_bio_noacct+0x1b5/0x5e0
submit_bio+0xb1/0x110
mpage_write_folio+0x538/0x7c0
__mpage_writepages+0xbe/0x130
fat_writepages+0x17/0x30
do_writepages+0xc4/0x180
__writeback_single_inode+0x45/0x340
writeback_sb_inodes+0x26b/0x5f0
wb_writeback+0x8c/0x360
wb_workfn+0xc9/0x450
process_one_work+0x1a9/0x3c0

The request reaches the driver through the normal blk-mq dispatch
path, so max_sectors is what allowed its size.


ANALYSIS
========

Disassembly of the faulting site maps 1:1 onto the PRP fill loop in
megasas_make_prp_nvme() (drivers/scsi/megaraid/megaraid_sas_fusion.c):

4c 89 f9 mov %r15,%rcx ptr_sgl advance
4c 8d 79 08 lea 0x8(%rcx),%r15 ptr_sgl + 1
45 85 fa test %r15d,%r10d & page_mask
0f 84 fd 03 00 00 je <chain pointer branch>
45 29 cc sub %r9d,%r12d sge_len -= mr_nvme_pg_size
4c 89 31 <-- mov %r14,(%rcx) *ptr_sgl = sge_addr FAULT
48 83 c0 08 add $0x8,%rax ptr_sgl_phys += 8
41 83 c0 01 add $0x1,%r8d num_prp_in_chain++
45 29 cd sub %r9d,%r13d data_len -= mr_nvme_pg_size
45 85 e4 test %r12d,%r12d
7f ab jg <loop>

Register state at the fault:

R09 = 0x1000 mr_nvme_pg_size (4 KiB)
R10 = 0xfff page_mask
R13 = 0x200000 data_len remaining (2 MiB)
R12 = 0x1ff000 sge_len remaining
R14 = 0xe1200000 sge_addr
R08 = 0x200 512 entries written
RCX = ...b0e000 ptr_sgl, page aligned, faulting store

512 entries * 8 bytes = 4096 bytes, i.e. exactly one page. Counting
the chain pointer inserted at the page boundary, the buffer holds 511
data entries, capping any transfer at roughly 2044 KiB.

The KERN_NOTICE fires from the page-boundary branch:

page_mask_result = (uintptr_t)(ptr_sgl + 1) & page_mask;
if (unlikely(!page_mask_result)) {
scmd_printk(KERN_NOTICE, scmd,
"page boundary ptr_sgl: 0x%p\n", ptr_sgl);
ptr_sgl_phys += 8;
*ptr_sgl = cpu_to_le64(ptr_sgl_phys);
ptr_sgl++; /* now past end of allocation */
num_prp_in_chain++;
}
*ptr_sgl = cpu_to_le64(sge_addr); /* faults */

The chaining logic advances ptr_sgl into what it assumes is the next
page of the SGL buffer, but cmd->sg_frame is a single page. The loop
terminates only on data_len, never on buffer capacity.

Recomputing the request sizes from the register state:

crash A: 512 iterations + 0x200000 remaining -> ~4096 KiB
crash B: 512 iterations + 0x2000 remaining -> ~2056 KiB

4096 KiB is exactly max_sectors_kb on the affected kernel.

On 6.8 the same code path is safe only by coincidence: max_sectors was
capped at 1280 KiB, i.e. 320 PRP entries, leaving a 1.6x margin the
driver never enforced.


REPRODUCTION
============

Not immediate; requires sustained large sequential I/O. Two nodes,
identical hardware:

node A, boot 1: 19 hours idle, no crash; failed shortly after
normal morning workload started
node A, boot 2: crashed after 11 minutes under load
node A, boot 3: crashed after 14 minutes under load
node B: same signature, same day

Every oops carries R08 = 0x200. Booting the same kernel on an idle
node with no VM workload has not crashed so far.


WORKAROUND
==========

Constraining max_sectors below the PRP capacity avoids the crash.
Writing to sysfs sets max_user_sectors, which takes priority over the
default cap in blk_validate_limits():

echo 1280 > /sys/block/sdX/queue/max_sectors_kb

Persistent, via udev:

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*", \
SUBSYSTEMS=="pci", DRIVERS=="megaraid_sas", \
ATTR{queue/max_sectors_kb}="1280"


SUGGESTED FIX
=============

The driver should not rely on the block layer sending less than it can
handle. megasas_set_nvme_device_properties() could clamp max_hw_sectors
to the PRP list capacity, roughly:

prp_capacity_kb = ((instance->max_chain_frame_sz / 8) - 1)
* (mr_nvme_pg_size / 1024);
lim->max_hw_sectors = min(max_io_size / 512, prp_capacity_kb * 2);

A defensive bound check in the megasas_make_prp_nvme() loop would also
be worthwhile, since silently walking off the allocation is a poor
failure mode for a storage driver.

I am happy to test patches on this hardware and can provide full
journals from the affected boots.

Best Regards,

I hope some fix will be implemented for all - because reason exist and I can reproduce it on more than one Server. - (I had help from Claude .. )

 

[scontin]

I add my notes, i flew to the same conclusions:

Problem Description:
Under recent Proxmox VE kernels (6.8+ and 7.0+), servers equipped with Broadcom MegaRAID / HPE Smart Array Tri-Mode controllers (e.g., MR408i, MR216i, LSI 9500/9600 series using megaraid_sas) running ZFS or high-throughput async I/O on NVMe pass-through drives experience severe I/O stalls, PCIe timeouts, host kernel page faults, and catastrophic boot partition corruption.

Root Cause Analysis: The Broadcom SoC & SRAM Bottleneck
The root cause lies not in the NVMe drives themselves, but in how the Broadcom Tri-Mode SoC manages hardware DMA descriptors in its internal Static RAM (SRAM) when translating SCSI commands to NVMe PRPs.

* Driver Over-Promising (max_hw_sectors_kb = 4096):
The megaraid_sas driver reads max_hw_sectors_kb = 4096 (4 MiB) from the Tri-Mode controller firmware when exposing NVMe namespaces to the block layer. The kernel takes this at face value.

* Sequential vs. Scattered DMA Pointers & Kernel Call Chain:
If a 4 MiB transfer is perfectly contiguous in host RAM, it requires very few DMA pointers. However, ZFS is a Copy-On-Write filesystem. When ZFS flushes the ARC via spa_sync() or performs heavy async I/O, a 4 MiB payload is severely fragmented across physical RAM pages.

To map and describe this fragmented transfer, the kernel's block and NVMe subsystems invoke specific mapping and list-building functions—primarily nvme_map_data(), which relies on dma_map_sg_attrs() to translate the scatterlist into hardware-addressable chunks, followed by megasas_build_dma_fusion() (or megasas_make_cmd_fusion() inside megaraid_sas) to invoke nvme_setup_prps() (or nvme_setup_sgls()).

For a heavily fragmented 4 MiB payload, the host OS must build a descriptor table with up to 1024 individual DMA pointers for a single I/O command.

* SoC SRAM Descriptor Overflow & FTL Collapse:
The legacy LSI/Broadcom silicon architecture running behind the Tri-Mode translation layer has limited on-chip SRAM allocated for active I/O queues. When the kernel fires multiple concurrent 4 MiB I/O requests (high iodepth) packed with thousands of scattered DMA pointers, the controller's internal SRAM descriptor buffer physically overflows.

Budget-oriented, OEM Tri-Mode, or MegaRAID translation controllers under-allocate internal hardware resources (on-chip SRAM). They struggle with cache/buffer descriptor management when processing deep queues of these high-count PRP/SGL entries concurrently with massive ZFS write bursts.

When the SRAM buffer overflows, the controller's internal Flash Translation Layer (FTL) and queue descriptors collapse. The SoC freezes, drops response interrupts, or misdirects DMA writes, manifesting on the host side as:

[ 1432.891024] DMAR: [DMA Write] Request device [04:00.0] PASID 0xffffffff fault addr 0x7f3bc40000 [fault reason 0x05] PTE Write access is not set
[ 1432.891100] megaraid_sas 0000:04:00.0: megasas_disable_intr_fusion: Failed to disable interrupts / Command Timeout
[ 1432.891150] sd 4:0:0:0: [sda] failing command: Out of memory / DMA mapping failed
[ 1432.891210] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 1432.891250] RIP: 0010:megasas_complete_cmd_fusion+0x4a/0x1d0 [megaraid_sas]
[ 1432.891300] Call Trace:
[ 1432.891320] <IRQ>
[ 1432.891340] megasas_isr_fusion+0x1a4/0x380 [megaraid_sas]

* The ESP (EFI System Partition) Destruction:
Crucially, if background ZFS flushes exhaust the controller's DMA buffers while `proxmox-boot-tool` is syncing kernel updates to the unjournaled VFAT EFI partition (ESP), the controller drops write frames or misdirects write pointers. This silently destroys the VFAT filesystem, leaving the server completely unbootable (`grub rescue` or missing EFI boot targets) upon the next reboot.

Note: Native SATA/SAS drives managed by the same controller automatically default to max_sectors_kb = 1024. This restricts the maximum pointer count per command, safely insulating the SoC from SRAM exhaustion.

* Why Module Parameters and Initramfs Fail (And Why Boot Succeeds)

Attempts to pass max sector parameters via /etc/modprobe.d/megaraid_sas.conf fail because the driver does not expose a global sector limit parameter. Furthermore, udev rules do not bind to sysfs queue attributes during early initramfs boot before the root filesystem switches.

* Why doesn't the server crash during boot?
Because early-boot I/O (loading the kernel, basic initramfs) is light, linear, and highly sequential. A 4 MiB sequential read requires very few DMA pointers, keeping the SoC's SRAM well within limits. The controller's pointer buffers only overflow later, once the OS has fully booted and ZFS begins issuing massive, fragmented multi-megabyte I/O bursts.

The Working Workaround / Fix

Override max_sectors_kb via a persistent user-space udev rule. As soon as user-space init completes (systemd-udevd), the rule kicks in and clamps the queue limit to 1280 KB before production ZFS workloads and VM I/O flushes hit the array.

Setting max_sectors_kb = 1280 strictly limits the kernel to generating a maximum of ~320 DMA pointers per I/O command. This matches historical kernel defaults and keeps PRP descriptor allocation safely within the Broadcom SoC's physical SRAM limits, eliminating all I/O lockups, ESP corruption, and system panics.

Implementation Steps

1. Create a custom udev rule:

cat << 'EOF' > /etc/udev/rules.d/99-megaraid-nvme-max-sectors.rules
# Limit max_sectors_kb on MegaRAID / Tri-Mode devices to prevent SoC descriptor SRAM overflow
ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_VENDOR}=="LSI|Broadcom|HPE", ATTR{queue/max_sectors_kb}="1280"
EOF

2. Reload udev rules and trigger on existing running block devices:

udevadm control --reload-rules
udevadm trigger --subsystem-match=block

3. Verify the setting applied across your array/disks:

for dev in /sys/block/sd*; do echo "$dev -> $(cat $dev/queue/max_sectors_kb) KB"; done