Hi
I tried this way ....
Subject: [BUG] megaraid_sas: PRP list overflow on NVMe drives with kernel >= 7.0
(BLK_DEF_MAX_SECTORS_CAP raised to 4M)
To: linux-scsi@vger.kernel.org
Cc: megaraidlinux.pdl@broadcom.com
Cc: Kashyap Desai <kashyap.desai@broadcom.com>
Cc: Sumit Saxena <sumit.saxena@broadcom.com>
Cc: Chandrakanth patil <chandrakanth.patil@broadcom.com>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-block@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>
Hello,
We are seeing a reproducible kernel oops in megasas_make_prp_nvme() on
NVMe drives attached behind a MegaRAID SAS39xx controller. The crash
appears after upgrading from 6.8 to 7.0 and is triggered by any large
sequential write.
The root cause looks like an interaction between two independent
changes, neither of which is wrong on its own:
1. BLK_DEF_MAX_SECTORS_CAP was raised from 2560 sectors (1280 KiB)
to SZ_4M >> SECTOR_SHIFT (8192 sectors, 4096 KiB), and moved from
include/linux/blkdev.h to block/blk.h.
2. megasas_make_prp_nvme() builds the NVMe PRP list into a single
page (cmd->sg_frame, instance->max_chain_frame_sz) with no bound
check against the number of entries it emits.
With max_hw_sectors reported as 4096 KiB by the controller firmware,
max_sectors is now also 4096 KiB, which requires 1024 PRP entries.
The buffer holds 512.
HARDWARE
========
Controller : Broadcom MegaRAID 12GSAS/PCIe Secure SAS39xx [1000:10e2]
(HPE MR416i, HPE ProLiant DL380 Gen11, BIOS 2.84)
Drives : 5x HPE MO003200KXPAA, 3.2 TB NVMe, FW HPS7, 512 B LBA
presented as [0:2:1:0] .. [0:2:5:0], vendor string "NVMe"
Layout : per drive - 1007K BIOS boot, 1 GiB vfat ESP, rest ZFS
(Proxmox VE, rpool mirror)
KERNEL VERSIONS
===============
OK : 6.8.12-34-pve megaraid_sas 07.727.03.00-rc1
BAD : 7.0.14-4-pve megaraid_sas 07.734.00.00-rc1
BAD : 7.0.14-11-pve megaraid_sas 07.734.00.00-rc1
(Proxmox VE kernels, Ubuntu-derived. The relevant code paths are
unmodified upstream.)
QUEUE LIMITS - THE ACTUAL DIFFERENCE
====================================
Identical hardware, same node, only the kernel differs:
attribute (sysfs) 6.8.12-34 7.0.14-11
--------------------- ----------- -----------
max_hw_sectors_kb 4096 4096
max_sectors_kb 1280 4096 <-- 3.2x increase
optimal_io_size 0 0
minimum_io_size 512 512
virt_boundary_mask 4095 4095
nomerges 2 0
optimal_io_size and minimum_io_size are both 0/512, so the io_opt and
io_min branches added to blk_validate_limits() are not involved. The
increase comes purely from the raised default cap:
/* v6.8, include/linux/blkdev.h */
#define BLK_DEF_MAX_SECTORS_CAP 2560u
/* v7.1, block/blk.h */
#define BLK_DEF_MAX_SECTORS_CAP (SZ_4M >> SECTOR_SHIFT)
lim->max_sectors = min(max_hw_sectors, BLK_DEF_MAX_SECTORS_CAP);
= min(4096 KiB, 4096 KiB) = 4096 KiB
Separately, QUEUE_FLAG_NOMERGES is no longer set by the driver
(nomerges 2 -> 0). This is consistent with the atomic queue limits
conversion but removes a second constraint on request size.
OOPS
====
Full trace from 7.0.14-4-pve. Note the driver's own KERN_NOTICE
immediately preceding the fault:
sd 0:2:1:0: [sda] tag#281 page boundary ptr_sgl: 0x00000000f5092361
BUG: unable to handle page fault for address: ff7ff5ac80b0e000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 100000067 P4D 100d30067 PUD 100d31067 PMD 103a39067 PTE 0
Oops: Oops: 0002 [#1] SMP NOPTI
CPU: 38 UID: 0 PID: 12 Comm: kworker/u256:0 Tainted: P S O 7.0.14-4-pve #1
Hardware name: HPE ProLiant DL380 Gen11/ProLiant DL380 Gen11, BIOS 2.84
Workqueue: writeback wb_workfn (flush-8:0)
RIP: 0010:megasas_build_and_issue_cmd_fusion+0xe88/0x1850 [megaraid_sas]
RSP: 0018:ff7ff5ac800c72f8 EFLAGS: 00010206
RAX: 00000000fe9e7000 RBX: ff307b1c028d3948 RCX: ff7ff5ac80b0e000
RDX: ff7ff5ac80b0e008 RSI: ff307b1c028d3810 RDI: 0000000000000000
RBP: ff7ff5ac800c73c8 R08: 0000000000000200 R09: 0000000000001000
R10: 0000000000000fff R11: 0000000000001000 R12: 00000000001ff000
R13: 0000000000200000 R14: 00000000e1200000 R15: ff7ff5ac80b0e008
Call Trace:
scsi_queue_rq+0x82e/0xcf0
blk_mq_dispatch_rq_list+0x136/0x7c0
__blk_mq_sched_dispatch_requests+0x415/0x620
blk_mq_sched_dispatch_requests+0x2d/0x80
blk_mq_run_hw_queue+0x2c3/0x330
blk_mq_dispatch_list+0x16a/0x4d0
blk_mq_flush_plug_list+0x62/0x1d0
blk_add_rq_to_plug+0xfc/0x1e0
blk_mq_submit_bio+0x695/0x8c0
__submit_bio+0xaf/0x250
submit_bio_noacct_nocheck+0x29d/0x370
submit_bio_noacct+0x1b5/0x5e0
submit_bio+0xb1/0x110
mpage_write_folio+0x538/0x7c0
__mpage_writepages+0xbe/0x130
fat_writepages+0x17/0x30
do_writepages+0xc4/0x180
__writeback_single_inode+0x45/0x340
writeback_sb_inodes+0x26b/0x5f0
wb_writeback+0x8c/0x360
wb_workfn+0xc9/0x450
process_one_work+0x1a9/0x3c0
The request reaches the driver through the normal blk-mq dispatch
path, so max_sectors is what allowed its size.
ANALYSIS
========
Disassembly of the faulting site maps 1:1 onto the PRP fill loop in
megasas_make_prp_nvme() (drivers/scsi/megaraid/megaraid_sas_fusion.c):
4c 89 f9 mov %r15,%rcx ptr_sgl advance
4c 8d 79 08 lea 0x8(%rcx),%r15 ptr_sgl + 1
45 85 fa test %r15d,%r10d & page_mask
0f 84 fd 03 00 00 je <chain pointer branch>
45 29 cc sub %r9d,%r12d sge_len -= mr_nvme_pg_size
4c 89 31 <-- mov %r14,(%rcx) *ptr_sgl = sge_addr FAULT
48 83 c0 08 add $0x8,%rax ptr_sgl_phys += 8
41 83 c0 01 add $0x1,%r8d num_prp_in_chain++
45 29 cd sub %r9d,%r13d data_len -= mr_nvme_pg_size
45 85 e4 test %r12d,%r12d
7f ab jg <loop>
Register state at the fault:
R09 = 0x1000 mr_nvme_pg_size (4 KiB)
R10 = 0xfff page_mask
R13 = 0x200000 data_len remaining (2 MiB)
R12 = 0x1ff000 sge_len remaining
R14 = 0xe1200000 sge_addr
R08 = 0x200 512 entries written
RCX = ...b0e000 ptr_sgl, page aligned, faulting store
512 entries * 8 bytes = 4096 bytes, i.e. exactly one page. Counting
the chain pointer inserted at the page boundary, the buffer holds 511
data entries, capping any transfer at roughly 2044 KiB.
The KERN_NOTICE fires from the page-boundary branch:
page_mask_result = (uintptr_t)(ptr_sgl + 1) & page_mask;
if (unlikely(!page_mask_result)) {
scmd_printk(KERN_NOTICE, scmd,
"page boundary ptr_sgl: 0x%p\n", ptr_sgl);
ptr_sgl_phys += 8;
*ptr_sgl = cpu_to_le64(ptr_sgl_phys);
ptr_sgl++; /* now past end of allocation */
num_prp_in_chain++;
}
*ptr_sgl = cpu_to_le64(sge_addr); /* faults */
The chaining logic advances ptr_sgl into what it assumes is the next
page of the SGL buffer, but cmd->sg_frame is a single page. The loop
terminates only on data_len, never on buffer capacity.
Recomputing the request sizes from the register state:
crash A: 512 iterations + 0x200000 remaining -> ~4096 KiB
crash B: 512 iterations + 0x2000 remaining -> ~2056 KiB
4096 KiB is exactly max_sectors_kb on the affected kernel.
On 6.8 the same code path is safe only by coincidence: max_sectors was
capped at 1280 KiB, i.e. 320 PRP entries, leaving a 1.6x margin the
driver never enforced.
REPRODUCTION
============
Not immediate; requires sustained large sequential I/O. Two nodes,
identical hardware:
node A, boot 1: 19 hours idle, no crash; failed shortly after
normal morning workload started
node A, boot 2: crashed after 11 minutes under load
node A, boot 3: crashed after 14 minutes under load
node B: same signature, same day
Every oops carries R08 = 0x200. Booting the same kernel on an idle
node with no VM workload has not crashed so far.
WORKAROUND
==========
Constraining max_sectors below the PRP capacity avoids the crash.
Writing to sysfs sets max_user_sectors, which takes priority over the
default cap in blk_validate_limits():
echo 1280 > /sys/block/sdX/queue/max_sectors_kb
Persistent, via udev:
ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*", \
SUBSYSTEMS=="pci", DRIVERS=="megaraid_sas", \
ATTR{queue/max_sectors_kb}="1280"
SUGGESTED FIX
=============
The driver should not rely on the block layer sending less than it can
handle. megasas_set_nvme_device_properties() could clamp max_hw_sectors
to the PRP list capacity, roughly:
prp_capacity_kb = ((instance->max_chain_frame_sz / 8) - 1)
* (mr_nvme_pg_size / 1024);
lim->max_hw_sectors = min(max_io_size / 512, prp_capacity_kb * 2);
A defensive bound check in the megasas_make_prp_nvme() loop would also
be worthwhile, since silently walking off the allocation is a poor
failure mode for a storage driver.
I am happy to test patches on this hardware and can provide full
journals from the affected boots.
Best Regards,
I hope some fix will be implemented for all - because reason exist and I can reproduce it on more than one Server. - (I had help from Claude .. )
I tried this way ....
Subject: [BUG] megaraid_sas: PRP list overflow on NVMe drives with kernel >= 7.0
(BLK_DEF_MAX_SECTORS_CAP raised to 4M)
To: linux-scsi@vger.kernel.org
Cc: megaraidlinux.pdl@broadcom.com
Cc: Kashyap Desai <kashyap.desai@broadcom.com>
Cc: Sumit Saxena <sumit.saxena@broadcom.com>
Cc: Chandrakanth patil <chandrakanth.patil@broadcom.com>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-block@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>
Hello,
We are seeing a reproducible kernel oops in megasas_make_prp_nvme() on
NVMe drives attached behind a MegaRAID SAS39xx controller. The crash
appears after upgrading from 6.8 to 7.0 and is triggered by any large
sequential write.
The root cause looks like an interaction between two independent
changes, neither of which is wrong on its own:
1. BLK_DEF_MAX_SECTORS_CAP was raised from 2560 sectors (1280 KiB)
to SZ_4M >> SECTOR_SHIFT (8192 sectors, 4096 KiB), and moved from
include/linux/blkdev.h to block/blk.h.
2. megasas_make_prp_nvme() builds the NVMe PRP list into a single
page (cmd->sg_frame, instance->max_chain_frame_sz) with no bound
check against the number of entries it emits.
With max_hw_sectors reported as 4096 KiB by the controller firmware,
max_sectors is now also 4096 KiB, which requires 1024 PRP entries.
The buffer holds 512.
HARDWARE
========
Controller : Broadcom MegaRAID 12GSAS/PCIe Secure SAS39xx [1000:10e2]
(HPE MR416i, HPE ProLiant DL380 Gen11, BIOS 2.84)
Drives : 5x HPE MO003200KXPAA, 3.2 TB NVMe, FW HPS7, 512 B LBA
presented as [0:2:1:0] .. [0:2:5:0], vendor string "NVMe"
Layout : per drive - 1007K BIOS boot, 1 GiB vfat ESP, rest ZFS
(Proxmox VE, rpool mirror)
KERNEL VERSIONS
===============
OK : 6.8.12-34-pve megaraid_sas 07.727.03.00-rc1
BAD : 7.0.14-4-pve megaraid_sas 07.734.00.00-rc1
BAD : 7.0.14-11-pve megaraid_sas 07.734.00.00-rc1
(Proxmox VE kernels, Ubuntu-derived. The relevant code paths are
unmodified upstream.)
QUEUE LIMITS - THE ACTUAL DIFFERENCE
====================================
Identical hardware, same node, only the kernel differs:
attribute (sysfs) 6.8.12-34 7.0.14-11
--------------------- ----------- -----------
max_hw_sectors_kb 4096 4096
max_sectors_kb 1280 4096 <-- 3.2x increase
optimal_io_size 0 0
minimum_io_size 512 512
virt_boundary_mask 4095 4095
nomerges 2 0
optimal_io_size and minimum_io_size are both 0/512, so the io_opt and
io_min branches added to blk_validate_limits() are not involved. The
increase comes purely from the raised default cap:
/* v6.8, include/linux/blkdev.h */
#define BLK_DEF_MAX_SECTORS_CAP 2560u
/* v7.1, block/blk.h */
#define BLK_DEF_MAX_SECTORS_CAP (SZ_4M >> SECTOR_SHIFT)
lim->max_sectors = min(max_hw_sectors, BLK_DEF_MAX_SECTORS_CAP);
= min(4096 KiB, 4096 KiB) = 4096 KiB
Separately, QUEUE_FLAG_NOMERGES is no longer set by the driver
(nomerges 2 -> 0). This is consistent with the atomic queue limits
conversion but removes a second constraint on request size.
OOPS
====
Full trace from 7.0.14-4-pve. Note the driver's own KERN_NOTICE
immediately preceding the fault:
sd 0:2:1:0: [sda] tag#281 page boundary ptr_sgl: 0x00000000f5092361
BUG: unable to handle page fault for address: ff7ff5ac80b0e000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 100000067 P4D 100d30067 PUD 100d31067 PMD 103a39067 PTE 0
Oops: Oops: 0002 [#1] SMP NOPTI
CPU: 38 UID: 0 PID: 12 Comm: kworker/u256:0 Tainted: P S O 7.0.14-4-pve #1
Hardware name: HPE ProLiant DL380 Gen11/ProLiant DL380 Gen11, BIOS 2.84
Workqueue: writeback wb_workfn (flush-8:0)
RIP: 0010:megasas_build_and_issue_cmd_fusion+0xe88/0x1850 [megaraid_sas]
RSP: 0018:ff7ff5ac800c72f8 EFLAGS: 00010206
RAX: 00000000fe9e7000 RBX: ff307b1c028d3948 RCX: ff7ff5ac80b0e000
RDX: ff7ff5ac80b0e008 RSI: ff307b1c028d3810 RDI: 0000000000000000
RBP: ff7ff5ac800c73c8 R08: 0000000000000200 R09: 0000000000001000
R10: 0000000000000fff R11: 0000000000001000 R12: 00000000001ff000
R13: 0000000000200000 R14: 00000000e1200000 R15: ff7ff5ac80b0e008
Call Trace:
scsi_queue_rq+0x82e/0xcf0
blk_mq_dispatch_rq_list+0x136/0x7c0
__blk_mq_sched_dispatch_requests+0x415/0x620
blk_mq_sched_dispatch_requests+0x2d/0x80
blk_mq_run_hw_queue+0x2c3/0x330
blk_mq_dispatch_list+0x16a/0x4d0
blk_mq_flush_plug_list+0x62/0x1d0
blk_add_rq_to_plug+0xfc/0x1e0
blk_mq_submit_bio+0x695/0x8c0
__submit_bio+0xaf/0x250
submit_bio_noacct_nocheck+0x29d/0x370
submit_bio_noacct+0x1b5/0x5e0
submit_bio+0xb1/0x110
mpage_write_folio+0x538/0x7c0
__mpage_writepages+0xbe/0x130
fat_writepages+0x17/0x30
do_writepages+0xc4/0x180
__writeback_single_inode+0x45/0x340
writeback_sb_inodes+0x26b/0x5f0
wb_writeback+0x8c/0x360
wb_workfn+0xc9/0x450
process_one_work+0x1a9/0x3c0
The request reaches the driver through the normal blk-mq dispatch
path, so max_sectors is what allowed its size.
ANALYSIS
========
Disassembly of the faulting site maps 1:1 onto the PRP fill loop in
megasas_make_prp_nvme() (drivers/scsi/megaraid/megaraid_sas_fusion.c):
4c 89 f9 mov %r15,%rcx ptr_sgl advance
4c 8d 79 08 lea 0x8(%rcx),%r15 ptr_sgl + 1
45 85 fa test %r15d,%r10d & page_mask
0f 84 fd 03 00 00 je <chain pointer branch>
45 29 cc sub %r9d,%r12d sge_len -= mr_nvme_pg_size
4c 89 31 <-- mov %r14,(%rcx) *ptr_sgl = sge_addr FAULT
48 83 c0 08 add $0x8,%rax ptr_sgl_phys += 8
41 83 c0 01 add $0x1,%r8d num_prp_in_chain++
45 29 cd sub %r9d,%r13d data_len -= mr_nvme_pg_size
45 85 e4 test %r12d,%r12d
7f ab jg <loop>
Register state at the fault:
R09 = 0x1000 mr_nvme_pg_size (4 KiB)
R10 = 0xfff page_mask
R13 = 0x200000 data_len remaining (2 MiB)
R12 = 0x1ff000 sge_len remaining
R14 = 0xe1200000 sge_addr
R08 = 0x200 512 entries written
RCX = ...b0e000 ptr_sgl, page aligned, faulting store
512 entries * 8 bytes = 4096 bytes, i.e. exactly one page. Counting
the chain pointer inserted at the page boundary, the buffer holds 511
data entries, capping any transfer at roughly 2044 KiB.
The KERN_NOTICE fires from the page-boundary branch:
page_mask_result = (uintptr_t)(ptr_sgl + 1) & page_mask;
if (unlikely(!page_mask_result)) {
scmd_printk(KERN_NOTICE, scmd,
"page boundary ptr_sgl: 0x%p\n", ptr_sgl);
ptr_sgl_phys += 8;
*ptr_sgl = cpu_to_le64(ptr_sgl_phys);
ptr_sgl++; /* now past end of allocation */
num_prp_in_chain++;
}
*ptr_sgl = cpu_to_le64(sge_addr); /* faults */
The chaining logic advances ptr_sgl into what it assumes is the next
page of the SGL buffer, but cmd->sg_frame is a single page. The loop
terminates only on data_len, never on buffer capacity.
Recomputing the request sizes from the register state:
crash A: 512 iterations + 0x200000 remaining -> ~4096 KiB
crash B: 512 iterations + 0x2000 remaining -> ~2056 KiB
4096 KiB is exactly max_sectors_kb on the affected kernel.
On 6.8 the same code path is safe only by coincidence: max_sectors was
capped at 1280 KiB, i.e. 320 PRP entries, leaving a 1.6x margin the
driver never enforced.
REPRODUCTION
============
Not immediate; requires sustained large sequential I/O. Two nodes,
identical hardware:
node A, boot 1: 19 hours idle, no crash; failed shortly after
normal morning workload started
node A, boot 2: crashed after 11 minutes under load
node A, boot 3: crashed after 14 minutes under load
node B: same signature, same day
Every oops carries R08 = 0x200. Booting the same kernel on an idle
node with no VM workload has not crashed so far.
WORKAROUND
==========
Constraining max_sectors below the PRP capacity avoids the crash.
Writing to sysfs sets max_user_sectors, which takes priority over the
default cap in blk_validate_limits():
echo 1280 > /sys/block/sdX/queue/max_sectors_kb
Persistent, via udev:
ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*", \
SUBSYSTEMS=="pci", DRIVERS=="megaraid_sas", \
ATTR{queue/max_sectors_kb}="1280"
SUGGESTED FIX
=============
The driver should not rely on the block layer sending less than it can
handle. megasas_set_nvme_device_properties() could clamp max_hw_sectors
to the PRP list capacity, roughly:
prp_capacity_kb = ((instance->max_chain_frame_sz / 8) - 1)
* (mr_nvme_pg_size / 1024);
lim->max_hw_sectors = min(max_io_size / 512, prp_capacity_kb * 2);
A defensive bound check in the megasas_make_prp_nvme() loop would also
be worthwhile, since silently walking off the allocation is a poor
failure mode for a storage driver.
I am happy to test patches on this hardware and can provide full
journals from the affected boots.
Best Regards,
I hope some fix will be implemented for all - because reason exist and I can reproduce it on more than one Server. - (I had help from Claude .. )
