Summary
After upgrading from
The oops itself is not the interesting part.
The previous kernel,
System
This is not a memory-starved box: it currently sits at 2.8 GiB used, 29 GiB free, 15 GiB in buff/cache, with swap completely untouched. (Those figures are from after the reboot, not from the moment of the incident - but the workload is unchanged.)
Timeline
Note the
Root cause
Decoding the faulting instruction from
So the prev pointer of a list_head inside the inode LRU contained garbage (
Immediately after:
Consequence: everything else piles up on the orphaned lock
26 seconds later, the first soft lockups appear. Every single one of them is spinning on the same lock address,
and, on the other NUMA node,
The stuck counter goes 26s -> 52s and never resets, which is what you would expect from an orphaned lock rather than from contention. CPU#14, #1, #3 and #16 followed with identical stacks; I have omitted them since they add nothing.
One detail that may matter: on this box node0 owns the even CPUs and node1 the odd ones. The CPUs that piled up were 18, 19, 14, 1, 3 and 16 - that is both NUMA nodes, all waiting on the single address
What I ruled out, and how
Good/bad boundary
This is what makes me think it is worth looking at:
I checked
Possible reason nobody else is seeing it
This host is dual-socket Westmere-EP (2x X5675) on an Intel 5520 / Tylersburg platform, with 2 NUMA nodes.
It is also worth noting that the host has 47 GiB of RAM and is nowhere near memory exhaustion. Reclaim here is per-node and per-memcg, so a single node can be under pressure while the global figures look comfortable. Whatever raced, it did not need a low-memory condition to do it.
The kernel also reports a firmware workaround taint flag on this board:
Reproducing
Not reproduced on demand yet. It happened once, ~3h20m into the first boot on
After upgrading from
7.0.14-8-pve to 7.0.14-9-pve, a Proxmox Backup Server host died after 3h20m of uptime with a general protection fault in list_lru_isolate_move(), hit by kswapd0 while pruning the inode cache.The oops itself is not the interesting part.
kswapd0 died while holding locks (exited with preempt_count 2), so the list_lru spinlock was never released. Every other CPU that subsequently entered reclaim piled up on that exact same lock address and soft-locked, one after another, until the machine stopped responding.The previous kernel,
7.0.14-8-pve, ran 6 days 11 hours on the same host with the same workload and was shut down cleanly. This was the first boot on -9.System
- Proxmox Backup Server
4.2.5-1(running version 4.2.5) - Kernel:
7.0.14-9-pve(bad) /7.0.14-8-pve(good).proxmox-kernel-6.17.13-21also installed. - Hardware: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0069.032620181413 03/26/2018
- CPU: 2x Intel Xeon X5675 @ 3.07GHz (Westmere-EP), 12 cores / 24 threads
- 2 NUMA nodes: node0 = CPUs 0,2,4,...,22 - node1 = CPUs 1,3,5,...,23
- RAM: 47 GiB, swap 47 GiB
- Root/datastore filesystem: ext4 (the faulting writer path is
ext4_da_write_begin) - ZFS module is loaded but no ZFS pool is in use on this host
- Boot: plain GRUB,
proxmox-boot-toolis not initialised zfsutils-linux 2.4.3-pve1,smartmontools 7.5-pve2
This is not a memory-starved box: it currently sits at 2.8 GiB used, 29 GiB free, 15 GiB in buff/cache, with swap completely untouched. (Those figures are from after the reboot, not from the moment of the incident - but the workload is unchanged.)
Timeline
Code:
05:40 clean shutdown on 7.0.14-8-pve (after 6d11h uptime)
05:42 first boot on 7.0.14-9-pve
09:00:17 Oops [#1] - GPF in list_lru_isolate_move, kswapd0 dies
09:00:17 note: kswapd0[194] exited with preempt_count 2
09:00:43 soft lockup CPU#18 (tokio-runtime-w) and CPU#19 (kswapd1), 26s
09:01:11 same two CPUs, now 52s - the lock is never released
09:01:15 CPU#14
09:01:27 CPU#1
09:02:18 CPU#3
09:03:09 CPU#16, then silence
~09:15 host rebooted on its ownNote the
Tainted flags on the oops itself: P IO only. No D, no W, no L. This is genuinely the first event of the boot - nothing preceded it.Root cause
Code:
Oops: general protection fault, probably for non-canonical address 0xffff5000c2004f08: 0000 [#1] SMP PTI
CPU: 6 UID: 0 PID: 194 Comm: kswapd0 Tainted: P IO 7.0.14-9-pve #1 PREEMPT(lazy)
Tainted: [P]=PROPRIETARY_MODULE, [I]=FIRMWARE_WORKAROUND, [O]=OOT_MODULE
Hardware name: Intel Corporation S5520UR/S5520UR, BIOS S5500.86B.01.00.0069.032620181413 03/26/2018
RIP: 0010:list_lru_isolate_move+0x27/0x80
Code: 90 90 90 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 89 f7 48 89 d6 48 83 ec 10 48 8b 47 08 48 8b 17 48 3b 38 0f 85 0f 2a ae ff <48> 3b 7a 08 0f 85 05 2a ae ff 48 89 42 08 48 89 10 48 8b 16 48 3b
RSP: 0018:ffffd326c6ccf928 EFLAGS: 00010246
RAX: ffff8ea638ddfa30 RBX: ffff8ea638ddfa30 RCX: 0000000000000000
RDX: ffff5000c2004f00 RSI: ffffd326c6ccfa20 RDI: ffff8ea5c0000210
RBP: ffffd326c6ccf940 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8ea638ddfa30
R13: ffffd326c6ccfa20 R14: ffff8ea5c0000190 R15: ffff8ea5c0000110
FS: 0000000000000000(0000) GS:ffff8ea66628e000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000074236e1c7000 CR3: 00000001fda3e002 CR4: 00000000000226f0
Call Trace:
<TASK>
inode_lru_isolate+0xba/0x1f0
__list_lru_walk_one+0x10b/0x270
? __pfx_inode_lru_isolate+0x10/0x10
list_lru_walk_one+0x10/0x30
prune_icache_sb+0x4b/0x90
super_cache_scan+0x147/0x1f0
do_shrink_slab+0x157/0x3e0
shrink_slab+0x26e/0x380
shrink_one+0xe8/0x1c0
shrink_node+0x9f7/0xc10
? mem_cgroup_iter+0x1d8/0x230
balance_pgdat+0x5cf/0xb20
? __schedule+0x4a2/0x1780
? timer_delete_sync+0x27/0x50
kswapd+0x1c7/0x370
? __pfx_autoremove_wake_function+0x10/0x10
? __pfx_kswapd+0x10/0x10
kthread+0xf7/0x130
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2da/0x3a0
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
---[ end trace 0000000000000000 ]---Decoding the faulting instruction from
Code:, the byte at RIP is 48 3b 7a 08, i.e. cmp rdi, [rdx+8]. RDX was loaded two instructions earlier by 48 8b 17 (mov rdx, [rdi]) from RDI = ffff8ea5c0000210.So the prev pointer of a list_head inside the inode LRU contained garbage (
ffff5000c2004f00, non-canonical). The list debug checks caught it, but only after the walk had already taken the per-node list_lru lock.Immediately after:
Code:
note: kswapd0[194] exited with preempt_count 2
------------[ cut here ]------------
WARNING: kernel/exit.c:904 at do_exit+0x87d/0xb00, CPU#6: kswapd0/194
RIP: 0010:do_exit+0x87d/0xb00
Call Trace:
<TASK>
make_task_dead+0x93/0xa0
rewind_stack_and_make_dead+0x16/0x20
</TASK>preempt_count 2 is the key line: the task was torn down holding two non-preemptible sections. Nothing ever unlocks them.Consequence: everything else piles up on the orphaned lock
26 seconds later, the first soft lockups appear. Every single one of them is spinning on the same lock address,
ffff8ea638ddfa48 (RBX/RDI), which is adjacent to ffff8ea638ddfa30 seen in the oops registers:
Code:
watchdog: BUG: soft lockup - CPU#18 stuck for 26s! [tokio-runtime-w:227601]
CPU: 18 UID: 34 PID: 227601 Comm: tokio-runtime-w Tainted: P D W IO 7.0.14-9-pve #1
RIP: 0010:native_queued_spin_lock_slowpath+0x252/0x2f0
RBX: ffff8ea638ddfa48 ... RDI: ffff8ea638ddfa48 ... R15: ffff8ea638ddfa30
Call Trace:
<TASK>
_raw_spin_lock+0x3f/0x60
list_lru_add+0x93/0x1b0
list_lru_add_obj+0x7f/0xe0
iput+0x1cf/0x2f0
dentry_unlink_inode+0xdc/0x160
__dentry_kill+0x76/0x1a0
shrink_dentry_list+0x6b/0xf0
prune_dcache_sb+0x56/0x90
super_cache_scan+0x11b/0x1f0
do_shrink_slab+0x157/0x3e0
shrink_slab+0x26e/0x380
shrink_one+0xe8/0x1c0
shrink_node+0x9f7/0xc10
do_try_to_free_pages+0xc8/0x570
try_to_free_pages+0xd1/0x220
__alloc_frozen_pages_noprof+0xc40/0x1700
alloc_pages_mpol+0x80/0x180
folio_alloc_noprof+0x5b/0xc0
filemap_alloc_folio_noprof+0x61/0x80
__filemap_get_folio_mpol+0x197/0x340
ext4_da_write_begin+0x142/0x320
generic_perform_write+0x137/0x2d0
ext4_buffered_write_iter+0x6d/0x150
ext4_file_write_iter+0xb1/0x8d0
vfs_write+0x274/0x490
ksys_write+0x6f/0xf0
__x64_sys_write+0x19/0x30
x64_sys_call+0x22f/0x2390
do_syscall_64+0x10b/0x14e0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>and, on the other NUMA node,
kswapd1 blocked on the very same address:
Code:
watchdog: BUG: soft lockup - CPU#19 stuck for 26s! [kswapd1:195]
RIP: 0010:native_queued_spin_lock_slowpath+0x85/0x2f0
RBX: ffff8ea638ddfa48 ... RDI: ffff8ea638ddfa48 ... R15: ffff8ea638ddfa30
Call Trace:
_raw_spin_lock+0x3f/0x60
list_lru_add+0x93/0x1b0
list_lru_add_obj+0x7f/0xe0
iput+0x1cf/0x2f0
dentry_unlink_inode+0xdc/0x160
__dentry_kill+0x76/0x1a0
prune_dcache_sb+0x56/0x90
super_cache_scan+0x11b/0x1f0
shrink_slab+0x26e/0x380
shrink_node+0x9f7/0xc10
balance_pgdat+0x5cf/0xb20
kswapd+0x1c7/0x370The stuck counter goes 26s -> 52s and never resets, which is what you would expect from an orphaned lock rather than from contention. CPU#14, #1, #3 and #16 followed with identical stacks; I have omitted them since they add nothing.
One detail that may matter: on this box node0 owns the even CPUs and node1 the odd ones. The CPUs that piled up were 18, 19, 14, 1, 3 and 16 - that is both NUMA nodes, all waiting on the single address
ffff8ea638ddfa48. So the lock left behind by kswapd0 on node0 was also blocking reclaim originating on node1, which is consistent with an orphaned list_lru node lock rather than with per-node contention.What I ruled out, and how
- Bad RAM. The corrupted pointer is
ffff5000c2004f00where a valid one in the same structure isffff8ea5c0000210- that is not a single-bit difference, it is a completely different value. Additionallyi7core_edacwas loaded for the whole boot across both memory controllers on an ECC system and reported no correctable or uncorrectable errors. - Hardware / power. The host was alive and logging throughout; this was a lockup, not a reset.
- ZFS / ARC pressure. The module is loaded but no pool is in use on this host, and the writer in the trace is on ext4.
- pstore. Empty, which is consistent - a soft lockup is not a panic, so there was nothing to persist.
Good/bad boundary
This is what makes me think it is worth looking at:
Code:
7.0.14-8 2026-07-28 single backport, CVE-2026-64531
7.0.14-9 2026-08-05 rebase onto Ubuntu-7.0.0-31.31, plus 9 CVEs-8 and -9 are not neighbouring patch levels, they are two different bases. -8 ran 6d11h on this host; -9 died after 3h20m on its first boot.I checked
7.0.14-10 and 7.0.14-11 - both are network/x86 CVE backports and neither touches mm/, so I do not expect either of them to change this.Possible reason nobody else is seeing it
This host is dual-socket Westmere-EP (2x X5675) on an Intel 5520 / Tylersburg platform, with 2 NUMA nodes.
list_lru keeps its lists per NUMA node, and reclaim runs per-node and per-memcg. If this is a race in the per-node LRU handling, it would only be reachable on a multi-node machine, and the timing on an early Intel NUMA platform - QPI latencies, no modern cache-coherency optimisations - is very different from anything currently used for testing.It is also worth noting that the host has 47 GiB of RAM and is nowhere near memory exhaustion. Reclaim here is per-node and per-memcg, so a single node can be under pressure while the global figures look comfortable. Whatever raced, it did not need a low-memory condition to do it.
The kernel also reports a firmware workaround taint flag on this board:
Code:
Tainted: [P]=PROPRIETARY_MODULE, [I]=FIRMWARE_WORKAROUND, [O]=OOT_MODULEReproducing
Not reproduced on demand yet. It happened once, ~3h20m into the first boot on
-9, while the PBS proxy (tokio-runtime-w) was writing to ext4 and the host was under memory reclaim. I have left the host on -9 deliberately to see whether it recurs, and will update this thread either way.