Recent PVE 9 kernel update - OOPSfest in mm/hugetlb

D

danmac

Renowned Member
Dec 6, 2016
28
7
68
Hi there :)

I have two PostgreSQL servers running in Debian LXCs, both now fail after applying updates from the last 10 days.

systemctl start postgresql hangs, and i'm left with dead ("X") postgres processes, the containers cannot be shut down or stopped, nothing in the postgres logs, systemd journal only remarks that it didn't start in time ...

was working fine until i updated this morning with 10 days worth of updates (can post apt history.log or w/e)

doesn't seem to be the kernel which went from 6.14 to 6.17 as the first thing i tried was rebooting with the old kernel

haven't had much time to fiddle with it but im guessing it's permissions, both are unpriv + nesting, and i know postgres can be a little finicky on that front

any advice welcome, cheers :)
 
turns out it does seem to be kernel related

will elaborate more later

for now, going back to 6.14.8-2-pve and disabling huge pages seems to have worked for me

here's a taster, it's an old AMD FX-8320 (bulldozer?) 8 "core" w/ 32GB ECC

cheers

Code: 
Nov 18 14:25:33 myhost kernel: ------------[ cut here ]------------
Nov 18 14:25:33 myhost kernel: kernel BUG at mm/hugetlb.c:5854!
Nov 18 14:25:33 myhost kernel: Oops: invalid opcode: 0000 [#2] SMP NOPTI
Nov 18 14:25:33 myhost kernel: CPU: 6 UID: 100000 PID: 5463 Comm: php-fpm7.0 Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:33 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:33 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:33 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:33 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:33 myhost kernel: RSP: 0000:ffffce53e62cbaa8 EFLAGS: 00010206
Nov 18 14:25:33 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4f7d275240 RCX: 0000000000000009
Nov 18 14:25:33 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4f7d275240 RDI: ffffce53e62cbc60
Nov 18 14:25:33 myhost kernel: RBP: ffffce53e62cbb20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:33 myhost kernel: R10: 00007919c4005000 R11: 0000000000000000 R12: 00007919c0005000
Nov 18 14:25:33 myhost kernel: R13: ffffce53e62cbc60 R14: ffff8a4f7d275240 R15: ffffce53e62cbc18
Nov 18 14:25:33 myhost kernel: FS:  00007919cce32680(0000) GS:ffff8a54d3886000(0000) knlGS:0000000000000000
Nov 18 14:25:33 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:33 myhost kernel: CR2: 00007919c0005000 CR3: 000000030c0a2000 CR4: 00000000000406f0
Nov 18 14:25:33 myhost kernel: Call Trace:
Nov 18 14:25:33 myhost kernel:  <TASK>
Nov 18 14:25:33 myhost kernel:  unmap_single_vma.isra.0+0xc0/0xd0
Nov 18 14:25:33 myhost kernel:  unmap_vmas+0xa5/0x180
Nov 18 14:25:33 myhost kernel:  exit_mmap+0xff/0x400
Nov 18 14:25:33 myhost kernel:  __mmput+0x41/0x150
Nov 18 14:25:33 myhost kernel:  mmput+0x31/0x40
Nov 18 14:25:33 myhost kernel:  do_exit+0x264/0xa20
Nov 18 14:25:33 myhost kernel:  ? kmem_cache_free+0x406/0x470
Nov 18 14:25:33 myhost kernel:  do_group_exit+0x34/0x90
Nov 18 14:25:33 myhost kernel:  get_signal+0x833/0x880
Nov 18 14:25:33 myhost kernel:  arch_do_signal_or_restart+0x41/0x260
Nov 18 14:25:33 myhost kernel:  irqentry_exit_to_user_mode+0x1f4/0x290
Nov 18 14:25:33 myhost kernel:  irqentry_exit+0x43/0x50
Nov 18 14:25:33 myhost kernel:  exc_page_fault+0x90/0x1b0
Nov 18 14:25:33 myhost kernel:  asm_exc_page_fault+0x27/0x30
Nov 18 14:25:33 myhost kernel: RIP: 0033:0x7919cae5938e
Nov 18 14:25:33 myhost kernel: Code: Unable to access opcode bytes at 0x7919cae59364.
Nov 18 14:25:33 myhost kernel: RSP: 002b:00007ffda4396ce8 EFLAGS: 00010246
Nov 18 14:25:33 myhost kernel: RAX: 00007919c0005000 RBX: 00007919c0005000 RCX: 0000000004000000
Nov 18 14:25:33 myhost kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 00007919c0005000
Nov 18 14:25:33 myhost kernel: RBP: 0000000000000040 R08: 0000000000000040 R09: 0000000000000040
Nov 18 14:25:33 myhost kernel: R10: 0000000000040021 R11: 00007ffda4396d20 R12: 00007919c73fe6b0
Nov 18 14:25:33 myhost kernel: R13: 00007919c73f6460 R14: 0000000004000000 R15: 0000000000000001
Nov 18 14:25:33 myhost kernel:  </TASK>
Nov 18 14:25:33 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables
Nov 18 14:25:33 myhost kernel: ---[ end trace 0000000000000000 ]---
Nov 18 14:25:34 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:34 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:34 myhost kernel: RSP: 0000:ffffce53cf807aa8 EFLAGS: 00010206
Nov 18 14:25:34 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4d85c10000 RCX: 0000000000000009
Nov 18 14:25:34 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4d85c10000 RDI: ffffce53cf807c60
Nov 18 14:25:34 myhost kernel: RBP: ffffce53cf807b20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:34 myhost kernel: R10: 00007791e8605000 R11: 0000000000000000 R12: 00007791df605000
Nov 18 14:25:34 myhost kernel: R13: ffffce53cf807c60 R14: ffff8a4d85c10000 R15: ffffce53cf807c18
Nov 18 14:25:34 myhost kernel: FS:  00007919cce32680(0000) GS:ffff8a54d3886000(0000) knlGS:0000000000000000
Nov 18 14:25:34 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:34 myhost kernel: CR2: 00007919c0005000 CR3: 000000070783a000 CR4: 00000000000406f0
Nov 18 14:25:34 myhost kernel: Fixing recursive fault but reboot is needed!
Nov 18 14:25:34 myhost kernel: BUG: scheduling while atomic: php-fpm7.0/5463/0x00000000
Nov 18 14:25:34 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables
Nov 18 14:25:34 myhost kernel: CPU: 6 UID: 100000 PID: 5463 Comm: php-fpm7.0 Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:34 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:34 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:34 myhost kernel: Call Trace:
Nov 18 14:25:34 myhost kernel:  <TASK>
Nov 18 14:25:34 myhost kernel:  dump_stack_lvl+0x5f/0x90
Nov 18 14:25:34 myhost kernel:  dump_stack+0x10/0x18
Nov 18 14:25:34 myhost kernel:  __schedule_bug.cold+0x46/0x62
Nov 18 14:25:34 myhost kernel:  __schedule+0x109f/0x1310
Nov 18 14:25:34 myhost kernel:  ? vprintk+0x18/0x50
Nov 18 14:25:34 myhost kernel:  do_task_dead+0x4a/0xa0
Nov 18 14:25:34 myhost kernel:  make_task_dead.cold+0xdc/0xe8
Nov 18 14:25:34 myhost kernel:  rewind_stack_and_make_dead+0x16/0x20
Nov 18 14:25:34 myhost kernel: RIP: 0033:0x7919cae5938e
Nov 18 14:25:34 myhost kernel: Code: Unable to access opcode bytes at 0x7919cae59364.
Nov 18 14:25:34 myhost kernel: RSP: 002b:00007ffda4396ce8 EFLAGS: 00010246
Nov 18 14:25:34 myhost kernel: RAX: 00007919c0005000 RBX: 00007919c0005000 RCX: 0000000004000000
Nov 18 14:25:34 myhost kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 00007919c0005000
Nov 18 14:25:34 myhost kernel: RBP: 0000000000000040 R08: 0000000000000040 R09: 0000000000000040
Nov 18 14:25:34 myhost kernel: R10: 0000000000040021 R11: 00007ffda4396d20 R12: 00007919c73fe6b0
Nov 18 14:25:34 myhost kernel: R13: 00007919c73f6460 R14: 0000000004000000 R15: 0000000000000001
Nov 18 14:25:34 myhost kernel:  </TASK>
Nov 18 14:25:34 myhost kernel: ------------[ cut here ]------------
Nov 18 14:25:34 myhost kernel: kernel BUG at mm/hugetlb.c:5854!
Nov 18 14:25:34 myhost kernel: Oops: invalid opcode: 0000 [#3] SMP NOPTI
Nov 18 14:25:35 myhost kernel: CPU: 3 UID: 100108 PID: 5658 Comm: postgres Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:35 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:35 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:35 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:35 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:35 myhost kernel: RSP: 0000:ffffce53e5c8baa8 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4f78bc8f00 RCX: 0000000000000009
Nov 18 14:25:35 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4f78bc8f00 RDI: ffffce53e5c8bc60
Nov 18 14:25:35 myhost kernel: RBP: ffffce53e5c8bb20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:35 myhost kernel: R10: 00007a241d405000 R11: 0000000000000000 R12: 00007a2414605000
Nov 18 14:25:35 myhost kernel: R13: ffffce53e5c8bc60 R14: ffff8a4f78bc8f00 R15: ffffce53e5c8bc18
Nov 18 14:25:35 myhost kernel: FS:  00007a242581c300(0000) GS:ffff8a54d3706000(0000) knlGS:0000000000000000
Nov 18 14:25:35 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:35 myhost kernel: CR2: 00007a2414605000 CR3: 00000002fd26e000 CR4: 00000000000406f0
Nov 18 14:25:35 myhost kernel: Call Trace:
Nov 18 14:25:35 myhost kernel:  <TASK>
Nov 18 14:25:35 myhost kernel:  unmap_single_vma.isra.0+0xc0/0xd0
Nov 18 14:25:35 myhost kernel:  unmap_vmas+0xa5/0x180
Nov 18 14:25:35 myhost kernel:  exit_mmap+0xff/0x400
Nov 18 14:25:35 myhost kernel:  __mmput+0x41/0x150
Nov 18 14:25:35 myhost kernel:  mmput+0x31/0x40
Nov 18 14:25:35 myhost kernel:  do_exit+0x264/0xa20
Nov 18 14:25:35 myhost kernel:  ? kmem_cache_free+0x406/0x470
Nov 18 14:25:35 myhost kernel:  do_group_exit+0x34/0x90
Nov 18 14:25:35 myhost kernel:  get_signal+0x833/0x880
Nov 18 14:25:35 myhost kernel:  arch_do_signal_or_restart+0x41/0x260
Nov 18 14:25:35 myhost kernel:  irqentry_exit_to_user_mode+0x1f4/0x290
Nov 18 14:25:35 myhost kernel:  irqentry_exit+0x43/0x50
Nov 18 14:25:35 myhost kernel:  exc_page_fault+0x90/0x1b0
Nov 18 14:25:35 myhost kernel:  asm_exc_page_fault+0x27/0x30
Nov 18 14:25:35 myhost kernel: RIP: 0033:0x5daa9cabe63c
Nov 18 14:25:35 myhost kernel: Code: Unable to access opcode bytes at 0x5daa9cabe612.
Nov 18 14:25:35 myhost kernel: RSP: 002b:00007ffe6211e060 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 00007a2425805000 RBX: 00005daa9d0a3088 RCX: 000001472885750e
Nov 18 14:25:35 myhost kernel: RDX: 00007a2414605000 RSI: 00007ffe6211e090 RDI: 00005daacf1a82c0
Nov 18 14:25:35 myhost kernel: RBP: 00007ffe6211e1e0 R08: 0000000000000000 R09: 0000000000000014
Nov 18 14:25:35 myhost kernel: R10: 00007ffe6211a057 R11: 0000000000000246 R12: 0000000000000000
Nov 18 14:25:35 myhost kernel: R13: 00007ffe6211e120 R14: 000000000052e2c1 R15: 00007a2425805000
Nov 18 14:25:35 myhost kernel:  </TASK>
Nov 18 14:25:35 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables ip6table_raw
Nov 18 14:25:35 myhost kernel: ---[ end trace 0000000000000000 ]---
Nov 18 14:25:35 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:35 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:35 myhost kernel: RSP: 0000:ffffce53cf807aa8 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4d85c10000 RCX: 0000000000000009
Nov 18 14:25:35 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4d85c10000 RDI: ffffce53cf807c60
Nov 18 14:25:35 myhost kernel: RBP: ffffce53cf807b20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:35 myhost kernel: R10: 00007791e8605000 R11: 0000000000000000 R12: 00007791df605000
Nov 18 14:25:35 myhost kernel: R13: ffffce53cf807c60 R14: ffff8a4d85c10000 R15: ffffce53cf807c18
Nov 18 14:25:35 myhost kernel: FS:  00007a242581c300(0000) GS:ffff8a54d3706000(0000) knlGS:0000000000000000
Nov 18 14:25:35 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:35 myhost kernel: CR2: 00007a2414605000 CR3: 000000070783a000 CR4: 00000000000406f0
Nov 18 14:25:35 myhost kernel: Fixing recursive fault but reboot is needed!
Nov 18 14:25:35 myhost kernel: BUG: scheduling while atomic: postgres/5658/0x00000000
Nov 18 14:25:35 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables ip6table_raw
Nov 18 14:25:35 myhost kernel: CPU: 3 UID: 100108 PID: 5658 Comm: postgres Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:35 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:35 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:35 myhost kernel: Call Trace:
Nov 18 14:25:35 myhost kernel:  <TASK>
Nov 18 14:25:35 myhost kernel:  dump_stack_lvl+0x5f/0x90
Nov 18 14:25:35 myhost kernel:  dump_stack+0x10/0x18
Nov 18 14:25:35 myhost kernel:  __schedule_bug.cold+0x46/0x62
Nov 18 14:25:35 myhost kernel:  __schedule+0x109f/0x1310
Nov 18 14:25:35 myhost kernel:  ? vprintk+0x18/0x50
Nov 18 14:25:35 myhost kernel:  do_task_dead+0x4a/0xa0
Nov 18 14:25:35 myhost kernel:  make_task_dead.cold+0xdc/0xe8
Nov 18 14:25:35 myhost kernel:  rewind_stack_and_make_dead+0x16/0x20
Nov 18 14:25:35 myhost kernel: RIP: 0033:0x5daa9cabe63c
Nov 18 14:25:35 myhost kernel: Code: Unable to access opcode bytes at 0x5daa9cabe612.
Nov 18 14:25:35 myhost kernel: RSP: 002b:00007ffe6211e060 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 00007a2425805000 RBX: 00005daa9d0a3088 RCX: 000001472885750e
Nov 18 14:25:35 myhost kernel: RDX: 00007a2414605000 RSI: 00007ffe6211e090 RDI: 00005daacf1a82c0
Nov 18 14:25:35 myhost kernel: RBP: 00007ffe6211e1e0 R08: 0000000000000000 R09: 0000000000000014
Nov 18 14:25:35 myhost kernel: R10: 00007ffe6211a057 R11: 0000000000000246 R12: 0000000000000000
Nov 18 14:25:35 myhost kernel: R13: 00007ffe6211e120 R14: 000000000052e2c1 R15: 00007a2425805000
Nov 18 14:25:35 myhost kernel:  </TASK>
 
any interest in this easily reproduceable kernel bug which has surfaced in the last month or so on reliable hardware i am happy to inconvenience for a worthy cause?

i already figured out how to fix it for me, it's sub-optimal as i lose huge pages but good enough for now, and i've had no takers so far so im thinking further investigation and documentation thereof would only serve to waste my time

if i get a bite i'm happy to put the work in, otherwise i'll let it sink and someone more invested can trip over it

cheers
 
Hi Roland, i appreciate the interest - i thought that's what i was doing here, reporting it, tbh?

thanks for the link, i thought i remember seeing that bug report, and the purple links for the attachments confirmed it :) i couldn't be sure if it was the same bug - it occurred to me the owner of that bug report may not have caught the *first* kernel oops, just the aftermath.

honestly i left it on 6.14 kernel and hugepages disabled, it's been fine. i guess i could have reported it via the bugzilla but that didn't seem to do Mr. Horvat any good. i took my eye off that ball tbh, been busy writing a script to trigger if the power goes out, it fiddles with the power management, shuts down the various VMs & CTs nicely in specific order, in specific tiers depending on the severity of the power outage, keeps one eye on the UPS to rewind and bring stuff back up if the power returns in the meantime (it started with a fairly basic premise but "hey it'd be nice if it did this too" and here we are lol)

back to the hugepage oops tho I knew the odds when i started - "oh you think there's a bug in the kernel, do ya kid?" ;) (esp since hugepages isn't the default config, and web searches provide slim pickings on the oops)

and in fairness the last time i fell over a kernel bug and reported it, it was on a netburst-era Xeon, and it turned out it was caused by an LSI 8xSATA PCI-X card getting a little too warm :D

anyway, thanks for posting, i will be sure to update here when i get around to testing a current kernel - i was happy to leave it a month or two as 6.17 seemed to be a bit of a rush job that ended up biting a few other people with (i assume) unrelated issues

cheers Roland have a great day o7
 
  • Like
Reactions: RolandK
>i thought that's what i was doing here, reporting it, tbh

yes, thanks, but for a bug to be reproduced (which helps a lot resolving it), a "receipe" (i.e. detailed description of setup and what to do) is needed.

so if you can provide such, i'm sure it's very welcomed.
 
  • Like
Reactions: danmac
i'll see what i can do fella, do you need a separate account on that bug tracker yeah?

i also noticed at the time while grepping the logs, it hit the same bug a couple weeks prior, i would have to spend some more time with the logs as i did upgrade from proxmox 8 to 9 around that time

i was hesitant to invest the time documenting it thoroughly cos like i said, i know the perception when some random claims a bug in Our Kernel and Saviour with cobbled together gear (but muh ECC!) from over a decade ago :D so i wasn't expecting much of a response either way, tell ya the truth lol

i'll keep ya posted

cheers :)
 
You must log in or register to reply here.
Top Bottom