Recent PVE 9 kernel update - OOPSfest in mm/hugetlb

D

danmac

Renowned Member
Dec 6, 2016
23
6
68
Hi there :)

I have two PostgreSQL servers running in Debian LXCs, both now fail after applying updates from the last 10 days.

systemctl start postgresql hangs, and i'm left with dead ("X") postgres processes, the containers cannot be shut down or stopped, nothing in the postgres logs, systemd journal only remarks that it didn't start in time ...

was working fine until i updated this morning with 10 days worth of updates (can post apt history.log or w/e)

doesn't seem to be the kernel which went from 6.14 to 6.17 as the first thing i tried was rebooting with the old kernel

haven't had much time to fiddle with it but im guessing it's permissions, both are unpriv + nesting, and i know postgres can be a little finicky on that front

any advice welcome, cheers :)
 
turns out it does seem to be kernel related

will elaborate more later

for now, going back to 6.14.8-2-pve and disabling huge pages seems to have worked for me

here's a taster, it's an old AMD FX-8320 (bulldozer?) 8 "core" w/ 32GB ECC

cheers

Code: 
Nov 18 14:25:33 myhost kernel: ------------[ cut here ]------------
Nov 18 14:25:33 myhost kernel: kernel BUG at mm/hugetlb.c:5854!
Nov 18 14:25:33 myhost kernel: Oops: invalid opcode: 0000 [#2] SMP NOPTI
Nov 18 14:25:33 myhost kernel: CPU: 6 UID: 100000 PID: 5463 Comm: php-fpm7.0 Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:33 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:33 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:33 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:33 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:33 myhost kernel: RSP: 0000:ffffce53e62cbaa8 EFLAGS: 00010206
Nov 18 14:25:33 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4f7d275240 RCX: 0000000000000009
Nov 18 14:25:33 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4f7d275240 RDI: ffffce53e62cbc60
Nov 18 14:25:33 myhost kernel: RBP: ffffce53e62cbb20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:33 myhost kernel: R10: 00007919c4005000 R11: 0000000000000000 R12: 00007919c0005000
Nov 18 14:25:33 myhost kernel: R13: ffffce53e62cbc60 R14: ffff8a4f7d275240 R15: ffffce53e62cbc18
Nov 18 14:25:33 myhost kernel: FS:  00007919cce32680(0000) GS:ffff8a54d3886000(0000) knlGS:0000000000000000
Nov 18 14:25:33 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:33 myhost kernel: CR2: 00007919c0005000 CR3: 000000030c0a2000 CR4: 00000000000406f0
Nov 18 14:25:33 myhost kernel: Call Trace:
Nov 18 14:25:33 myhost kernel:  <TASK>
Nov 18 14:25:33 myhost kernel:  unmap_single_vma.isra.0+0xc0/0xd0
Nov 18 14:25:33 myhost kernel:  unmap_vmas+0xa5/0x180
Nov 18 14:25:33 myhost kernel:  exit_mmap+0xff/0x400
Nov 18 14:25:33 myhost kernel:  __mmput+0x41/0x150
Nov 18 14:25:33 myhost kernel:  mmput+0x31/0x40
Nov 18 14:25:33 myhost kernel:  do_exit+0x264/0xa20
Nov 18 14:25:33 myhost kernel:  ? kmem_cache_free+0x406/0x470
Nov 18 14:25:33 myhost kernel:  do_group_exit+0x34/0x90
Nov 18 14:25:33 myhost kernel:  get_signal+0x833/0x880
Nov 18 14:25:33 myhost kernel:  arch_do_signal_or_restart+0x41/0x260
Nov 18 14:25:33 myhost kernel:  irqentry_exit_to_user_mode+0x1f4/0x290
Nov 18 14:25:33 myhost kernel:  irqentry_exit+0x43/0x50
Nov 18 14:25:33 myhost kernel:  exc_page_fault+0x90/0x1b0
Nov 18 14:25:33 myhost kernel:  asm_exc_page_fault+0x27/0x30
Nov 18 14:25:33 myhost kernel: RIP: 0033:0x7919cae5938e
Nov 18 14:25:33 myhost kernel: Code: Unable to access opcode bytes at 0x7919cae59364.
Nov 18 14:25:33 myhost kernel: RSP: 002b:00007ffda4396ce8 EFLAGS: 00010246
Nov 18 14:25:33 myhost kernel: RAX: 00007919c0005000 RBX: 00007919c0005000 RCX: 0000000004000000
Nov 18 14:25:33 myhost kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 00007919c0005000
Nov 18 14:25:33 myhost kernel: RBP: 0000000000000040 R08: 0000000000000040 R09: 0000000000000040
Nov 18 14:25:33 myhost kernel: R10: 0000000000040021 R11: 00007ffda4396d20 R12: 00007919c73fe6b0
Nov 18 14:25:33 myhost kernel: R13: 00007919c73f6460 R14: 0000000004000000 R15: 0000000000000001
Nov 18 14:25:33 myhost kernel:  </TASK>
Nov 18 14:25:33 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables
Nov 18 14:25:33 myhost kernel: ---[ end trace 0000000000000000 ]---
Nov 18 14:25:34 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:34 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:34 myhost kernel: RSP: 0000:ffffce53cf807aa8 EFLAGS: 00010206
Nov 18 14:25:34 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4d85c10000 RCX: 0000000000000009
Nov 18 14:25:34 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4d85c10000 RDI: ffffce53cf807c60
Nov 18 14:25:34 myhost kernel: RBP: ffffce53cf807b20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:34 myhost kernel: R10: 00007791e8605000 R11: 0000000000000000 R12: 00007791df605000
Nov 18 14:25:34 myhost kernel: R13: ffffce53cf807c60 R14: ffff8a4d85c10000 R15: ffffce53cf807c18
Nov 18 14:25:34 myhost kernel: FS:  00007919cce32680(0000) GS:ffff8a54d3886000(0000) knlGS:0000000000000000
Nov 18 14:25:34 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:34 myhost kernel: CR2: 00007919c0005000 CR3: 000000070783a000 CR4: 00000000000406f0
Nov 18 14:25:34 myhost kernel: Fixing recursive fault but reboot is needed!
Nov 18 14:25:34 myhost kernel: BUG: scheduling while atomic: php-fpm7.0/5463/0x00000000
Nov 18 14:25:34 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables
Nov 18 14:25:34 myhost kernel: CPU: 6 UID: 100000 PID: 5463 Comm: php-fpm7.0 Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:34 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:34 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:34 myhost kernel: Call Trace:
Nov 18 14:25:34 myhost kernel:  <TASK>
Nov 18 14:25:34 myhost kernel:  dump_stack_lvl+0x5f/0x90
Nov 18 14:25:34 myhost kernel:  dump_stack+0x10/0x18
Nov 18 14:25:34 myhost kernel:  __schedule_bug.cold+0x46/0x62
Nov 18 14:25:34 myhost kernel:  __schedule+0x109f/0x1310
Nov 18 14:25:34 myhost kernel:  ? vprintk+0x18/0x50
Nov 18 14:25:34 myhost kernel:  do_task_dead+0x4a/0xa0
Nov 18 14:25:34 myhost kernel:  make_task_dead.cold+0xdc/0xe8
Nov 18 14:25:34 myhost kernel:  rewind_stack_and_make_dead+0x16/0x20
Nov 18 14:25:34 myhost kernel: RIP: 0033:0x7919cae5938e
Nov 18 14:25:34 myhost kernel: Code: Unable to access opcode bytes at 0x7919cae59364.
Nov 18 14:25:34 myhost kernel: RSP: 002b:00007ffda4396ce8 EFLAGS: 00010246
Nov 18 14:25:34 myhost kernel: RAX: 00007919c0005000 RBX: 00007919c0005000 RCX: 0000000004000000
Nov 18 14:25:34 myhost kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 00007919c0005000
Nov 18 14:25:34 myhost kernel: RBP: 0000000000000040 R08: 0000000000000040 R09: 0000000000000040
Nov 18 14:25:34 myhost kernel: R10: 0000000000040021 R11: 00007ffda4396d20 R12: 00007919c73fe6b0
Nov 18 14:25:34 myhost kernel: R13: 00007919c73f6460 R14: 0000000004000000 R15: 0000000000000001
Nov 18 14:25:34 myhost kernel:  </TASK>
Nov 18 14:25:34 myhost kernel: ------------[ cut here ]------------
Nov 18 14:25:34 myhost kernel: kernel BUG at mm/hugetlb.c:5854!
Nov 18 14:25:34 myhost kernel: Oops: invalid opcode: 0000 [#3] SMP NOPTI
Nov 18 14:25:35 myhost kernel: CPU: 3 UID: 100108 PID: 5658 Comm: postgres Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:35 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:35 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:35 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:35 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:35 myhost kernel: RSP: 0000:ffffce53e5c8baa8 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4f78bc8f00 RCX: 0000000000000009
Nov 18 14:25:35 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4f78bc8f00 RDI: ffffce53e5c8bc60
Nov 18 14:25:35 myhost kernel: RBP: ffffce53e5c8bb20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:35 myhost kernel: R10: 00007a241d405000 R11: 0000000000000000 R12: 00007a2414605000
Nov 18 14:25:35 myhost kernel: R13: ffffce53e5c8bc60 R14: ffff8a4f78bc8f00 R15: ffffce53e5c8bc18
Nov 18 14:25:35 myhost kernel: FS:  00007a242581c300(0000) GS:ffff8a54d3706000(0000) knlGS:0000000000000000
Nov 18 14:25:35 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:35 myhost kernel: CR2: 00007a2414605000 CR3: 00000002fd26e000 CR4: 00000000000406f0
Nov 18 14:25:35 myhost kernel: Call Trace:
Nov 18 14:25:35 myhost kernel:  <TASK>
Nov 18 14:25:35 myhost kernel:  unmap_single_vma.isra.0+0xc0/0xd0
Nov 18 14:25:35 myhost kernel:  unmap_vmas+0xa5/0x180
Nov 18 14:25:35 myhost kernel:  exit_mmap+0xff/0x400
Nov 18 14:25:35 myhost kernel:  __mmput+0x41/0x150
Nov 18 14:25:35 myhost kernel:  mmput+0x31/0x40
Nov 18 14:25:35 myhost kernel:  do_exit+0x264/0xa20
Nov 18 14:25:35 myhost kernel:  ? kmem_cache_free+0x406/0x470
Nov 18 14:25:35 myhost kernel:  do_group_exit+0x34/0x90
Nov 18 14:25:35 myhost kernel:  get_signal+0x833/0x880
Nov 18 14:25:35 myhost kernel:  arch_do_signal_or_restart+0x41/0x260
Nov 18 14:25:35 myhost kernel:  irqentry_exit_to_user_mode+0x1f4/0x290
Nov 18 14:25:35 myhost kernel:  irqentry_exit+0x43/0x50
Nov 18 14:25:35 myhost kernel:  exc_page_fault+0x90/0x1b0
Nov 18 14:25:35 myhost kernel:  asm_exc_page_fault+0x27/0x30
Nov 18 14:25:35 myhost kernel: RIP: 0033:0x5daa9cabe63c
Nov 18 14:25:35 myhost kernel: Code: Unable to access opcode bytes at 0x5daa9cabe612.
Nov 18 14:25:35 myhost kernel: RSP: 002b:00007ffe6211e060 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 00007a2425805000 RBX: 00005daa9d0a3088 RCX: 000001472885750e
Nov 18 14:25:35 myhost kernel: RDX: 00007a2414605000 RSI: 00007ffe6211e090 RDI: 00005daacf1a82c0
Nov 18 14:25:35 myhost kernel: RBP: 00007ffe6211e1e0 R08: 0000000000000000 R09: 0000000000000014
Nov 18 14:25:35 myhost kernel: R10: 00007ffe6211a057 R11: 0000000000000246 R12: 0000000000000000
Nov 18 14:25:35 myhost kernel: R13: 00007ffe6211e120 R14: 000000000052e2c1 R15: 00007a2425805000
Nov 18 14:25:35 myhost kernel:  </TASK>
Nov 18 14:25:35 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables ip6table_raw
Nov 18 14:25:35 myhost kernel: ---[ end trace 0000000000000000 ]---
Nov 18 14:25:35 myhost kernel: RIP: 0010:__unmap_hugepage_range+0x7b7/0x800
Nov 18 14:25:35 myhost kernel: Code: 89 ff 48 89 c6 e8 99 93 ff ff 48 8b 7d d0 e8 10 12 d5 00 e9 b0 fb ff ff 0f 0b 49 8b 51 30 48 f7 d2 4c 85 e2 0f 84 c6 f8 ff ff <0f> 0b 0f 0b 65 48 8b 05 95 61 70 02 48 8b 10 f7 c2 00 00 00 20 74
Nov 18 14:25:35 myhost kernel: RSP: 0000:ffffce53cf807aa8 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 0000000000400000 RBX: ffff8a4d85c10000 RCX: 0000000000000009
Nov 18 14:25:35 myhost kernel: RDX: 00000000001fffff RSI: ffff8a4d85c10000 RDI: ffffce53cf807c60
Nov 18 14:25:35 myhost kernel: RBP: ffffce53cf807b20 R08: 0000000000000000 R09: ffffffffab5ff858
Nov 18 14:25:35 myhost kernel: R10: 00007791e8605000 R11: 0000000000000000 R12: 00007791df605000
Nov 18 14:25:35 myhost kernel: R13: ffffce53cf807c60 R14: ffff8a4d85c10000 R15: ffffce53cf807c18
Nov 18 14:25:35 myhost kernel: FS:  00007a242581c300(0000) GS:ffff8a54d3706000(0000) knlGS:0000000000000000
Nov 18 14:25:35 myhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 14:25:35 myhost kernel: CR2: 00007a2414605000 CR3: 000000070783a000 CR4: 00000000000406f0
Nov 18 14:25:35 myhost kernel: Fixing recursive fault but reboot is needed!
Nov 18 14:25:35 myhost kernel: BUG: scheduling while atomic: postgres/5658/0x00000000
Nov 18 14:25:35 myhost kernel: Modules linked in: cfg80211 veth ebt_arp ebtable_filter ebtables ip6table_raw
Nov 18 14:25:35 myhost kernel: CPU: 3 UID: 100108 PID: 5658 Comm: postgres Tainted: P      D W  O        6.17.2-1-pve #1 PREEMPT(voluntary)
Nov 18 14:25:35 myhost kernel: Tainted: [P]=PROPRIETARY_MODULE, [D]=DIE, [W]=WARN, [O]=OOT_MODULE
Nov 18 14:25:35 myhost kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2901 05/04/2016
Nov 18 14:25:35 myhost kernel: Call Trace:
Nov 18 14:25:35 myhost kernel:  <TASK>
Nov 18 14:25:35 myhost kernel:  dump_stack_lvl+0x5f/0x90
Nov 18 14:25:35 myhost kernel:  dump_stack+0x10/0x18
Nov 18 14:25:35 myhost kernel:  __schedule_bug.cold+0x46/0x62
Nov 18 14:25:35 myhost kernel:  __schedule+0x109f/0x1310
Nov 18 14:25:35 myhost kernel:  ? vprintk+0x18/0x50
Nov 18 14:25:35 myhost kernel:  do_task_dead+0x4a/0xa0
Nov 18 14:25:35 myhost kernel:  make_task_dead.cold+0xdc/0xe8
Nov 18 14:25:35 myhost kernel:  rewind_stack_and_make_dead+0x16/0x20
Nov 18 14:25:35 myhost kernel: RIP: 0033:0x5daa9cabe63c
Nov 18 14:25:35 myhost kernel: Code: Unable to access opcode bytes at 0x5daa9cabe612.
Nov 18 14:25:35 myhost kernel: RSP: 002b:00007ffe6211e060 EFLAGS: 00010206
Nov 18 14:25:35 myhost kernel: RAX: 00007a2425805000 RBX: 00005daa9d0a3088 RCX: 000001472885750e
Nov 18 14:25:35 myhost kernel: RDX: 00007a2414605000 RSI: 00007ffe6211e090 RDI: 00005daacf1a82c0
Nov 18 14:25:35 myhost kernel: RBP: 00007ffe6211e1e0 R08: 0000000000000000 R09: 0000000000000014
Nov 18 14:25:35 myhost kernel: R10: 00007ffe6211a057 R11: 0000000000000246 R12: 0000000000000000
Nov 18 14:25:35 myhost kernel: R13: 00007ffe6211e120 R14: 000000000052e2c1 R15: 00007a2425805000
Nov 18 14:25:35 myhost kernel:  </TASK>
 
any interest in this easily reproduceable kernel bug which has surfaced in the last month or so on reliable hardware i am happy to inconvenience for a worthy cause?

i already figured out how to fix it for me, it's sub-optimal as i lose huge pages but good enough for now, and i've had no takers so far so im thinking further investigation and documentation thereof would only serve to waste my time

if i get a bite i'm happy to put the work in, otherwise i'll let it sink and someone more invested can trip over it

cheers
 
You must log in or register to reply here.
Top Bottom