I have updated to kernel 7.0.14-26 a few days ago:
pve-manager/9.2.24/66bc407c6d90a67a (running kernel: 7.0.14-26-pve)
Ever since, zfs receive of encrypted datasets (with syncoid) occasionally make the entire zfs pool hang.
Process info and stack trace shows that even pvestatd is hung on waiting for zfs in D state. See the logs below.
This prevents soft reboots, as the reboot itself is being blocked on zfs as well as renders the web UI largely useless, likely due to pvestatd being stuck.
Is this a known issue? I see in the changelog that there were very specific changes in the new kernel affecting encrypted zfs receive, something might have gone off the rails there:
https://github.com/proxmox/pve-kernel/blob/master/debian/changelog
Should I roll back the kernel version?
Hardware info:
https://pcpartpicker.com/b/f7sLrH
pve-manager/9.2.24/66bc407c6d90a67a (running kernel: 7.0.14-26-pve)
Ever since, zfs receive of encrypted datasets (with syncoid) occasionally make the entire zfs pool hang.
Process info and stack trace shows that even pvestatd is hung on waiting for zfs in D state. See the logs below.
This prevents soft reboots, as the reboot itself is being blocked on zfs as well as renders the web UI largely useless, likely due to pvestatd being stuck.
Is this a known issue? I see in the changelog that there were very specific changes in the new kernel affecting encrypted zfs receive, something might have gone off the rails there:
https://github.com/proxmox/pve-kernel/blob/master/debian/changelog
Should I roll back the kernel version?
Hardware info:
https://pcpartpicker.com/b/f7sLrH
Code:
Oct 30 19:47:30 prxcho01 kernel: </TASK>
Oct 30 19:49:33 prxcho01 kernel: INFO: task pvestatd:2905 blocked for more than 368 seconds.
Oct 30 19:49:33 prxcho01 kernel: Tainted: P O 7.0.14-26-pve #1
Oct 30 19:49:33 prxcho01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 30 19:49:33 prxcho01 kernel: taskvestatd state stack:0 pid:2905 tgid:2905 ppid:1 task_flags:0x400140 flags:0x00080000
Oct 30 19:49:33 prxcho01 kernel: Call Trace:
Oct 30 19:49:33 prxcho01 kernel: <TASK>
Oct 30 19:49:33 prxcho01 kernel: __schedule+0x448/0x1670
Oct 30 19:49:33 prxcho01 kernel: schedule+0x27/0xb0
Oct 30 19:49:33 prxcho01 kernel: cv_wait_common+0x10a/0x140 [spl]
Oct 30 19:49:33 prxcho01 kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Oct 30 19:49:33 prxcho01 kernel: __cv_wait+0x15/0x30 [spl]
Oct 30 19:49:33 prxcho01 kernel: rrw_enter_read_impl+0x53/0x1a0 [zfs]
Oct 30 19:49:33 prxcho01 kernel: rrm_enter_read+0x3f/0x50 [zfs]
Oct 30 19:49:33 prxcho01 kernel: zpl_show_devname+0x34/0x140 [zfs]
Oct 30 19:49:33 prxcho01 kernel: show_vfsmnt+0x54/0x180
Oct 30 19:49:33 prxcho01 kernel: m_show+0x1a/0x30
Oct 30 19:49:33 prxcho01 kernel: seq_read_iter+0x2c8/0x490
Oct 30 19:49:33 prxcho01 kernel: vfs_read+0x274/0x390
Oct 30 19:49:33 prxcho01 kernel: ksys_read+0x6f/0xf0
Oct 30 19:49:33 prxcho01 kernel: __x64_sys_read+0x19/0x30
Oct 30 19:49:33 prxcho01 kernel: x64_sys_call+0x1ff1/0x2390
Oct 30 19:49:33 prxcho01 kernel: do_syscall_64+0x10b/0x14c0
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? count_memcg_events+0xeb/0x190
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? handle_mm_fault+0x283/0x3a0
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? do_user_addr_fault+0x2f8/0x820
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? irqentry_exit+0xb2/0x710
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? exc_page_fault+0x92/0x1c0
Oct 30 19:49:33 prxcho01 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Oct 30 19:49:33 prxcho01 kernel: RIP: 0033:0x7741317383a7
Oct 30 19:49:33 prxcho01 kernel: RSP: 002b:00007ffca2a521c0 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
Oct 30 19:49:33 prxcho01 kernel: RAX: ffffffffffffffda RBX: 000077413166c240 RCX: 00007741317383a7
Oct 30 19:49:33 prxcho01 kernel: RDX: 0000000000002000 RSI: 00005d5bb3e51000 RDI: 0000000000000009
Oct 30 19:49:33 prxcho01 kernel: RBP: 00005d5bb3e51000 R08: 0000000000000000 R09: 0000000000000000
Oct 30 19:49:33 prxcho01 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
Oct 30 19:49:33 prxcho01 kernel: R13: 00005d5bb03dc2a0 R14: 0000000000000000 R15: 0000000000000000
Oct 30 19:49:33 prxcho01 kernel: </TASK>
Oct 30 19:49:33 prxcho01 kernel: </TASK>
Oct 30 19:49:33 prxcho01 kernel: INFO: task zfs:20661 blocked for more than 368 seconds.
Oct 30 19:49:33 prxcho01 kernel: Tainted: P O 7.0.14-26-pve #1
Oct 30 19:49:33 prxcho01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 30 19:49:33 prxcho01 kernel: task:zfs state stack:0 pid:20661 tgid:20661 ppid:20656 task_flags:0x480100 flags:0x00080800
Oct 30 19:49:33 prxcho01 kernel: Call Trace:
Oct 30 19:49:33 prxcho01 kernel: <TASK>
Oct 30 19:49:33 prxcho01 kernel: __schedule+0x448/0x1670
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? mod_memcg_lruvec_state+0xd7/0x1f0
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? count_memcg_events+0xeb/0x190
Oct 30 19:49:33 prxcho01 kernel: schedule+0x27/0xb0
Oct 30 19:49:33 prxcho01 kernel: io_schedule+0x4c/0x80
Oct 30 19:49:33 prxcho01 kernel: folio_wait_bit_common+0x136/0x330
Oct 30 19:49:33 prxcho01 kernel: ? __pfx_wake_page_function+0x10/0x10
Oct 30 19:49:33 prxcho01 kernel: __folio_lock+0x17/0x30
Oct 30 19:49:33 prxcho01 kernel: truncate_inode_pages_range+0x484/0x4e0
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: truncate_inode_pages+0x15/0x30
Oct 30 19:49:33 prxcho01 kernel: zfs_rezget+0x152/0x810 [zfs]
Oct 30 19:49:33 prxcho01 kernel: zfs_resume_fs+0x2bd/0x4a0 [zfs]
Oct 30 19:49:33 prxcho01 kernel: zfs_ioc_recv_impl.constprop.0+0x6e4/0x13e0 [zfs]
Oct 30 19:49:33 prxcho01 kernel: ? arc_access+0x2e8/0x4f0 [zfs]
Oct 30 19:49:33 prxcho01 kernel: zfs_ioc_recv_new+0x356/0x3c0 [zfs]
Oct 30 19:49:33 prxcho01 kernel: zfsdev_ioctl_common+0x525/0x970 [zfs]
Oct 30 19:49:33 prxcho01 kernel: zfsdev_ioctl+0x57/0xf0 [zfs]
Oct 30 19:49:33 prxcho01 kernel: __x64_sys_ioctl+0xa5/0x100
Oct 30 19:49:33 prxcho01 kernel: ? do_fcntl+0x155/0x7d0
Oct 30 19:49:33 prxcho01 kernel: x64_sys_call+0x103b/0x2390
Oct 30 19:49:33 prxcho01 kernel: do_syscall_64+0x10b/0x14c0
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? __x64_sys_fcntl+0x97/0x130
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? x64_sys_call+0x17bd/0x2390
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? do_syscall_64+0x148/0x14c0
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? x64_sys_call+0x17bd/0x2390
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Oct 30 19:49:33 prxcho01 kernel: ? do_syscall_64+0x148/0x14c0
Oct 30 19:49:33 prxcho01 kernel: ? srso_alias_return_thunk+0x5/0xfbef5