[SOLVED] since kernel 5.13.19-1-pve: 100% cpu of kworker/u8:1+events_unbound and slow write to disk performance

[abma]

since the upgrade to kernel 5.13, from time to time the process kworker uses 1 core with 100% even when no pbs task is active:

$ top
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  54147 root      20   0       0      0      0 R 100.0   0.0  73:38.66 kworker/u8:1+events_unbound

anyone else experieng this problem?

not sure what triggers it, after a fresh boot everything is ok. The problem seems to start, when the zfs datastore is used by a task (verify/backup). After that 100% Cpu is still used even when no task is running. See the screenshot (cpu usage / iowait), the problem started after the server was rebooted and using the new kernel:

 

[fiona]

Hi,
could you also check the package version for the kernel when it started happening? For example, in the uname -a output

Linux pbs1 5.13.19-1-pve #1 SMP PVE 5.13.19-3 (Tue, 23 Nov 2021 13:31:19 +0100) x86_64 GNU/Linux

the package version is 5.13.19-3 (the ABI version is 5.13.19-1). What was the previously installed package version (should be visible in /var/log/apt/history.log or in one of older /var/log/apt/history.log.<N>.gz log files)?

If the problematic kernel package version is 5.13.19-3, you could try downgrading with apt install pve-kernel-5.13.19-1-pve=5.13.19-2 to see if the problem was introduced between the two.

 

[abma]

apt/history.log:

Start-Date: 2021-11-21  03:34:23
Purge: pve-kernel-5.11.22-4-pve:amd64 ()
End-Date: 2021-11-21  03:34:33

Start-Date: 2021-11-24  03:28:41
Install: pve-kernel-5.13.19-1-pve:amd64 (5.13.19-2, automatic), pve-kernel-5.13:amd64 (7.1-4, automatic)
Upgrade: proxmox-widget-toolkit:amd64 (3.4-2, 3.4-3), pbs-i18n:amd64 (2.5-1, 2.6-2), proxmox-backup:amd64 (2.0-1, 2.1-1), proxmox-backup-client:amd64 (2.0.14-1, 2.1.2-1), proxmox-backup-docs:amd64 (2.0.14-1, 2.1.2-1), proxmox-backup-server:amd64 (2.0.14-1, 2.1.2-1)
End-Date: 2021-11-24  03:31:02

Start-Date: 2021-11-25  03:26:25
Upgrade: pve-kernel-5.13.19-1-pve:amd64 (5.13.19-2, 5.13.19-3)
Remove: pve-kernel-5.11.22-5-pve:amd64 (5.11.22-10)
End-Date: 2021-11-25  03:27:19

Start-Date: 2021-11-26  03:59:56
Purge: pve-kernel-5.11.22-5-pve:amd64 ()
End-Date: 2021-11-26  04:00:14

Start-Date: 2021-11-30  03:17:12
Upgrade: proxmox-widget-toolkit:amd64 (3.4-3, 3.4-4), proxmox-mini-journalreader:amd64 (1.2-1, 1.3-1)
End-Date: 2021-11-30  03:17:14

last |grep boot

reboot   system boot  5.13.19-1-pve    Thu Nov 25 11:53 - 14:15  (02:22)
reboot   system boot  5.11.22-7-pve    Sat Nov 20 14:05 - 11:53 (4+21:47)

so it seems the problem started to occur when switching from 5.11.22-7-pve to 5.13.19-1-pve. as pve-kernel-5.15 is available i try first to upgrade to that and see if the problems disappears as nobody else seems to have this problem.

 

[abma]

Side note: it could be an issue with some driver as the hardware of the host is a bit exotic, its an "Acer Veriton M480 with a Core 2 Duo". I've installed pve-kernel-5.15: lets see what happens.

 

[abma]

First difference with 5.15: write performance on ext4 is much better than with 5.13:

5.13: 30MB/s vs 5.15: 150MB/s

 

[abma]

Ok, with 5.15 the problem seems gone. Thanks for the feedback!

 

[fiona]

Ok, with 5.15 the problem seems gone. Thanks for the feedback!

Thank you for the feedback!