Problems with ksoftirqd/1

M

MikeyMike

New Member
Dec 30, 2019
6
0
1
47
Hi!

In the last days i recognized that my little Qotom IPC seems to be more hot than before. I noticed that my CPU is working 25%.

I tried the following steps:

- Stop all VMs and container - Nothing changed
- Upgrade everything - Nothing changed
- Install a fresh Proxmox via proxmox-ve_6.3-1.iso (only one LAN cable connected) without any VMs/Containers - Nothing changed
- Upgrade to 6.3-2 - Nothing Changed
- Check for new Bios - Is already the newest

I´m not a Linux specialist so i don´t now what i can do anymore.
I think since some of the last Proxmox versions the problem begins. Before some months everything was fine.
I hope somebody can help because i´m actual not happy with this Situation.

Here is my setup:

PC QOTOM Q575G6 - 6 Intel NICS

CPU(s) 4 x Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz (1 Socket)
Kernelversion Linux 5.4.78-1-pve #1 SMP PVE 5.4.78-1 (Mon, 30 Nov 2020 10:57:47 +0100)
PVE Manager Version pve-manager/6.3-2/22f57405
RAM 16GB
HDD Samsung EVO 860 512GB


Code: 
top - 18:27:41 up 15 min,  1 user,  load average: 1.38, 1.65, 1.33
Tasks: 249 total,   2 running, 247 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us, 23.5 sy,  0.0 ni, 66.7 id,  0.1 wa,  0.0 hi,  8.4 si,  0.0 st
MiB Mem :  15936.3 total,  10928.4 free,   4524.1 used,    483.8 buff/cache
MiB Swap:   8192.0 total,   5445.2 free,   2746.8 used.  11063.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                     
    5 root      20   0       0      0      0 R  85.0   0.0  12:52.54 kworker/0:0+pm                               
   10 root      20   0       0      0      0 S  15.3   0.0   2:26.72 ksoftirqd/0                                 
3497 root      20   0 1038828 328012  10836 S   5.7   2.0   0:48.72 kvm                                         
1559 root      20   0 8903688   5.9g   6592 S   3.3  38.2   1:45.82 kvm                                         
1026 www-data  20   0  369204  58884  11744 S   0.7   0.4   0:04.09 pveproxy worker                             
  277 root       0 -20       0      0      0 I   0.3   0.0   0:00.09 kworker/2:1H-kblockd                         
  771 root      20   0    6888    144      0 S   0.3   0.0   0:00.01 iscsid                                       
  856 root      20   0  528804  59948  49900 S   0.3   0.4   0:01.24 pmxcfs                                       
  990 root      20   0  304552  40892   3596 S   0.3   0.3   0:01.68 pve-firewall                                 
1324 100033    20   0   12832    716      0 S   0.3   0.0   0:00.02 lighttpd                                     
3577 root      20   0  328228 117160  23648 S   0.3   0.7   0:01.36 qm                                           
    1 root      20   0   22496   7568   5672 S   0.0   0.0   0:02.75 systemd                                     
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd                                     
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                                       
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp                                   
    6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-kblockd                         
    7 root      20   0       0      0      0 I   0.0   0.0   0:09.82 kworker/0:1-events                           
    9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq                                 
   11 root      20   0       0      0      0 I   0.0   0.0   0:00.13 rcu_sched                                   
   12 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/0                                 
   13 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/0                               
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                                     
   15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1                                     
   16 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/1                               
   17 root      rt   0       0      0      0 S   0.0   0.0   0:00.15 migration/1                                 
   18 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftirqd/1                                 
   20 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/1:0H-kblockd                         
   21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2

cat /proc/interrupts

Code: 
            CPU0       CPU1       CPU2       CPU3     
   0:          6          0          0          0  IR-IO-APIC    2-edge      timer
   8:          0          0          1          0  IR-IO-APIC    8-edge      rtc0
   9:          0          5          0          0  IR-IO-APIC    9-fasteoi   acpi
  14:          0          0          0          0  IR-IO-APIC   14-fasteoi   INT344B:00
  16:          0          0          0          0  IR-IO-APIC   16-fasteoi   idma64.0, i801_smbus, i2c_designware.0
  17:          0          0          0          0  IR-IO-APIC   17-fasteoi   idma64.1, i2c_designware.1
  20:          0          0          0          0  IR-IO-APIC   20-fasteoi   idma64.2
120:          0          0          0          0  DMAR-MSI    0-edge      dmar0
121:          0          0          0          0  DMAR-MSI    1-edge      dmar1
122:          0          0          0          0  IR-PCI-MSI 458752-edge      PCIe PME, aerdrv
123:          0          0          0          0  IR-PCI-MSI 460800-edge      PCIe PME, aerdrv
124:          0          0          0          0  IR-PCI-MSI 462848-edge      PCIe PME, aerdrv
125:          0          0          0          0  IR-PCI-MSI 464896-edge      PCIe PME, aerdrv
126:          0          0          0          0  IR-PCI-MSI 466944-edge      PCIe PME, aerdrv
127:          0          0          0          0  IR-PCI-MSI 468992-edge      PCIe PME, aerdrv
128:     192268          0          0          0  IR-PCI-MSI 376832-edge      ahci[0000:00:17.0]
129:          0          0          1          0  IR-PCI-MSI 524288-edge      enp1s0
130:          0          0          0       5613  IR-PCI-MSI 524289-edge      enp1s0-rx-0
131:       1106          0          0          0  IR-PCI-MSI 524290-edge      enp1s0-rx-1
132:          0       2988          0          0  IR-PCI-MSI 524291-edge      enp1s0-tx-0
133:          0          0       4487          0  IR-PCI-MSI 524292-edge      enp1s0-tx-1
134:          0          0     265549          0  IR-PCI-MSI 327680-edge      xhci_hcd
135:          0          0          0          1  IR-PCI-MSI 1048576-edge      enp2s0
136:       2252          0          0          0  IR-PCI-MSI 1048577-edge      enp2s0-rx-0
137:          0       1106          0          0  IR-PCI-MSI 1048578-edge      enp2s0-rx-1
138:          0          0       2536          0  IR-PCI-MSI 1048579-edge      enp2s0-tx-0
139:          0          0          0       3275  IR-PCI-MSI 1048580-edge      enp2s0-tx-1
150:          1          0          0          0  IR-PCI-MSI 2621440-edge      enp5s0
151:          0       4940          0          0  IR-PCI-MSI 2621441-edge      enp5s0-rx-0
152:          0          0       1107          0  IR-PCI-MSI 2621442-edge      enp5s0-rx-1
153:          0          0          0       8421  IR-PCI-MSI 2621443-edge      enp5s0-tx-0
154:      41048          0          0          0  IR-PCI-MSI 2621444-edge      enp5s0-tx-1
155:          0          1          0          0  IR-PCI-MSI 3145728-edge      enp6s0
156:          0          0      23932          0  IR-PCI-MSI 3145729-edge      enp6s0-rx-0
157:          0          0          0       1107  IR-PCI-MSI 3145730-edge      enp6s0-rx-1
158:       2743          0          0          0  IR-PCI-MSI 3145731-edge      enp6s0-tx-0
159:          0       1500          0          0  IR-PCI-MSI 3145732-edge      enp6s0-tx-1
160:          0         35          0          0  IR-PCI-MSI 360448-edge      mei_me
161:          0          0        189          0  IR-PCI-MSI 32768-edge      i915
162:          0          0          0        102  IR-PCI-MSI 514048-edge      snd_hda_intel:card0
NMI:        267         29         25         27   Non-maskable interrupts
LOC:     555886    1168078    1690050    1222918   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:        267         29         25         27   Performance monitoring interrupts
IWI:          1          1          0          0   IRQ work interrupts
RTR:          0          0          0          0   APIC ICR read retries
RES:       3364     575223     395105     347823   Rescheduling interrupts
CAL:       1988     213645     218124     227093   Function call interrupts
TLB:         28     224367     232873     253645   TLB shootdowns
TRM:       4934       4934       4934       4934   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          8          9          9          9   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0   Posted-interrupt wakeup event

My actual packages

Code: 
proxmox-ve: 6.3-1 (running kernel: 5.4.78-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-2
pve-kernel-helper: 6.3-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-1
libpve-common-perl: 6.3-1
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1
 
Last edited:
MikeyMike said:
In the last days i recognized that my little Qotom IPC seems to be more hot than before.
Click to expand...
Hm... that is a very subjective feeling. Do you have any monitoring that measures the CPU or other temperatures?

MikeyMike said:
I think since some of the last Proxmox versions the problem begins. Before some months everything was fine.
Click to expand...
That a lot of time and many things can have changed. Starting with more load from the Guests or the CPU thermal compound is not as effective anymore.
 
Alwin said:
Hm... that is a very subjective feeling. Do you have any monitoring that measures the CPU or other temperatures?


That a lot of time and many things can have changed. Starting with more load from the Guests or the CPU thermal compound is not as effective anymore.
Click to expand...

I don´t have any monitoring. The heat was the reason to look closer at my installation.

But there should be not such a high CPU consumption with a fresh installation without any containers.
How can i find the reason which Interrupt is responsible for the CPU load? Is it AHCI or the local timer interrupts? Are the counters in a regular range? What can i do?
 
You can try lm-sensors to start with measuring.
MikeyMike said:
How can i find the reason which Interrupt is responsible for the CPU load?
Click to expand...
Run something like atop and look at linux perf.
 
Thank you very much! I installed this tools and found my problem: xhci_hub_control

With echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/unbind and
echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/bind i can reproduce the CPU load.

I need to turn USB austosuspend off. I don´t know why but it works.
I changed the boot option in grub to GRUB_CMDLINE_LINUX_DEFAULT="quiet usbcore.autosuspend=-1"

Why do i need to deactivate this wihout have anything plugged in the Ports? Actual i have, but after my fresh install i had the same behavior with the cpu load. Can somebody help me interpret whats happen here?
I think it will saving power to get the suspend working again correctly.

perf_4.19 record -g -a sleep 10
perf_4.19 report

Code: 
Samples: 66K of event 'cycles:ppp', Event count (approx.): 41639072631
  Children      Self  Command          Shared Object             Symbol
+   71.27%     0.00%  kworker/1:2+pm   [kernel.kallsyms]         [k] ret_from_fork                                          ◆
+   71.27%     0.00%  kworker/1:2+pm   [kernel.kallsyms]         [k] kthread                                                ▒
+   71.27%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] worker_thread                                          ▒
+   71.22%     0.04%  kworker/1:2+pm   [kernel.kallsyms]         [k] process_one_work                                       ▒
+   64.74%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] rpm_idle                                               ▒
+   64.60%     0.07%  kworker/1:2+pm   [kernel.kallsyms]         [k] rpm_suspend                                            ▒
+   64.58%     0.01%  kworker/1:2+pm   [kernel.kallsyms]         [k] pm_runtime_work                                        ▒
+   64.50%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_runtime_idle                                       ▒
+   64.50%     0.03%  kworker/1:2+pm   [kernel.kallsyms]         [k] __rpm_callback                                         ▒
+   64.48%     0.00%  kworker/1:2+pm   [kernel.kallsyms]         [k] __pm_runtime_suspend                                   ▒
+   64.23%     0.01%  kworker/1:2+pm   [kernel.kallsyms]         [k] rpm_callback                                           ▒
+   64.18%     0.03%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_runtime_suspend                                    ▒
+   64.08%     0.05%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_suspend_both                                       ▒
+   59.48%     0.26%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_control_msg                                        ▒
+   57.52%     0.19%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_start_wait_urb                                     ▒
+   53.64%     0.27%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_submit_urb                                         ▒
+   53.14%     0.65%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_hcd_submit_urb                                     ▒
+   46.36%    46.08%  kworker/1:2+pm   [xhci_hcd]                [k] xhci_hub_control                                       ▒
+   41.96%     0.14%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_suspend                                            ▒
+   41.20%     0.06%  kworker/1:2+pm   [kernel.kallsyms]         [k] set_port_feature                                       ▒
+   18.73%     0.19%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_ext_port_status                                    ▒
+   13.78%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_resume_interface.isra.11                           ▒
+   13.75%     0.06%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_resume                                             ▒
+   13.66%     0.19%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_activate                                           ▒
+    9.86%     0.00%  ksoftirqd/1      [kernel.kallsyms]         [k] ret_from_fork                                          ▒
+    9.86%     0.00%  ksoftirqd/1      [kernel.kallsyms]         [k] kthread                                                ▒
+    9.35%     0.56%  ksoftirqd/1      [kernel.kallsyms]         [k] smpboot_thread_fn                                      ▒
+    8.25%     0.01%  kworker/1:2+pm   [kernel.kallsyms]         [k] generic_suspend                                        ▒
+    8.23%     0.03%  kworker/1:2+pm   [kernel.kallsyms]         [k] hcd_bus_suspend                                        ▒
+    8.18%     8.15%  kworker/1:2+pm   [xhci_hcd]                [k] xhci_bus_suspend                                       ▒
+    7.75%     0.00%  swapper          [kernel.kallsyms]         [k] secondary_startup_64                                   ▒
+    7.75%     0.00%  swapper          [kernel.kallsyms]         [k] cpu_startup_entry
 
Last edited:
MikeyMike said:
Why do i need to deactivate this wihout have anything plugged in the Ports? Actual i have, but after my fresh install i had the same behavior with the cpu load.
Click to expand...
So, do you have something plugged in or not?
 
Having the exact same issue on 7.0-11 with Kernel 5.11.22-1-pve and 5.11.22-3-pve

It seems at least somewhat related to this very old kernel bug that was picked up on Ubuntu
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1488426

Editing the grub boot option to include usbcore.autosuspend=-1 does not seem to work for me though. Most likely Proxmox has different ways of updating grub config files these days. I will post the method here once I figure it out
 
You must log in or register to reply here.
Top Bottom