Problems with ksoftirqd/1

M

MikeyMike

New Member
Dec 30, 2019
6
0
1
46
Hi!

In the last days i recognized that my little Qotom IPC seems to be more hot than before. I noticed that my CPU is working 25%.

I tried the following steps:

- Stop all VMs and container - Nothing changed
- Upgrade everything - Nothing changed
- Install a fresh Proxmox via proxmox-ve_6.3-1.iso (only one LAN cable connected) without any VMs/Containers - Nothing changed
- Upgrade to 6.3-2 - Nothing Changed
- Check for new Bios - Is already the newest

I´m not a Linux specialist so i don´t now what i can do anymore.
I think since some of the last Proxmox versions the problem begins. Before some months everything was fine.
I hope somebody can help because i´m actual not happy with this Situation.

Here is my setup:

PC QOTOM Q575G6 - 6 Intel NICS

CPU(s) 4 x Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz (1 Socket)
Kernelversion Linux 5.4.78-1-pve #1 SMP PVE 5.4.78-1 (Mon, 30 Nov 2020 10:57:47 +0100)
PVE Manager Version pve-manager/6.3-2/22f57405
RAM 16GB
HDD Samsung EVO 860 512GB


Code: 
top - 18:27:41 up 15 min,  1 user,  load average: 1.38, 1.65, 1.33
Tasks: 249 total,   2 running, 247 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us, 23.5 sy,  0.0 ni, 66.7 id,  0.1 wa,  0.0 hi,  8.4 si,  0.0 st
MiB Mem :  15936.3 total,  10928.4 free,   4524.1 used,    483.8 buff/cache
MiB Swap:   8192.0 total,   5445.2 free,   2746.8 used.  11063.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                     
    5 root      20   0       0      0      0 R  85.0   0.0  12:52.54 kworker/0:0+pm                               
   10 root      20   0       0      0      0 S  15.3   0.0   2:26.72 ksoftirqd/0                                 
3497 root      20   0 1038828 328012  10836 S   5.7   2.0   0:48.72 kvm                                         
1559 root      20   0 8903688   5.9g   6592 S   3.3  38.2   1:45.82 kvm                                         
1026 www-data  20   0  369204  58884  11744 S   0.7   0.4   0:04.09 pveproxy worker                             
  277 root       0 -20       0      0      0 I   0.3   0.0   0:00.09 kworker/2:1H-kblockd                         
  771 root      20   0    6888    144      0 S   0.3   0.0   0:00.01 iscsid                                       
  856 root      20   0  528804  59948  49900 S   0.3   0.4   0:01.24 pmxcfs                                       
  990 root      20   0  304552  40892   3596 S   0.3   0.3   0:01.68 pve-firewall                                 
1324 100033    20   0   12832    716      0 S   0.3   0.0   0:00.02 lighttpd                                     
3577 root      20   0  328228 117160  23648 S   0.3   0.7   0:01.36 qm                                           
    1 root      20   0   22496   7568   5672 S   0.0   0.0   0:02.75 systemd                                     
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd                                     
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                                       
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp                                   
    6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-kblockd                         
    7 root      20   0       0      0      0 I   0.0   0.0   0:09.82 kworker/0:1-events                           
    9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq                                 
   11 root      20   0       0      0      0 I   0.0   0.0   0:00.13 rcu_sched                                   
   12 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/0                                 
   13 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/0                               
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                                     
   15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1                                     
   16 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/1                               
   17 root      rt   0       0      0      0 S   0.0   0.0   0:00.15 migration/1                                 
   18 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftirqd/1                                 
   20 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/1:0H-kblockd                         
   21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2

cat /proc/interrupts

Code: 
            CPU0       CPU1       CPU2       CPU3     
   0:          6          0          0          0  IR-IO-APIC    2-edge      timer
   8:          0          0          1          0  IR-IO-APIC    8-edge      rtc0
   9:          0          5          0          0  IR-IO-APIC    9-fasteoi   acpi
  14:          0          0          0          0  IR-IO-APIC   14-fasteoi   INT344B:00
  16:          0          0          0          0  IR-IO-APIC   16-fasteoi   idma64.0, i801_smbus, i2c_designware.0
  17:          0          0          0          0  IR-IO-APIC   17-fasteoi   idma64.1, i2c_designware.1
  20:          0          0          0          0  IR-IO-APIC   20-fasteoi   idma64.2
120:          0          0          0          0  DMAR-MSI    0-edge      dmar0
121:          0          0          0          0  DMAR-MSI    1-edge      dmar1
122:          0          0          0          0  IR-PCI-MSI 458752-edge      PCIe PME, aerdrv
123:          0          0          0          0  IR-PCI-MSI 460800-edge      PCIe PME, aerdrv
124:          0          0          0          0  IR-PCI-MSI 462848-edge      PCIe PME, aerdrv
125:          0          0          0          0  IR-PCI-MSI 464896-edge      PCIe PME, aerdrv
126:          0          0          0          0  IR-PCI-MSI 466944-edge      PCIe PME, aerdrv
127:          0          0          0          0  IR-PCI-MSI 468992-edge      PCIe PME, aerdrv
128:     192268          0          0          0  IR-PCI-MSI 376832-edge      ahci[0000:00:17.0]
129:          0          0          1          0  IR-PCI-MSI 524288-edge      enp1s0
130:          0          0          0       5613  IR-PCI-MSI 524289-edge      enp1s0-rx-0
131:       1106          0          0          0  IR-PCI-MSI 524290-edge      enp1s0-rx-1
132:          0       2988          0          0  IR-PCI-MSI 524291-edge      enp1s0-tx-0
133:          0          0       4487          0  IR-PCI-MSI 524292-edge      enp1s0-tx-1
134:          0          0     265549          0  IR-PCI-MSI 327680-edge      xhci_hcd
135:          0          0          0          1  IR-PCI-MSI 1048576-edge      enp2s0
136:       2252          0          0          0  IR-PCI-MSI 1048577-edge      enp2s0-rx-0
137:          0       1106          0          0  IR-PCI-MSI 1048578-edge      enp2s0-rx-1
138:          0          0       2536          0  IR-PCI-MSI 1048579-edge      enp2s0-tx-0
139:          0          0          0       3275  IR-PCI-MSI 1048580-edge      enp2s0-tx-1
150:          1          0          0          0  IR-PCI-MSI 2621440-edge      enp5s0
151:          0       4940          0          0  IR-PCI-MSI 2621441-edge      enp5s0-rx-0
152:          0          0       1107          0  IR-PCI-MSI 2621442-edge      enp5s0-rx-1
153:          0          0          0       8421  IR-PCI-MSI 2621443-edge      enp5s0-tx-0
154:      41048          0          0          0  IR-PCI-MSI 2621444-edge      enp5s0-tx-1
155:          0          1          0          0  IR-PCI-MSI 3145728-edge      enp6s0
156:          0          0      23932          0  IR-PCI-MSI 3145729-edge      enp6s0-rx-0
157:          0          0          0       1107  IR-PCI-MSI 3145730-edge      enp6s0-rx-1
158:       2743          0          0          0  IR-PCI-MSI 3145731-edge      enp6s0-tx-0
159:          0       1500          0          0  IR-PCI-MSI 3145732-edge      enp6s0-tx-1
160:          0         35          0          0  IR-PCI-MSI 360448-edge      mei_me
161:          0          0        189          0  IR-PCI-MSI 32768-edge      i915
162:          0          0          0        102  IR-PCI-MSI 514048-edge      snd_hda_intel:card0
NMI:        267         29         25         27   Non-maskable interrupts
LOC:     555886    1168078    1690050    1222918   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:        267         29         25         27   Performance monitoring interrupts
IWI:          1          1          0          0   IRQ work interrupts
RTR:          0          0          0          0   APIC ICR read retries
RES:       3364     575223     395105     347823   Rescheduling interrupts
CAL:       1988     213645     218124     227093   Function call interrupts
TLB:         28     224367     232873     253645   TLB shootdowns
TRM:       4934       4934       4934       4934   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          8          9          9          9   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0   Posted-interrupt wakeup event

My actual packages

Code: 
proxmox-ve: 6.3-1 (running kernel: 5.4.78-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-2
pve-kernel-helper: 6.3-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-1
libpve-common-perl: 6.3-1
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1
 
Last edited:
MikeyMike said:
In the last days i recognized that my little Qotom IPC seems to be more hot than before.
Click to expand...
Hm... that is a very subjective feeling. Do you have any monitoring that measures the CPU or other temperatures?

MikeyMike said:
I think since some of the last Proxmox versions the problem begins. Before some months everything was fine.
Click to expand...
That a lot of time and many things can have changed. Starting with more load from the Guests or the CPU thermal compound is not as effective anymore.
 
Alwin said:
Hm... that is a very subjective feeling. Do you have any monitoring that measures the CPU or other temperatures?


That a lot of time and many things can have changed. Starting with more load from the Guests or the CPU thermal compound is not as effective anymore.
Click to expand...

I don´t have any monitoring. The heat was the reason to look closer at my installation.

But there should be not such a high CPU consumption with a fresh installation without any containers.
How can i find the reason which Interrupt is responsible for the CPU load? Is it AHCI or the local timer interrupts? Are the counters in a regular range? What can i do?
 
You can try lm-sensors to start with measuring.
MikeyMike said:
How can i find the reason which Interrupt is responsible for the CPU load?
Click to expand...
Run something like atop and look at linux perf.
 
Thank you very much! I installed this tools and found my problem: xhci_hub_control

With echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/unbind and
echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/bind i can reproduce the CPU load.

I need to turn USB austosuspend off. I don´t know why but it works.
I changed the boot option in grub to GRUB_CMDLINE_LINUX_DEFAULT="quiet usbcore.autosuspend=-1"

Why do i need to deactivate this wihout have anything plugged in the Ports? Actual i have, but after my fresh install i had the same behavior with the cpu load. Can somebody help me interpret whats happen here?
I think it will saving power to get the suspend working again correctly.

perf_4.19 record -g -a sleep 10
perf_4.19 report

Code: 
Samples: 66K of event 'cycles:ppp', Event count (approx.): 41639072631
  Children      Self  Command          Shared Object             Symbol
+   71.27%     0.00%  kworker/1:2+pm   [kernel.kallsyms]         [k] ret_from_fork                                          ◆
+   71.27%     0.00%  kworker/1:2+pm   [kernel.kallsyms]         [k] kthread                                                ▒
+   71.27%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] worker_thread                                          ▒
+   71.22%     0.04%  kworker/1:2+pm   [kernel.kallsyms]         [k] process_one_work                                       ▒
+   64.74%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] rpm_idle                                               ▒
+   64.60%     0.07%  kworker/1:2+pm   [kernel.kallsyms]         [k] rpm_suspend                                            ▒
+   64.58%     0.01%  kworker/1:2+pm   [kernel.kallsyms]         [k] pm_runtime_work                                        ▒
+   64.50%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_runtime_idle                                       ▒
+   64.50%     0.03%  kworker/1:2+pm   [kernel.kallsyms]         [k] __rpm_callback                                         ▒
+   64.48%     0.00%  kworker/1:2+pm   [kernel.kallsyms]         [k] __pm_runtime_suspend                                   ▒
+   64.23%     0.01%  kworker/1:2+pm   [kernel.kallsyms]         [k] rpm_callback                                           ▒
+   64.18%     0.03%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_runtime_suspend                                    ▒
+   64.08%     0.05%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_suspend_both                                       ▒
+   59.48%     0.26%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_control_msg                                        ▒
+   57.52%     0.19%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_start_wait_urb                                     ▒
+   53.64%     0.27%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_submit_urb                                         ▒
+   53.14%     0.65%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_hcd_submit_urb                                     ▒
+   46.36%    46.08%  kworker/1:2+pm   [xhci_hcd]                [k] xhci_hub_control                                       ▒
+   41.96%     0.14%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_suspend                                            ▒
+   41.20%     0.06%  kworker/1:2+pm   [kernel.kallsyms]         [k] set_port_feature                                       ▒
+   18.73%     0.19%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_ext_port_status                                    ▒
+   13.78%     0.02%  kworker/1:2+pm   [kernel.kallsyms]         [k] usb_resume_interface.isra.11                           ▒
+   13.75%     0.06%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_resume                                             ▒
+   13.66%     0.19%  kworker/1:2+pm   [kernel.kallsyms]         [k] hub_activate                                           ▒
+    9.86%     0.00%  ksoftirqd/1      [kernel.kallsyms]         [k] ret_from_fork                                          ▒
+    9.86%     0.00%  ksoftirqd/1      [kernel.kallsyms]         [k] kthread                                                ▒
+    9.35%     0.56%  ksoftirqd/1      [kernel.kallsyms]         [k] smpboot_thread_fn                                      ▒
+    8.25%     0.01%  kworker/1:2+pm   [kernel.kallsyms]         [k] generic_suspend                                        ▒
+    8.23%     0.03%  kworker/1:2+pm   [kernel.kallsyms]         [k] hcd_bus_suspend                                        ▒
+    8.18%     8.15%  kworker/1:2+pm   [xhci_hcd]                [k] xhci_bus_suspend                                       ▒
+    7.75%     0.00%  swapper          [kernel.kallsyms]         [k] secondary_startup_64                                   ▒
+    7.75%     0.00%  swapper          [kernel.kallsyms]         [k] cpu_startup_entry
 
Last edited:
MikeyMike said:
Why do i need to deactivate this wihout have anything plugged in the Ports? Actual i have, but after my fresh install i had the same behavior with the cpu load.
Click to expand...
So, do you have something plugged in or not?
 
Having the exact same issue on 7.0-11 with Kernel 5.11.22-1-pve and 5.11.22-3-pve

It seems at least somewhat related to this very old kernel bug that was picked up on Ubuntu
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1488426

Editing the grub boot option to include usbcore.autosuspend=-1 does not seem to work for me though. Most likely Proxmox has different ways of updating grub config files these days. I will post the method here once I figure it out
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom