7.0.14-6-pve: acpi_pad consumes CPU and ZFS write performance drops from 257 MiB/s to 7.5 MiB/s

[Radik Makhmudov]

Hello,

After upgrading from kernel 7.0.2-6-pve to 7.0.14-6-pve, I observed a reproducible and severe storage performance regression.

System​

• Proxmox VE 9.2.5
• 4 × Samsung 870 EVO 1TB SSD
• ZFS pool: two mirrored vdevs
• Pool status: ONLINE
• No read/write/checksum errors
• Latest scrub completed successfully

Test​

The same fio test was executed on both kernels.

fio \
--name=seq-write \
--filename=/tank/testfile \
--rw=write \
--bs=1M \
--direct=1 \
--ioengine=psync \
--size=8G \
--group_reporting

Results​

Kernel 7.0.2-6-pve​

WRITE: 257 MiB/s (270 MB/s)
runtime: 31.9 seconds
average latency: ~3.9 ms

Kernel 7.0.14-6-pve​

WRITE: 7736 KiB/s (~7.5 MiB/s)
runtime: 1084 seconds
average latency: ~132 ms
maximum latency: >2 seconds
This is approximately a 34× performance regression using the identical hardware and test.

Host behaviour on 7.0.14-6-pve​

At the same time the system shows:

• system CPU: ~85%
• idle CPU: 2–4%
• iowait: ~0%
• load average: 54–60

htop shows all 32 logical CPUs continuously loaded at approximately 90–100%.

The highest CPU consumers are kernel threads:

acpi_pad/2
acpi_pad/6
acpi_pad/9
acpi_pad/14
...
Each thread consumes approximately 95% CPU.

Example:

PID COMMAND
1791079 acpi_pad/18
1790964 acpi_pad/6
1790995 acpi_pad/14
...

Rollback​

This is a production server, so I did not unload or blacklist the acpi_pad module.

Instead, I rebooted back to kernel 7.0.2-6-pve.

Immediately after the rollback:

• storage performance returned to approximately 257 MiB/s
• load average returned to normal
• excessive acpi_pad CPU usage disappeared

The issue is reproducible every time I switch between the two kernels.

Question​

Has anyone seen similar behaviour on kernel 7.0.14-6-pve?

Could this be a kernel regression related to ACPI, CPU scheduling, or another subsystem?

Complete report.txt, pveversion -v, fio output, dstat, htop screenshots, and additional diagnostics is attached

 

[daanw]

Kudos for the thorough report!
What platform/server/hardware are you on?
Anything related in the journal?

 

[Radik Makhmudov]

Thanks for your reply.

Here's the system information for the affected host.
I also reviewed the journal from the affected boot (kernel 7.0.14-6-pve). I couldn't find any obvious kernel warnings or errors related to this issue. The filtered log is attached.

Filters:
journalctl -b -1 -p warning
journalctl -b -1 -p err
journalctl -b -1 | grep -Ei 'BUG|Call Trace|Oops|watchdog|lockup'
journalctl -b -1 | grep -Ei 'acpi_pad|powercap|intel_rapl'


Chassis: server
Machine ID: bac9f42a6bfb404a86efde6502e5715f
Boot ID: 5eab2535db104d39ab18e1b6a47c7671
Operating System: Debian GNU/Linux 13 (trixie)
Kernel: Linux 7.0.2-6-pve
Architecture: x86-64
Hardware Vendor: Lenovo
Hardware Model: 70QNS01J00
Firmware Version: VB3TS505
Firmware Date: Fri 2022-10-14


PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
DEBIAN_VERSION_FULL=13.6



CPU INFORMATION
Logical CPU Cores: 32

Architecture: x86_64
CPU(s): 32
On-line CPU(s) list: 0-31
Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
CPU(s) scaling MHz: 25%
Virtualization: VT-x
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31

MEMORY INFORMATION
total used free shared buff/cache available
Mem: 157Gi 64Gi 5.8Gi 81Mi 87Gi 92Gi
Swap: 8.0Gi 20Mi 8.0Gi