Update – controlled restore/backup tests and CPU behavior
I performed three controlled tests on the same PVE host and the same ZFS pool:
During all three tests the host was otherwise idle: all VMs were stopped and there was no other significant workload on the host.
TEST 1 - Restore, acpi_pad ON
During the restore I observed very high CPU utilization, with a significant amount of CPU time spent in %system. The acpi_pad/* kernel threads were also highly active.
I unloaded the module: modprobe -r acpi_pad and repeated the same restore.
The restore completed approximately 3.5% faster. However, the important result is that the high %system utilization did not disappear. Periods of very high kernel CPU utilization were still present during the write workload.
So unloading acpi_pad has a small measurable effect, but does not resolve the underlying problem.
TEST 3 - Backup, acpi_pad ON
For a reverse-direction control test, I backed up the same VM from the same ZFS pool to PBS. I verified with lsmod that acpi_pad was loaded.
The VM consists of two ZVOLs:
300 GiB+100 GiB
The backup completed in 45m 42s, with an average reported throughput of approximately 149.5 MiB/s.
CPU behavior during this read workload was completely different:
average %sys ~2.5%
average %iowait ~0.4%
average %idle ~94%
Current observation:
The three tests show a fairly clear asymmetry:
PBS -> ZFS/ZVOL WRITE acpi_pad ON -> high %system, performance problem
PBS -> ZFS/ZVOL WRITE acpi_pad OFF -> high %system remains, ~3.5% faster
ZFS/ZVOL -> PBS READ acpi_pad ON -> normal CPU, ~149.5 MiB/s average
This makes me less convinced that acpi_pad itself is the root cause. It has a small measurable effect on restore performance, but its presence alone is clearly not sufficient to reproduce the abnormal CPU behavior.
At this point, the stronger correlation appears to be the I/O direction:
WRITE to ZFS/ZVOL -> high kernel/system CPU utilization
READ from ZFS/ZVOL -> normal CPU utilization
This suggests that the abnormal behavior is associated specifically with the write path, rather than with acpi_pad or general ZFS activity.
I do not want to speculate yet whether the cause is inside ZFS/ZVOL, the MegaRAID/driver/storage path, the SSDs, or somewhere else in the platform.
I have attached the CPU and storage statistics from all three tests.
The filenames are prefixed with restore_pad-on_, restore_pad-off_, and backup_pad-on_ to make the three test conditions easy to compare.
My next step will probably be to capture perf during one of the high-%system periods of the restore/write workload to determine where the kernel is actually spending CPU time.
I performed three controlled tests on the same PVE host and the same ZFS pool:
- VM restore from PBS → ZFS/ZVOL with acpi_pad loaded
- The same VM restore from PBS → ZFS/ZVOL with acpi_pad unloaded
- VM backup from ZFS/ZVOL → PBS with acpi_pad loaded
During all three tests the host was otherwise idle: all VMs were stopped and there was no other significant workload on the host.
TEST 1 - Restore, acpi_pad ON
During the restore I observed very high CPU utilization, with a significant amount of CPU time spent in %system. The acpi_pad/* kernel threads were also highly active.
TEST 2 - Restore, acpi_pad OFFI unloaded the module: modprobe -r acpi_pad and repeated the same restore.
The restore completed approximately 3.5% faster. However, the important result is that the high %system utilization did not disappear. Periods of very high kernel CPU utilization were still present during the write workload.
So unloading acpi_pad has a small measurable effect, but does not resolve the underlying problem.
TEST 3 - Backup, acpi_pad ON
For a reverse-direction control test, I backed up the same VM from the same ZFS pool to PBS. I verified with lsmod that acpi_pad was loaded.
The VM consists of two ZVOLs:
300 GiB+100 GiB
The backup completed in 45m 42s, with an average reported throughput of approximately 149.5 MiB/s.
CPU behavior during this read workload was completely different:
average %sys ~2.5%
average %iowait ~0.4%
average %idle ~94%
There was no CPU saturation comparable to the restore/write tests, despite acpi_pad being loaded.Current observation:
The three tests show a fairly clear asymmetry:
PBS -> ZFS/ZVOL WRITE acpi_pad ON -> high %system, performance problem
PBS -> ZFS/ZVOL WRITE acpi_pad OFF -> high %system remains, ~3.5% faster
ZFS/ZVOL -> PBS READ acpi_pad ON -> normal CPU, ~149.5 MiB/s average
This makes me less convinced that acpi_pad itself is the root cause. It has a small measurable effect on restore performance, but its presence alone is clearly not sufficient to reproduce the abnormal CPU behavior.
At this point, the stronger correlation appears to be the I/O direction:
WRITE to ZFS/ZVOL -> high kernel/system CPU utilization
READ from ZFS/ZVOL -> normal CPU utilization
This suggests that the abnormal behavior is associated specifically with the write path, rather than with acpi_pad or general ZFS activity.
I do not want to speculate yet whether the cause is inside ZFS/ZVOL, the MegaRAID/driver/storage path, the SSDs, or somewhere else in the platform.
I have attached the CPU and storage statistics from all three tests.
The filenames are prefixed with restore_pad-on_, restore_pad-off_, and backup_pad-on_ to make the three test conditions easy to compare.
My next step will probably be to capture perf during one of the high-%system periods of the restore/write workload to determine where the kernel is actually spending CPU time.