PVE 9 live migration varies from ~190 to ~510 MiB/s on 10GbE; destination memory compaction—how to identify the underlying cause?

[Dad1434]

Hello,

I’m investigating inconsistent live-migration performance on a three-node Proxmox cluster 9.2.21. We found significant memory-compaction overhead on the destination during slow migrations, but I would like to understand what creates that condition before making permanent THP changes.

Hostnames below are anonymized as node A/B/C.

Environment​

• Three Dell PowerEdge R530 hosts.
• Each host: 2 × Xeon E5-2640 v4, 20 physical cores / 40 logical CPUs, 384 GB RAM.
• Shared Ceph storage; the migration logs show no disk-copy phase.
• Dedicated migration VLAN over 10GbE, using secure/SSH migration.
• OVS with a two-port 10GbE LACP bond and MTU 9000.
• Test guest: relatively idle Windows 11 VM, 4 vCPUs, 16 GiB maximum RAM, 4 GiB balloon minimum, VirtIO drivers.
• Guest CPU model: x86-64-v2-AES; machine: pc-q35-11.0+pve2.

Relevant versions:

pve-manager: 9.2.21
running kernel: 7.0.14-20-pve
pve-qemu-kvm: 11.0.3-4
qemu-server: 9.2.10
ceph: 20.2.4-pve5
A newer kernel is installed on node C but was not booted during these tests.

Network testing​

Single-stream iperf3 between node A and node C, bound to the migration addresses:

A → C: 9.81 Gbit/s, 4085 retransmissions over 30 seconds
C → A: 9.72 Gbit/s, 2562 retransmissions over 30 seconds
This confirms near-line-rate throughput for that pair. We have not independently benchmarked B ↔ C yet.

Migration results​

These tests used the same idle 16 GiB guest, with secure migration enabled throughout.

Direction	Destination THP defrag	Reported average	Active RAM migration	Entire task
B → C	madvise	195.3 MiB/s	~84 seconds	100 seconds
C → B	madvise	455.7 MiB/s	~36 seconds	50 seconds
B → C	madvise	188.6 MiB/s	~87 seconds	100 seconds
B → C	defer	512.6 MiB/s	~32 seconds	45 seconds
B → C	Restored to madvise	482.5 MiB/s	~34 seconds	47 seconds

The last three runs transferred approximately 15.8, 15.6 and 15.7 GiB respectively. Actual guest downtime was 165, 79 and 77 ms.

The final fast run after restoring madvise is important: the policy change alone does not explain all the variation. Repeated migrations, background compaction and other memory activity may have changed the host’s memory state.

The per-line speed values also fluctuate considerably. We have been comparing cumulative progress, overall timings and NIC counters rather than treating individual speed values as sustained throughput.

CPU and perf observations​

During a slow B → C migration:

• Destination QEMU main thread averaged approximately 91% of one logical CPU and peaked around 99%.
• Source SSH averaged approximately 40%.
• Migration-interface traffic averaged approximately 208 MiB/s.
• There was little swap-in activity and no meaningful swap-out activity during the capture.

A perf profile of the destination QEMU main thread showed:

13.56% _raw_spin_unlock_irqrestore
10.06% __reset_isolation_pfn
9.85% isolate_freepages_block
6.71% kernel_init_pages
5.75% pageblock_skip_persistent
5.10% suitable_migration_target
4.60% copy_mc_enhanced_fast_string
4.60% isolate_migratepages_block
2.25% migrate_pages_batch
The named compaction/page-migration functions account for roughly 38% of samples, excluding locking and copying.

With defrag=defer, the profile changed to:

14.71% kernel_init_pages
8.74% libc [unresolved symbol]
8.17% rep_movs_alternative
7.60% do_user_addr_fault
3.84% try_charge_memcg
3.06% _raw_spin_unlock_irqrestore
The previous compaction functions were no longer above the 2% reporting threshold.

We attempted a later call-stack capture, but it started after migration completed. We excluded that profile and its post-migration CPU-placement observations from the diagnosis.

THP, memory and NUMA​

Original settings were identical on all three hosts:

THP enabled: always [madvise] never
THP defrag: always defer defer+madvise [madvise] never
khugepaged defrag: 1

vm.compaction_proactiveness = 20
vm.extfrag_threshold = 500
All hosts reported:

mem auto-init: stack:all(zero), heap allocn, heap freeff
Node C is currently back on defrag=madvise. No persistent THP changes were made.

All hosts have 12 × 32 GB DIMMs: A1–A8 and B1–B4. This matches the R530’s asymmetric slot layout and produces approximately 256 GB on one NUMA node and 128 GB on the other.

We observed:

• Much less free memory on the smaller NUMA node.
• Some snapshots with no free order-9-or-higher blocks in a Normal zone, despite substantial aggregate available memory.
• Other snapshots with large blocks available.
• Extensive cumulative compaction scanning/failures on multiple hosts, including the host that received faster.

These snapshots were not all taken during migration, so they do not establish allocation conditions at the exact time of the slowdown.

The test guest’s large anonymous mapping used default memory policy, with both NUMA nodes and all CPUs allowed. No explicit binding was visible. We have not yet captured reliable receiving-thread placement during a slow migration.

Ballooning​

The test guest reported its full 16 GiB allocation.

However, another VM on node C has a 96 GiB maximum and was below that allocation. Between two status snapshots around the latest test, its allocation decreased by another 1000 MiB.

This confirms other guests are changing host memory allocations. It does not establish that ballooning caused the fragmentation. We have not disabled ballooning or performed a controlled fixed-memory comparison.

Questions​

1. Does this pattern suggest ordinary fragmentation/direct THP compaction, NUMA-local allocation pressure, or something else?
2. What tracing or counters would best identify why compaction becomes expensive during incoming migration?
3. Could repeated balloon inflation/deflation in other VMs contribute materially to this behavior, and how would you demonstrate that?
4. Are there known issues relevant to these kernel/QEMU versions?
5. Is defrag=defer an appropriate long-term tradeoff here, or should we first address a different allocation/configuration issue?

I would prefer to preserve the default behavior until we understand the cause. I can provide sanitized migration logs, buddy/pagetype information, counter deltas and correctly timed profiles.

Thanks for any suggestions.