Hi everyone,
Older migration logs showed rates around 500 MiB/s, but recent migrations are much slower. Switching from secure to insecure migration (confirmed direct TCP) has not restored performance.
This is a significant operational blocker for us: at these rates, migrating large-memory VMs, becomes impractical.
Environment:
- pve-manager 9.2.20, kernel 7.0.14-17-pve.
- Destination: 2 × Xeon Gold 6146, MemTotal approximately 503 GiB, two NUMA nodes, no swap.
- A migration subnet is configured; VM disks are on shared storage, with no disk-copy phase in the migration logs.
- The destination has been on this kernel for about 41 days, including when migrations were faster.
For the same 64 GiB VM and source/destination pair, the average VM-state transfer rate fell from approximately 468 MiB/s on October 15 to 122 MiB/s on November 5. The workload was not controlled between these dates.
Another 64 GiB VM averaged approximately 178 MiB/s and completed with TASK OK. One destination snapshot during this transfer showed approximately 238 GiB MemAvailable.
I measured these host-wide counter increments over separate 30-second intervals during and after that migration:
- compact_stall: during 6,527; after 3
- compact_fail: during 6,492; after 3
- allocstall_movable: during 2,188; after 1
- thp_fault_alloc: during 107; after 0
- thp_fault_fallback: during 2,115; after 1
- Memory PSI full (% of the interval): during 4.88%; after 0.0027%
The PSI percentages were calculated from the change in the full total counter, not avg10/avg60. Source-node memory PSI was zero in the available minute-resolution samples.
At measurement time, THP enabled and defrag were both madvise. The kernel configuration has CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y, with no kho= boot override. An earlier destination snapshot showed CmaTotal=0 and CmaFree=442548 kB. The tested VM has no PCI passthrough.
Separately, I observed a roughly 15–20 minute startup/RAM-allocation delay with a 768 GiB VM on another node; whether this is related is unknown.
Does this suggest incoming migration is being limited by THP compaction/direct reclaim? Would temporarily changing only THP defrag to never or defer be a useful test? Are there known kernel/kexec handover (KHO) issues to investigate, or additional diagnostics you would collect first?
No THP/KHO changes were made for these measurements. I can provide pveversion -v if needed.
Thanks!
Older migration logs showed rates around 500 MiB/s, but recent migrations are much slower. Switching from secure to insecure migration (confirmed direct TCP) has not restored performance.
This is a significant operational blocker for us: at these rates, migrating large-memory VMs, becomes impractical.
Environment:
- pve-manager 9.2.20, kernel 7.0.14-17-pve.
- Destination: 2 × Xeon Gold 6146, MemTotal approximately 503 GiB, two NUMA nodes, no swap.
- A migration subnet is configured; VM disks are on shared storage, with no disk-copy phase in the migration logs.
- The destination has been on this kernel for about 41 days, including when migrations were faster.
For the same 64 GiB VM and source/destination pair, the average VM-state transfer rate fell from approximately 468 MiB/s on October 15 to 122 MiB/s on November 5. The workload was not controlled between these dates.
Another 64 GiB VM averaged approximately 178 MiB/s and completed with TASK OK. One destination snapshot during this transfer showed approximately 238 GiB MemAvailable.
I measured these host-wide counter increments over separate 30-second intervals during and after that migration:
- compact_stall: during 6,527; after 3
- compact_fail: during 6,492; after 3
- allocstall_movable: during 2,188; after 1
- thp_fault_alloc: during 107; after 0
- thp_fault_fallback: during 2,115; after 1
- Memory PSI full (% of the interval): during 4.88%; after 0.0027%
The PSI percentages were calculated from the change in the full total counter, not avg10/avg60. Source-node memory PSI was zero in the available minute-resolution samples.
At measurement time, THP enabled and defrag were both madvise. The kernel configuration has CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y, with no kho= boot override. An earlier destination snapshot showed CmaTotal=0 and CmaFree=442548 kB. The tested VM has no PCI passthrough.
Separately, I observed a roughly 15–20 minute startup/RAM-allocation delay with a 768 GiB VM on another node; whether this is related is unknown.
Does this suggest incoming migration is being limited by THP compaction/direct reclaim? Would temporarily changing only THP defrag to never or defer be a useful test? Are there known kernel/kexec handover (KHO) issues to investigate, or additional diagnostics you would collect first?
No THP/KHO changes were made for these measurements. I can provide pveversion -v if needed.
Thanks!
Last edited: