PVE 9.1.4 - Shutdown During Bulk Migration

V

vdisk

New Member
Proxmox Subscriber
Feb 5, 2026
3
0
1
Hello!

Earlier today, I put two of my datacenter nodes into maintenance mode as I planned on making some PCIe configuration changes. After putting the nodes into maintenance mode, I used bulk migration to split/migrate their workloads to the remaining two nodes in my datacenter.

The bulk migration tasks showed an OK status, along with OK statuses returned for start events on my VMs, but the guests did not actually come online and remained in a powered-off state.

Is a shutdown during bulk migration expected behavior? In my previous use of individual live migrations, I had no problems with the guest states being preserved and remaining active.

Thanks!
 
For more context, all hypervisor hosts have identical hardware as well as network configuration through SDN.

The processors are all set to "host" in the guest configurations.

Additionally, all guests have their disks on shared datacenter storage -- no ISOs or other devices are being utilized locally.
 
Last edited:
SteveITS said:
We have ours set to automatically migrate VMs when a node enters maintenance mode and it works fine. (Using Ceph)
Did they log errors starting?
Click to expand...

They did not log any errors - both start tasks returned an OK status and the main VMs in concern both show a successful live/online migration in their logs:

Guest 1:
Code: 
2026-07-14 10:12:24 conntrack state migration not supported or disabled, active connections might get dropped
2026-07-14 10:12:24 use dedicated network address for sending migration traffic (x.x.x.x)
2026-07-14 10:12:24 starting migration of VM 123 to node 'pve04' (x.x.x.x)
2026-07-14 10:12:24 starting VM 123 on remote node 'pve04'
2026-07-14 10:12:28 start remote tunnel
2026-07-14 10:12:29 ssh tunnel ver 1
2026-07-14 10:12:29 starting online/live migration on unix:/run/qemu-server/123.migrate
2026-07-14 10:12:29 set migration capabilities
2026-07-14 10:12:29 migration speed limit: 500.0 MiB/s
2026-07-14 10:12:29 migration downtime limit: 100 ms
2026-07-14 10:12:29 migration cachesize: 2.0 GiB
2026-07-14 10:12:29 set migration parameters
2026-07-14 10:12:29 start migrate command to unix:/run/qemu-server/123.migrate
2026-07-14 10:12:30 migration active, transferred 164.9 MiB of 16.0 GiB VM-state, 214.0 MiB/s
2026-07-14 10:12:31 migration active, transferred 563.9 MiB of 16.0 GiB VM-state, 257.1 MiB/s
2026-07-14 10:12:32 migration active, transferred 925.2 MiB of 16.0 GiB VM-state, 506.2 MiB/s
2026-07-14 10:12:33 migration active, transferred 1.2 GiB of 16.0 GiB VM-state, 505.3 MiB/s
2026-07-14 10:12:34 migration active, transferred 1.6 GiB of 16.0 GiB VM-state, 174.8 MiB/s
2026-07-14 10:12:35 migration active, transferred 1.9 GiB of 16.0 GiB VM-state, 322.2 MiB/s
2026-07-14 10:12:36 migration active, transferred 2.4 GiB of 16.0 GiB VM-state, 323.2 MiB/s
2026-07-14 10:12:37 migration active, transferred 2.6 GiB of 16.0 GiB VM-state, 528.9 MiB/s
2026-07-14 10:12:38 migration active, transferred 3.1 GiB of 16.0 GiB VM-state, 336.4 MiB/s
2026-07-14 10:12:39 migration active, transferred 3.4 GiB of 16.0 GiB VM-state, 461.2 MiB/s
2026-07-14 10:12:40 migration active, transferred 3.8 GiB of 16.0 GiB VM-state, 515.7 MiB/s
2026-07-14 10:12:41 migration active, transferred 4.1 GiB of 16.0 GiB VM-state, 355.8 MiB/s
2026-07-14 10:12:42 migration active, transferred 4.5 GiB of 16.0 GiB VM-state, 509.2 MiB/s
2026-07-14 10:12:43 migration active, transferred 4.9 GiB of 16.0 GiB VM-state, 317.4 MiB/s
2026-07-14 10:12:44 migration active, transferred 5.2 GiB of 16.0 GiB VM-state, 238.1 MiB/s
2026-07-14 10:12:45 migration active, transferred 5.6 GiB of 16.0 GiB VM-state, 510.6 MiB/s
2026-07-14 10:12:46 migration active, transferred 6.0 GiB of 16.0 GiB VM-state, 514.5 MiB/s
2026-07-14 10:12:47 migration active, transferred 6.4 GiB of 16.0 GiB VM-state, 500.5 MiB/s
2026-07-14 10:12:48 migration active, transferred 6.8 GiB of 16.0 GiB VM-state, 382.3 MiB/s
2026-07-14 10:12:49 migration active, transferred 7.2 GiB of 16.0 GiB VM-state, 512.6 MiB/s
2026-07-14 10:12:50 migration active, transferred 7.7 GiB of 16.0 GiB VM-state, 513.0 MiB/s
2026-07-14 10:12:51 migration active, transferred 8.1 GiB of 16.0 GiB VM-state, 353.1 MiB/s
2026-07-14 10:12:52 migration active, transferred 8.4 GiB of 16.0 GiB VM-state, 454.0 MiB/s
2026-07-14 10:12:53 migration active, transferred 8.8 GiB of 16.0 GiB VM-state, 428.3 MiB/s
2026-07-14 10:12:54 migration active, transferred 9.2 GiB of 16.0 GiB VM-state, 290.5 MiB/s
2026-07-14 10:12:55 migration active, transferred 9.5 GiB of 16.0 GiB VM-state, 519.0 MiB/s
2026-07-14 10:12:56 migration active, transferred 9.9 GiB of 16.0 GiB VM-state, 614.2 MiB/s
2026-07-14 10:12:57 migration active, transferred 10.2 GiB of 16.0 GiB VM-state, 67.9 MiB/s
2026-07-14 10:12:58 migration active, transferred 10.5 GiB of 16.0 GiB VM-state, 315.0 MiB/s
2026-07-14 10:12:59 migration active, transferred 10.7 GiB of 16.0 GiB VM-state, 84.6 MiB/s
2026-07-14 10:13:00 migration active, transferred 10.9 GiB of 16.0 GiB VM-state, 314.4 MiB/s
2026-07-14 10:13:01 migration active, transferred 11.0 GiB of 16.0 GiB VM-state, 288.5 MiB/s
2026-07-14 10:13:02 migration active, transferred 11.4 GiB of 16.0 GiB VM-state, 383.6 MiB/s
2026-07-14 10:13:03 migration active, transferred 11.7 GiB of 16.0 GiB VM-state, 676.1 MiB/s
2026-07-14 10:13:04 migration active, transferred 11.8 GiB of 16.0 GiB VM-state, 326.3 MiB/s
2026-07-14 10:13:05 migration active, transferred 11.9 GiB of 16.0 GiB VM-state, 340.5 MiB/s
2026-07-14 10:13:06 migration active, transferred 12.1 GiB of 16.0 GiB VM-state, 357.2 MiB/s
2026-07-14 10:13:07 migration active, transferred 12.2 GiB of 16.0 GiB VM-state, 307.2 MiB/s
2026-07-14 10:13:08 migration active, transferred 12.3 GiB of 16.0 GiB VM-state, 318.5 MiB/s
2026-07-14 10:13:09 migration active, transferred 12.4 GiB of 16.0 GiB VM-state, 314.4 MiB/s
2026-07-14 10:13:10 migration active, transferred 12.6 GiB of 16.0 GiB VM-state, 320.5 MiB/s
2026-07-14 10:13:11 migration active, transferred 12.7 GiB of 16.0 GiB VM-state, 563.8 MiB/s
2026-07-14 10:13:12 migration active, transferred 13.2 GiB of 16.0 GiB VM-state, 565.0 MiB/s
2026-07-14 10:13:13 migration active, transferred 13.6 GiB of 16.0 GiB VM-state, 615.5 MiB/s
2026-07-14 10:13:15 average migration speed: 356.6 MiB/s - downtime 87 ms
2026-07-14 10:13:15 migration completed, transferred 14.2 GiB VM-state
2026-07-14 10:13:15 migration status: completed
2026-07-14 10:13:18 migration finished successfully (duration 00:00:55)
TASK OK

Guest 2:

Code: 
2026-07-14 10:13:20 conntrack state migration not supported or disabled, active connections might get dropped
2026-07-14 10:13:20 use dedicated network address for sending migration traffic (x.x.x.x)
2026-07-14 10:13:21 starting migration of VM 456 to node 'pve04' (x.x.x.x)
2026-07-14 10:13:21 starting VM 456 on remote node 'pve04'
2026-07-14 10:13:24 start remote tunnel
2026-07-14 10:13:25 ssh tunnel ver 1
2026-07-14 10:13:25 starting online/live migration on unix:/run/qemu-server/456.migrate
2026-07-14 10:13:25 set migration capabilities
2026-07-14 10:13:25 migration speed limit: 500.0 MiB/s
2026-07-14 10:13:25 migration downtime limit: 100 ms
2026-07-14 10:13:25 migration cachesize: 2.0 GiB
2026-07-14 10:13:25 set migration parameters
2026-07-14 10:13:25 start migrate command to unix:/run/qemu-server/456.migrate
2026-07-14 10:13:26 migration active, transferred 302.8 MiB of 16.0 GiB VM-state, 523.5 MiB/s
2026-07-14 10:13:27 migration active, transferred 778.2 MiB of 16.0 GiB VM-state, 511.6 MiB/s
2026-07-14 10:13:28 migration active, transferred 1.1 GiB of 16.0 GiB VM-state, 512.5 MiB/s
2026-07-14 10:13:29 migration active, transferred 1.3 GiB of 16.0 GiB VM-state, 268.9 MiB/s
2026-07-14 10:13:30 migration active, transferred 1.4 GiB of 16.0 GiB VM-state, 109.6 MiB/s
2026-07-14 10:13:31 migration active, transferred 1.6 GiB of 16.0 GiB VM-state, 245.3 MiB/s
2026-07-14 10:13:32 migration active, transferred 1.9 GiB of 16.0 GiB VM-state, 379.0 MiB/s
2026-07-14 10:13:33 migration active, transferred 2.4 GiB of 16.0 GiB VM-state, 748.3 MiB/s
2026-07-14 10:13:34 migration active, transferred 2.6 GiB of 16.0 GiB VM-state, 435.3 MiB/s
2026-07-14 10:13:35 migration active, transferred 2.9 GiB of 16.0 GiB VM-state, 238.2 MiB/s
2026-07-14 10:13:36 migration active, transferred 3.2 GiB of 16.0 GiB VM-state, 313.2 MiB/s
2026-07-14 10:13:37 migration active, transferred 3.3 GiB of 16.0 GiB VM-state, 113.0 MiB/s
2026-07-14 10:13:38 migration active, transferred 3.5 GiB of 16.0 GiB VM-state, 88.3 MiB/s
2026-07-14 10:13:39 migration active, transferred 3.6 GiB of 16.0 GiB VM-state, 245.3 MiB/s
2026-07-14 10:13:40 migration active, transferred 3.8 GiB of 16.0 GiB VM-state, 240.5 MiB/s
2026-07-14 10:13:41 migration active, transferred 3.9 GiB of 16.0 GiB VM-state, 475.8 MiB/s
2026-07-14 10:13:42 migration active, transferred 4.4 GiB of 16.0 GiB VM-state, 511.5 MiB/s
2026-07-14 10:13:43 migration active, transferred 4.8 GiB of 16.0 GiB VM-state, 514.7 MiB/s
2026-07-14 10:13:44 migration active, transferred 5.0 GiB of 16.0 GiB VM-state, 240.5 MiB/s
2026-07-14 10:13:45 migration active, transferred 5.2 GiB of 16.0 GiB VM-state, 242.9 MiB/s
2026-07-14 10:13:46 migration active, transferred 5.5 GiB of 16.0 GiB VM-state, 406.9 MiB/s
2026-07-14 10:13:47 migration active, transferred 5.8 GiB of 16.0 GiB VM-state, 238.2 MiB/s
2026-07-14 10:13:48 migration active, transferred 6.0 GiB of 16.0 GiB VM-state, 116.2 MiB/s
2026-07-14 10:13:49 migration active, transferred 6.1 GiB of 16.0 GiB VM-state, 274.1 MiB/s
2026-07-14 10:13:50 migration active, transferred 6.3 GiB of 16.0 GiB VM-state, 243.0 MiB/s
2026-07-14 10:13:51 migration active, transferred 6.5 GiB of 16.0 GiB VM-state, 437.6 MiB/s
2026-07-14 10:13:52 migration active, transferred 6.8 GiB of 16.0 GiB VM-state, 504.1 MiB/s
2026-07-14 10:13:53 migration active, transferred 7.2 GiB of 16.0 GiB VM-state, 264.5 MiB/s
2026-07-14 10:13:54 migration active, transferred 7.4 GiB of 16.0 GiB VM-state, 242.9 MiB/s
2026-07-14 10:13:55 migration active, transferred 7.5 GiB of 16.0 GiB VM-state, 108.3 MiB/s
2026-07-14 10:13:56 migration active, transferred 7.7 GiB of 16.0 GiB VM-state, 237.9 MiB/s
2026-07-14 10:13:57 migration active, transferred 8.0 GiB of 16.0 GiB VM-state, 357.2 MiB/s
2026-07-14 10:13:58 migration active, transferred 8.4 GiB of 16.0 GiB VM-state, 518.6 MiB/s
2026-07-14 10:13:59 migration active, transferred 8.5 GiB of 16.0 GiB VM-state, 97.6 MiB/s
2026-07-14 10:14:00 migration active, transferred 8.7 GiB of 16.0 GiB VM-state, 109.6 MiB/s
2026-07-14 10:14:01 migration active, transferred 9.1 GiB of 16.0 GiB VM-state, 404.8 MiB/s
2026-07-14 10:14:02 migration active, transferred 9.5 GiB of 16.0 GiB VM-state, 519.3 MiB/s
2026-07-14 10:14:03 migration active, transferred 9.9 GiB of 16.0 GiB VM-state, 471.2 MiB/s
2026-07-14 10:14:04 migration active, transferred 10.2 GiB of 16.0 GiB VM-state, 397.3 MiB/s
2026-07-14 10:14:05 migration active, transferred 10.6 GiB of 16.0 GiB VM-state, 539.8 MiB/s
2026-07-14 10:14:06 migration active, transferred 11.0 GiB of 16.0 GiB VM-state, 566.7 MiB/s
2026-07-14 10:14:07 migration active, transferred 11.4 GiB of 16.0 GiB VM-state, 611.8 MiB/s
2026-07-14 10:14:08 migration active, transferred 11.8 GiB of 16.0 GiB VM-state, 389.5 MiB/s
2026-07-14 10:14:09 migration active, transferred 12.2 GiB of 16.0 GiB VM-state, 616.9 MiB/s
2026-07-14 10:14:10 migration active, transferred 12.6 GiB of 16.0 GiB VM-state, 271.3 MiB/s
2026-07-14 10:14:11 migration active, transferred 13.0 GiB of 16.0 GiB VM-state, 451.0 MiB/s
2026-07-14 10:14:12 migration active, transferred 13.4 GiB of 16.0 GiB VM-state, 569.2 MiB/s
2026-07-14 10:14:13 migration active, transferred 13.8 GiB of 16.0 GiB VM-state, 442.2 MiB/s
2026-07-14 10:14:14 migration active, transferred 14.2 GiB of 16.0 GiB VM-state, 326.7 MiB/s
2026-07-14 10:14:15 migration active, transferred 14.5 GiB of 16.0 GiB VM-state, 382.3 MiB/s
2026-07-14 10:14:17 average migration speed: 315.5 MiB/s - downtime 72 ms
2026-07-14 10:14:17 migration completed, transferred 14.9 GiB VM-state
2026-07-14 10:14:17 migration status: completed
2026-07-14 10:14:20 migration finished successfully (duration 00:01:00)
TASK OK
 
Hi,
is there anything in the system logs/journal of the target node around the time of the issue? What about the in-guest logs?
 
Bulk migration itself shouldn't shut down VMs. If the migration tasks completed successfully but the guests remained powered off, that's not expected behavior. I'd recommend checking the migration logs (journalctl, task logs, and /var/log/pve/tasks) for any errors, and also verify whether live migration requirements (shared storage, network connectivity, CPU compatibility, etc.) were fully met. Knowing your Proxmox VE version and migration settings would also help troubleshoot.
 
You must log in or register to reply here.
Top Bottom
Back