VM unsuspend after migrating

[lDemoNl]

Hi guys! I've found one interesting thing in my cluster (5 nodes w ceph). I suspended VM and would like to migrate to another host in cluster, but after migrating VM is enabled (not suspended!)
Maybe answer is in migrate settings? There I have Mode: Enabled? If it is, how could I migrate VM in current state?

PVE Manager Version pve-manager/6.4-13/9f411e79

 

[fiona]

Hi,
it should stay in the current state. Please post the full migration log. is the VM HA-managed by any chance?

 

[lDemoNl]

Hi,
it should stay in the current state. Please post the full migration log. is the VM HA-managed by any chance?

Yes, VM's managed by HA (state set "ignored")
Migrate logs:

Spoiler: LOGS

2022-03-17 17:08:22 starting migration of VM 115 to node 'pve01' (10.10.0.15)
2022-03-17 17:08:22 starting VM 115 on remote node 'pve01'
2022-03-17 17:08:24 start remote tunnel
2022-03-17 17:08:25 ssh tunnel ver 1
2022-03-17 17:08:25 starting online/live migration on unix:/run/qemu-server/115.migrate
2022-03-17 17:08:25 set migration capabilities
2022-03-17 17:08:25 migration downtime limit: 100 ms
2022-03-17 17:08:25 migration cachesize: 64.0 MiB
2022-03-17 17:08:25 set migration parameters
2022-03-17 17:08:25 start migrate command to unix:/run/qemu-server/115.migrate
2022-03-17 17:08:26 migration active, transferred 247.4 MiB of 529.1 MiB VM-state, 516.3 MiB/s
2022-03-17 17:08:27 average migration speed: 264.5 MiB/s - downtime 68 ms
2022-03-17 17:08:27 migration status: completed
2022-03-17 17:08:30 migration finished successfully (duration 00:00:08)
TASK OK

After migrating I see next task to start VM.

 

[fiona]

Thanks for the report! I'm able to reproduce the issue here and will work out a patch. Unfortunately, I don't see a good workaround in the meantime and I'm not sure the patch will be back-ported to Proxmox VE 6.x (which will reach end-of-life in a few months - see here for how to upgrade to 7.x), because it's nothing critical.

EDIT: patch is available here, but not yet reviewed.

 

[lDemoNl]

Thanks for the report! I'm able to reproduce the issue here and will work out a patch. Unfortunately, I don't see a good workaround in the meantime and I'm not sure the patch will be back-ported to Proxmox VE 6.x (which will reach end-of-life in a few months - see here for how to upgrade to 7.x), because it's nothing critical.

EDIT: patch is available here, but not yet reviewed.

Thank you Fabian. How could I install and test this patch? I have never patched PVE

 

[fiona]

I'd recommend against doing it on a production system if you are not experienced with such things!

The patch for 6.4 would be slightly different

cat migrate-keep-paused.patch

diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm
index 104e62ce..891edfb2 100644
--- a/PVE/QemuMigrate.pm
+++ b/PVE/QemuMigrate.pm
@@ -158,6 +158,8 @@ sub prepare {
                $self->{forcecpu} = PVE::QemuServer::CPUConfig::get_cpu_from_running_vm($pid);
            }
        }
+
+       $self->{vm_was_paused} = 1 if PVE::QemuServer::vm_is_paused($vmid);
     }
 
     my $loc_res = PVE::QemuServer::check_local_resources($conf, 1);
@@ -1274,31 +1276,37 @@ sub phase3_cleanup {
            }
        }
 
-       # config moved and nbd server stopped - now we can resume vm on target
-       if ($tunnel && $tunnel->{version} && $tunnel->{version} >= 1) {
-           eval {
-               $self->write_tunnel($tunnel, 30, "resume $vmid");
-           };
-           if (my $err = $@) {
-               $self->log('err', $err);
-               $self->{errors} = 1;
-           }
-       } else {
-           my $cmd = [@{$self->{rem_ssh}}, 'qm', 'resume', $vmid, '--skiplock', '--nocheck'];
-           my $logf = sub {
-               my $line = shift;
-               $self->log('err', $line);
-           };
-           eval { PVE::Tools::run_command($cmd, outfunc => sub {}, errfunc => $logf); };
-           if (my $err = $@) {
-               $self->log('err', $err);
-               $self->{errors} = 1;
+       if (!$self->{vm_was_paused}) {
+           # config moved and nbd server stopped - now we can resume vm on target
+           if ($tunnel && $tunnel->{version} && $tunnel->{version} >= 1) {
+               eval {
+                   $self->write_tunnel($tunnel, 30, "resume $vmid");
+               };
+               if (my $err = $@) {
+                   $self->log('err', $err);
+                   $self->{errors} = 1;
+               }
+           } else {
+               my $cmd = [@{$self->{rem_ssh}}, 'qm', 'resume', $vmid, '--skiplock', '--nocheck'];
+               my $logf = sub {
+                   my $line = shift;
+                   $self->log('err', $line);
+               };
+               eval { PVE::Tools::run_command($cmd, outfunc => sub {}, errfunc => $logf); };
+               if (my $err = $@) {
+                   $self->log('err', $err);
+                   $self->{errors} = 1;
+               }
            }
-       }
 
-       if ($self->{storage_migration} && PVE::QemuServer::parse_guest_agent($conf)->{fstrim_cloned_disks} && $self->{running}) {
-           my $cmd = [@{$self->{rem_ssh}}, 'qm', 'guest', 'cmd', $vmid, 'fstrim'];
-           eval{ PVE::Tools::run_command($cmd, outfunc => sub {}, errfunc => sub {}) };
+           if (
+               $self->{storage_migration}
+               && PVE::QemuServer::parse_guest_agent($conf)->{fstrim_cloned_disks}
+               && $self->{running}
+           ) {
+               my $cmd = [@{$self->{rem_ssh}}, 'qm', 'guest', 'cmd', $vmid, 'fstrim'];
+               eval{ PVE::Tools::run_command($cmd, outfunc => sub {}, errfunc => sub {}) };
+           }
        }
     }

You can apply it with

patch /usr/share/perl5/PVE/QemuMigrate.pm migrate-keep-paused.patch
systemctl reload-or-restart pveproxy.service pvedaemon.service

(on the source node of the migration).

To get back to the unpatched version, use

apt install --reinstall qemu-server