[SOLVED] Force DHCP renew after live migration to pick up new gateway

P

phip

New Member
Aug 13, 2024
12
1
3
Hi all,

we're trying to improve the robustness of our IT deployment by moving one node to an offsite location. The nodes will stay in the same subnet using a site-2-site VPN handled by the external firewall. A test setup of this works already, but one potential issue has surfaced: When a VM is migrated from site A to site B, it will keep its IP configuration and have the default gateway still pointing at the firewall of site A. If site A then goes down, this will cause trouble for the VM, even though it would have perfectly fine hardware to run on.
The idea is now to have the VMs get their IP configuration via DHCP from the firewall. The IP address would be configured the same in each case, so this wouldn't change, but the default gateway would then be set to the proper firewall. This, however, requires that a DHCP renew is triggered when a VM has been migrated. This could be done manually, but an automated way would be beneficial. Is there a mechanism available in PVE to do this? Like a hook or something to invoke a command on the guest after the migration?

A workaround would be just set the DHCP lease time to something like 5min, but that feels like the second best idea at most...

Thanks and best regards,
Philipp
 
Did some more searching and found that this should be possible as a combination of a hookscript on the host and qm guest exec. Maybe I could then also just resort to directly update the default gateway to the proper value instead of relying on a separate DHCP request.
 
For documentation purposes, here's a hookscript for what I've managed to come up with so far:
Perl: 
#!/usr/bin/perl                                                                                                                                                                       
                                                                                                                                                                                      
# Sets the default gateway of a VM after starting it, also after         
# a live migration. The gateway is set to the same value as on           
# the host.                                                                               
                                                                                          
# Attach to a VM using                                                                     
# qm set <vmid> --hookscript local:snippets/set-default-gateway-hook.pl 
                                                                                          
use strict;                                                                               
use warnings;                                                                             
                                                                                          
# Change /dev/null to a file to get outputs (on the host) for debugging.
open(FH, '>>', '/dev/null') or die $!;                                                     
                                                                                          
print FH "GUEST HOOK: " . join(' ', @ARGV). "\n";                       
                                                                                          
# First argument is the vmid                                                               
                                                                                          
my $vmid = shift;                                                                         
                                                                                          
# Second argument is the phase                                                             
                                                                                          
my $phase = shift;                                                                         
                                                                                          
if ($phase eq 'pre-start') {                                                               
                                                                                          
    # First phase 'pre-start' will be executed before the guest         
    # is started. Exiting with a code != 0 will abort the start         
                                                                                          
    #print "$vmid is starting, doing preparations.\n";                   
                                            
    # print "preparations failed, aborting."                                                                                                                                           
    # exit(1);                   
                                                                                          
} elsif ($phase eq 'post-start') {                                                         
                                            
    # Second phase 'post-start' will be executed after the guest
    # successfully started.                                                               
                                            
    #print "$vmid started successfully.\n";                                               
                                                                                          
    # Get default gateway from host.
    my $gateway = `ip -4 -c=never route show default`;
    if (!defined $gateway) {                                                               
        die "Failed to get default gateway of host";                                       
    }                                                                                     
    $gateway =~ s/default via ([0-9.]+) .*/$1/;
    print FH "Default gateway of host: $gateway";                                         
    my $guest_command = "ip route replace default via $gateway";
    my $host_command = "qm guest exec $vmid $guest_command";
    my $limit = 60;
    while ($limit && system($host_command)) {                                                                                                                                         
        sleep(1);
        $limit -= 1;
    }           
    if ($limit) {                                                                         
        print FH "Executed $host_command\n";
    } else {
        print FH "Failed to execute $host_command\n";                                     
    }             
                                            
} elsif ($phase eq 'pre-stop') {                                                           
                                            
    # Third phase 'pre-stop' will be executed before stopping the guest                   
    # via the API. Will not be executed if the guest is stopped from                       
    # within e.g., with a 'poweroff'
                                            
    #print "$vmid will be stopped.\n";                                                     
                                            
} elsif ($phase eq 'post-stop') {

    # Last phase 'post-stop' will be executed after the guest stopped.                     
    # This should even be executed in case the guest crashes or stopped
    # unexpectedly.                                                                       
                                                                                          
    #print "$vmid stopped. Doing cleanup.\n";
                                            
} else {                                                                                   
    die "got unknown phase '$phase'\n";     
}                               
                                            
close(FH);                                                                                 
exit(0);

This works fine when the VM is rebooted. But it seems that a live migration behaves differently. During a boot, I see this sequence:
  1. pre-start hook
  2. Right afterwards (milliseconds later) post-start hook with display of host default gateway
  3. VM is booting up, as can be seen in the console
  4. When the login prompt finally appears, the script says "Executed qm guest exec ..." and the gateway is properly set
But with a live migration, this is different:
  1. Migration task runs normally until "starting VM 103 on remote node 'pve-test-2'" and "volume 'local-zfs:vm-103-disk-0' is 'local-zfs:vm-103-disk-0' on the target"
  2. Now the task hangs for 1-2 minutes, subject to the timeout (`$limit = 60`) set in the script
  3. Eventually, the task continue because the loop is cancelled
  4. The VM runs normally on the migration target, but obviously without having the new gateway set
So it looks to me that the migration task waits for the hookscript's post-start sequence to finish before it actually completes the migration and lets the VM CPU run. That's not what I would expect from a phase named "post-start", but that's probably a question of definitions.
I think my options are now to either rebuild the script so that it forks off a separate process that waits for the VM agent to be available in the background, i.e., without blocking the hookscript, or to find an entirely different solution to the root problem...
 
if the node is supposed to (logically) stay in the original cluster, please be aware of the requirements regarding network stability and latency!

other than that, yes, a hook script is the way to go for this at the moment. you might consider taking a look at Proxmox Datacenter Manager and SDN in the future ;)
 
@fabian Thanks for your response. I'm aware of the <5ms latency requirement and will need to test how things behave once they are actually separated. Both sites are connected via 1G/10G fiber and the distance isn't more than about 100km, but it's still going via the public internet, so we'll see.
PDM certainly looks interesting and I'll need to check it out. We actually were into SDN already, trying to get the VXLAN in our test setup working, but failed also at the gateway assignment point. That's why we then resorted to using the firewall's Site2Site VPN.

Anyway, I think I now got a working solution for the gateway updating using a hookscript. It's actually something that can be considered dead simple, as all the tools are there, but I pulled my hair figuring out how to invoke the commands with proper detaching, so the migration isn't blocked. When I got it working the first time, I figured that I also need to update the VM's nameserver and then failed to find a way to do this via `qm guest exec`. So I now use a simple helper script inside the VMs, which takes the new gateway as argument and does the updates as needed. The advantage of the script approach is that certain VMs now could even do other stuff when migrated, should this be necessary.

Here's the (so far working) hookscript, which should be placed on each host/node and then configured for the VMs:
Perl: 
#!/usr/bin/perl

# Sets the default gateway of a VM after starting it, also after
# a live migration. The gateway is set to the same value as on
# the host.

# Attach to a VM using
# qm set <vmid> --hookscript local:snippets/set-default-gateway-hook.pl

# NOTE: The VM needs to have the /root/update-gateway.sh script to apply
# the necessary changes, which could look like this:
# cat /root/update-gateway.sh
# #!/bin/sh
# GW=$1
# sleep 60
# ip route replace default via $GW
# echo "nameserver $GW" > /etc/resolv.conf

# See: https://forum.proxmox.com/threads/force-dhcp-renew-after-live-migration-to-pick-up-new-gateway.177062/post-821319


use strict;
use warnings;

# Change to a file for debugging purposes.
my $LOG_OUTPUT = '/dev/null';

open(FH, '>>', $LOG_OUTPUT) or die $!;

print FH "GUEST HOOK: " . join(' ', @ARGV). "\n";

my $vmid = shift;
my $phase = shift;

if ($phase eq 'post-start') {

    # Get default gateway from host.
    my $gateway = `ip -4 -c=never route show default`;
    if (!defined $gateway) {
        die "Failed to get default gateway of host";
    }
    $gateway =~ s/default via ([0-9.]+) .*/$1/g;
    $gateway =~ s/^\s+|\s+$//g;
    print FH "Default gateway of host: $gateway\n";
    my $guest_command = "/root/update-gateway.sh $gateway";
    my $host_command = "qm guest exec $vmid $guest_command";
    my $host_script = "echo \"Loop starting...\"; i=0; while [ \"\$i\" -lt 60 ]; do if $host_command; then break; fi; i=\$((i+1)); sleep 1; done; echo \"Loop exited.\"";
    my $system_call = "/usr/bin/setsid -f /bin/sh -c '$host_script' 1>>$LOG_OUTPUT 2>>$LOG_OUTPUT";
    print "Calling: $system_call\n";
    system $system_call;

}

close(FH);
exit(0);

And this is the helper script that should go inside each VM to be migrated, for example as `/root/update-gateway.sh`:
Bash: 
#!/bin/sh

# Updates common system settings to the gateway given as first argument.

# See: https://forum.proxmox.com/threads/force-dhcp-renew-after-live-migration-to-pick-up-new-gateway.177062/post-821319

GW=$1

# Give the migration time to complete fully before messing with the
# network settings to improve the reliability of established TCP connections.
sleep 60

ip route replace default via $GW
echo "nameserver $GW" > /etc/resolv.conf

I'll see whether all this really works as it should during further testing. I'll mark the thread as solved for now, even though the solution doesn't exactly match the title. However, it would be simple to instead of updating routes just trigger a DHCP renewal in the helper script.

In any case, thanks for the good work Proxmox! It's amazing what you've built!
 
Last edited:
  • Like
Reactions: fabian
Hm, looks like my approach had a rough edge: While the migrations itself and the updates of default gateway and nameserver work fine, established TCP connections would sometimes become stuck. This manifests for example in iperf3 runs, where traffic is fine until the migration and afterwards it would just say "0.00 bits/sec" for any subsequent output line. It may also cause open SSH connections to hang. However, I discovered that the issue is not perfectly reproducible, sometimes it also worked. This made me think that it could be an interference of the gateway update with Proxmox's conntrack migration (even though the migration process says "migrated 0 conntrack state entries"). My workaround/fix is to insert a delay/sleep of, say, 60s before actually updating the default gateway. With this change, I've still faced the issue once more (of maybe 10 test migrations), so it appears to have the situation improved at least. Not very pretty, but should suffice for now.

I'll add the line to the script above for future reference.
 
You must log in or register to reply here.
Top Bottom