falling over to a zfs replica

R

randyqx

Member
May 21, 2024
27
3
8
US Left Coast
maybe i do not understand how to fall a VM from one node to another
  • VM foo was running on vm1
  • i replicate it to vm0 and let it stabilize
  • i fail it over vm1.sea.rg.net:/etc/pve/nodes/vm1/qemu-server# mv 104.conf ../../vm0/qemu-server/
  • it is fine on vm0, but ps on vm1 shows it still running on vm1, i.e. there is a kvm process in its name
so maybe i have the wrong recipe for failover?

pve-manager/9.2.11/f6997e698c7933ea (running kernel: 7.0.14-14-pve)
 
Last edited:
Moving the config file doesn't stop the VM; nothing watches /etc/pve and kills a running kvm process when its config moves away. All you've changed is which node PVE thinks owns the guest, so the process on vm1 carries on running and writing to vm1's copy of the dataset.

To get back to a clean state: move 104.conf back to vm1, run qm stop 104 there, confirm the process is gone, then move with QM migrate:
qm migrate 104 vm0 --online

For actual failover (node dies unexpectedly) that's what HA is for. ha-manager works with replication and does exactly what you're trying to do by hand
- fence the dead node, then bring the guest up on another one from the replicated dataset. Manually moving config files is the thing HA exists to replace [0].

HA needs reliable quorum, so if vm0 and vm1 are your only two nodes you could use a QDevice as a third vote [1] before turning it on.

[0]: https://pve.proxmox.com/wiki/High_Availability#_how_it_works
[1]: https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support
 
  • Like
Reactions: fba, Kingneutron and UdoB
after qm migrate, the source image remains on the source node, if i free it, it comes back!
Code: 
# qm list
  VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID      
   100 archive.psg.com      running    2048             128.00 2716    
   109 nagasaki.bogus.com   running    6144             250.00 3384    
   110 bvi.bgp.nu           running    8192             500.00 324940

# pvesm list images
Volid                Format  Type               Size VMID
images:vm-100-disk-0 raw     images     137438953472 100
images:vm-104-disk-0 raw     images     107374182400 104
images:vm-106-disk-0 raw     images      68719476736 106
images:vm-106-disk-1 raw     images      34359738368 106
images:vm-109-disk-0 raw     images     268435456000 109
images:vm-110-disk-0 raw     images     536870912000 110

# pvesm free images:vm-104-disk-0
Removed volume 'images:vm-104-disk-0'
# pvesm free images:vm-106-disk-0
Removed volume 'images:vm-106-disk-0'
# pvesm free images:vm-106-disk-1
Removed volume 'images:vm-106-disk-1'

# pvesm list images
Volid                Format  Type              Size VMID
images:vm-100-disk-0 raw     images    137438953472 100
images:vm-109-disk-0 raw     images    268435456000 109
images:vm-110-disk-0 raw     images    536870912000 110

< wait a few minutes>

# pvesm list images
Volid                Format  Type              Size VMID
images:vm-100-disk-0 raw     images    137438953472 100
images:vm-106-disk-0 raw     images     68719476736 106
images:vm-106-disk-1 raw     images     34359738368 106
images:vm-109-disk-0 raw     images    268435456000 109
images:vm-110-disk-0 raw     images    536870912000 110
there are no running kvm processes for 106. the migration code i used was
Code: 
qm migrate   $1 $2 \
   --migration_network 10.0.0.0/24 \
   --migration_type insecure \
   --online \
   --with-local-disks
reboot, re-delete, wait, and they come back.
no kvm processes running
 
Last edited:
randyqx said:
after qm migrate, the source image remains on the source node, if i free it, it comes back!
Code: 
# qm list
  VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID    
   100 archive.psg.com      running    2048             128.00 2716  
   109 nagasaki.bogus.com   running    6144             250.00 3384  
   110 bvi.bgp.nu           running    8192             500.00 324940

# pvesm list images
Volid                Format  Type               Size VMID
images:vm-100-disk-0 raw     images     137438953472 100
images:vm-104-disk-0 raw     images     107374182400 104
images:vm-106-disk-0 raw     images      68719476736 106
images:vm-106-disk-1 raw     images      34359738368 106
images:vm-109-disk-0 raw     images     268435456000 109
images:vm-110-disk-0 raw     images     536870912000 110

# pvesm free images:vm-104-disk-0
Removed volume 'images:vm-104-disk-0'
# pvesm free images:vm-106-disk-0
Removed volume 'images:vm-106-disk-0'
# pvesm free images:vm-106-disk-1
Removed volume 'images:vm-106-disk-1'

# pvesm list images
Volid                Format  Type              Size VMID
images:vm-100-disk-0 raw     images    137438953472 100
images:vm-109-disk-0 raw     images    268435456000 109
images:vm-110-disk-0 raw     images    536870912000 110

< wait a few minutes>

# pvesm list images
Volid                Format  Type              Size VMID
images:vm-100-disk-0 raw     images    137438953472 100
images:vm-106-disk-0 raw     images     68719476736 106
images:vm-106-disk-1 raw     images     34359738368 106
images:vm-109-disk-0 raw     images    268435456000 109
images:vm-110-disk-0 raw     images    536870912000 110
there are no running kvm processes for 106. the migration code i used was
Code: 
qm migrate   $1 $2 \
   --migration_network 10.0.0.0/24 \
   --migration_type insecure \
   --online \
   --with-local-disks
reboot, re-delete, wait, and they come back.
no kvm processes running
Click to expand...
Thats what it should do if the vm is configured for HA (HA replicates the vm disks from PVE node vm0 to PVE node vm1 with ZFS replicas) if you migrate a vm to different PVE node the replication is automaticaly reversed and it replicates from PVE node vm1 to PVE node vm0

If you want to move a vm from one to another host and freeing up the local disk space you have to remove the vm from HA and do the migration - then the vm disks will get mirrored/replicated and cleared from the source PVE node storage

rg
Christian
 
randyqx said:
after qm migrate, the source image remains on the source node, if i free it, it comes back!
Click to expand...
That's the idea of replication, isn't it? If you migrate the vm, the replication direction is turned around, meaning the node owning the vm replicates the disks to the other one.
before migration: node1 -> node2
after migration: node2 -> node1
and so on.
You have to remove the replication job first. Afterwards delete the disk from storage of the node, where the vm isn't running.
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom
Back