Migrate VM hangs

J

Jim J

New Member
Oct 26, 2016
17
1
1
54
Hi Gang,

I've been seeing a problem throughout the entire Prox 4.X series. Namely I can not migrate VMs or Containers from one Node in the cluster to another. I'm using Local LVM storage and the job always hangs in the same place. NOTE I am not attempting a LIVE migration. I shutdown the VM and then try to migrate. It fails for both VMs and Containers. The cluster looks fine in all other respects.

Here is the result every time...
Code: 
Mar 31 17:09:04 starting migration of CT 115 to node 'prox6' (10.3.0.10)
Mar 31 17:09:04 found local volume 'local-lvm:vm-115-disk-1' (in current VM config)
  Logical volume "vm-115-disk-1" created.
successfully created 'local-lvm:vm-115-disk-1'
......HHHHHAAAAAANNNNNGGGGGG

pveversion -v

Code: 
proxmox-ve: 4.4-79 (running kernel: 4.4.35-1-pve)
pve-manager: 4.4-12 (running version: 4.4-12/e71b7a74)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-108
pve-firmware: 1.1-10
libpve-common-perl: 4.0-91
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-73
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-3
pve-qemu-kvm: 2.7.1-1
pve-container: 1.0-93
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-1
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve14~bpo80
 
please upgrade to the current version. if the problem persists, please post the VM and storage configuration files.
 
This is still an issue. While attempting to mgirate I see SSHD and DD fire up, but waiting forever shows now change even when attempting to migrate small Containers. On the destination node, I see the new LV created for the container I am trying to move.

/etc/pve/lxc/110.conf
Code: 
arch: amd64
cores: 1
hostname: support1
lock: migrate
memory: 1024
nameserver: 10.3.0.1
net0: name=eth0,bridge=vmbr0,gw=10.3.0.1,hwaddr=5E:CB:52:95:43:CE,ip=10.3.0.21/24,type=veth
ostype: ubuntu
rootfs: local-lvm:vm-110-disk-1,size=20G
searchdomain: foobar.com
swap: 2048

Code: 
  --- Logical volume ---
  LV Path                /dev/pve/vm-110-disk-1
  LV Name                vm-110-disk-1
  VG Name                pve
  LV UUID                7C9BeX-5x8A-EXuV-8dFp-dyUt-K3Ej-CNYugi
  LV Write Access        read/write
  LV Creation host, time prox6, 2017-06-15 20:40:11 -0400
  LV Pool name           data
  LV Status              available
  # open                 1
  LV Size                20.00 GiB
  Mapped size            8.92%
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:12

pveversion -v
Code: 
proxmox-ve: 4.4-88 (running kernel: 4.4.62-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.62-1-pve: 4.4.62-88
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-50
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-95
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-100
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80


/etc/pve/storage.cfg
Code: 
dir: local
        path /var/lib/vz
        content backup,iso,vztmpl

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

nfs: freenas
        export /mnt/storage1/florida
        path /mnt/pve/freenas
        server 10.1.0.13
        content vztmpl,iso,backup
        maxfiles 2
        options vers=3
 
if it still looks like in your original post, that would indicate it is hanging while transferring the data from source node to target node. are you sure it is hanging and not just copying very slowly? you can send SIGUSR1 to a running dd process to get it to print transfer statistics on stderr:

first, found out the PID of your dd process:
Code: 
pidof dd

if this only prints one number, you should be good to go. otherwise, check the following to find out which is the dd you are looking for:
Code: 
ps aux | grep dd

if you know the PID, run the following (replace PID with the actual number):
Code: 
kill -USR1 PID

now, your migrate output should contain new lines saying how much it has copied so far and at what average rate. if you wait a few minutes and repeat, you should see whether it is still progressing or not.
 
Sure enough, the VM migrates, but there is no visible indication of progress until it completes. In the past we would see a % count climb, but the UX doesn't seem to show that anymore.
 
  • Like
Reactions: ryan4yin
Jim J said:
Sure enough, the VM migrates, but there is no visible indication of progress until it completes. In the past we would see a % count climb, but the UX doesn't seem to show that anymore.
Click to expand...
same problem here, the migration is running background, and there is no visible indication of progress until the migration completed.


```
# pveversion -v

proxmox-ve: 7.2-1 (running kernel: 5.15.30-2-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.15: 7.2-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
```

and the full migration logs(it hang on `2022-11-21 23:08:44 Logical volume "vm-103-cloudinit" created.` until the migration completed):

```
2022-11-21 23:08:43 starting migration of VM 103 to node 'pve-s500plus' (192.168.5.174)
2022-11-21 23:08:43 found generated disk 'local-lvm:vm-103-cloudinit' (in current VM config)
2022-11-21 23:08:43 found local disk 'local-lvm:vm-103-disk-0' (via storage)
2022-11-21 23:08:43 found local disk 'local-lvm:vm-103-disk-1' (in current VM config)
2022-11-21 23:08:43 copying local disk images
2022-11-21 23:08:44 Logical volume "vm-103-cloudinit" created.
2022-11-21 23:08:44 64+0 records in
2022-11-21 23:08:44 64+0 records out
2022-11-21 23:08:44 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.386207 s, 10.9 MB/s
2022-11-21 23:08:44 successfully imported 'local-lvm:vm-103-cloudinit'
2022-11-21 23:08:44 33+87 records in
2022-11-21 23:08:44 33+87 records out
2022-11-21 23:08:44 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0199016 s, 211 MB/s
2022-11-21 23:08:44 volume 'local-lvm:vm-103-cloudinit' is 'local-lvm:vm-103-cloudinit' on the target
2022-11-21 23:08:45 volume pve/vm-103-disk-0 already exists - importing with a different name
2022-11-21 23:08:45 Logical volume "vm-103-disk-2" created.
2022-11-21 23:10:55 573440+0 records in
2022-11-21 23:10:55 573440+0 records out
2022-11-21 23:10:55 37580963840 bytes (38 GB, 35 GiB) copied, 129.577 s, 290 MB/s
2022-11-21 23:10:55 successfully imported 'local-lvm:vm-103-disk-2'
2022-11-21 23:10:55 152+2292266 records in
2022-11-21 23:10:55 152+2292266 records out
2022-11-21 23:10:55 37580963840 bytes (38 GB, 35 GiB) copied, 129.225 s, 291 MB/s
2022-11-21 23:10:55 volume 'local-lvm:vm-103-disk-0' is 'local-lvm:vm-103-disk-2' on the target
2022-11-21 23:10:56 volume pve/vm-103-disk-1 already exists - importing with a different name
2022-11-21 23:10:56 Logical volume "vm-103-disk-3" created.
2022-11-21 23:14:01 819200+0 records in
2022-11-21 23:14:01 819200+0 records out
2022-11-21 23:14:01 53687091200 bytes (54 GB, 50 GiB) copied, 184.946 s, 290 MB/s
2022-11-21 23:14:02 successfully imported 'local-lvm:vm-103-disk-3'
2022-11-21 23:14:02 298+3275262 records in
2022-11-21 23:14:02 298+3275262 records out
2022-11-21 23:14:02 53687091200 bytes (54 GB, 50 GiB) copied, 186.223 s, 288 MB/s
2022-11-21 23:14:02 volume 'local-lvm:vm-103-disk-1' is 'local-lvm:vm-103-disk-3' on the target
Logical volume "vm-103-cloudinit" successfully removed
Logical volume "vm-103-disk-0" successfully removed
Logical volume "vm-103-disk-1" successfully removed
2022-11-21 23:14:04 migration finished successfully (duration 00:05:21)
TASK OK
```
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back