Live migration

[Alessandro 123]

Hi to all
I'm testing proxmox for the very first time.
I've created a cluster with Gluster as shared storage.

One VM is running fine.

Now I'm trying to live migrate this VM to the other node, all seem to run fine, live migration is running and i'm still able to use the VM during the whole process.

But this process is taking a very, very, long time, at least 5-10 minutes. The VM is very simple: 2GB RAM and 50GB disk.

In the migration window I can see the following:

Sep 27 11:48:40 migration xbzrle cachesize: 134217728 transferred 0 pages 0 cachemiss 0 overflow 0
Sep 27 11:48:42 migration status: active (transferred 707193155, remaining 949309440), total 2156732416)
Sep 27 11:48:42 migration xbzrle cachesize: 134217728 transferred 0 pages 0 cachemiss 0 overflow 0
Sep 27 11:48:44 migration status: active (transferred 726022403, remaining 930516992), total 2156732416)
Sep 27 11:48:44 migration xbzrle cachesize: 134217728 transferred 0 pages 0 cachemiss 0 overflow 0
Sep 27 11:48:46 migration status: active (transferred 746431699, remaining 910147584), total 2156732416)
Sep 27 11:48:46 migration xbzrle cachesize: 134217728 transferred 0 pages 0 cachemiss 0 overflow 0
Sep 27 11:48:48 migration status: active (transferred 770001091, remaining 886624256), total 2156732416)
Sep 27 11:48:48 migration xbzrle cachesize: 134217728 transferred 0 pages 0 cachemiss 0 overflow 0

Is this the RAM live migration? Any chance to get this faster ? Is this normal? Usually I have VM with 16GB of ram, if migrating a single 2GB VM requires 10 minutes, a standard VM 8 times bigger, would require 8*10 minutes..................

 

[Alessandro 123]

This is a stripped down migration log.
As you can see, the amount of ram to be transfered is increasing, I think because on the VM there were some tasks running.
When I've stopped all the running tasks making I/O, the dirty RAM stopped to increase.

If this is normal, it would be impossible to live migrate a running VM with I/O, as the dirty ram will always increase

http://pastebin.com/raw/DRn1ET21

 

[fabian]

This is a stripped down migration log.
As you can see, the amount of ram to be transfered is increasing, I think because on the VM there were some tasks running.
When I've stopped all the running tasks making I/O, the dirty RAM stopped to increase.

If this is normal, it would be impossible to live migrate a running VM with I/O, as the dirty ram will always increase

http://pastebin.com/raw/DRn1ET21

it's only impossible if your network is slower than your RAM dirtying. there are some tricks like slowly slowing down the VM's CPU and hoping that this will also decrease RAM usage to a point where live migration is possible, but that only gives you a bit of an advantage and is very work load depending..

 

[Alessandro 123]

I though the same.
This is a test server on a test network (100mbit) with Gluster (sharing the same network, thus far from being optimized)

Based on your experience, which is an average ram dirtying speed ? Production cluster will have a dedicated gigabit network for the storage.
Is the ram sync made thorugh the storage network ? In example, if I have 2 networks: one for public access and 1 for storage, the ram sync is made through the public or through the storage? I think is made through the cluster network (the one used when adding nodes to proxmox cluster). If this is the case, a suggested config would be to use at least 3 networks: 1 for cluster, 1 for public, 1 for storage.