Speed up the VM (storage) migration

Hi

Pasting the very same here since this also applies to v.2.

-----
Can anyone tell me how to change the default cipher blowfish to arcfour when doing migrations of VM's? By replacing the cipher we would be able to migrate servers alot faster.

I have found that changing the setting in /usr/bin/qmigrate from blowfish to arcfour drastically speed up the migration of live VM's on shared storager. But for some reason when doing a offline migration of a VM that is stored locally on one host to another host (storage migration) this setting is override by something else cause then it still uses blowfish and not arcfour and the transfer is so much slower that what it has to be. If we could change this setting I'm quite sure that we can do the storage migration in at least half the time maybe even one third of the time (assuming 1Gbit network ofcourse).
-----

Still when migrating in version 2 the ssh connection uses cipher blowfish. Can we change this to arcfour instead which would speed things up a bit. Or even better, somehow utilize the rsync deamon and do native rsync without ssh between cluster nodes. That would be optimal for storage migrations.

task UPIDm01:00001182:00041E95:4EC6E34D:qmigrate:100:root@pam:
root 4502 31.9 0.0 11352 1280 ? R Nov18 1:51 /usr/bin/rsync --progress --sparse --whole-file /var/lib/vz/images/100/vm-100-disk-1.raw root@192.168.10.22:/var/lib/vz/images/100/vm-100-disk-1.raw
root 4503 95.8 0.0 38656 3000 ? R Nov18 5:33 /usr/bin/ssh -c blowfish -o BatchMode=yes -l root 192.168.10.22 rsync --server -vWSe.Lsf . /var/lib/vz/images/100/vm-100-disk-1.raw

Would also be nice if we could move/migrate a VM storage from local to NFS or ISCSI directly from the gui...

Hi and thanks e100 for your reply.

I will try changing /usr/share/perl5/PVE/Storage.pm and /usr/share/perl5/PVE/QemuMigrate.pm to see if it works on my end as well.

This is exactly what I'm talking about. Arcfour is the fastest of the ciphers even with AMD processors and all the other Intel processors that I have ever encountered.
And also if we could use what I would call "native" rsync (requires the setup of rsync daemon on the servers) then we would not require a ssh tunnel at all thus we can truly utilize the full bandwidth of our networks and the migrations would complete even faster. Of course you should put this on it's own isolated migration network if possible.

Ok, done some testing. Changed from blowfish to arcfour in those files you mentioned but in my case doing a offline migration of a VM that uses local storage the rsync process is still done over ssh session using blowfish.
/usr/bin/ssh -c blowfish -o BatchMode=yes -l root 192.168.10.22 rsync --server -vWSe.Lsf . /var/lib/vz/images/102/vm-102-disk-1.raw

NOTE. I had to reboot the nodes for them to pick up the changes. It does now in fact use arcfour and the migration is now a lot shorter.
Almost half the time...

Heres your answer Dietmar.

Here are some real data
Using ssh cipher blowfish
sent 8590983247 bytes received 31 bytes 19414651.48 bytes/sec
total size is 8589934592 speedup is 1.00
Nov 19 15:05:13 migration finished successfuly (duration 00:07:24)

Using ssh cipher arcfour
sent 8590983247 bytes received 31 bytes 38961375.41 bytes/sec
total size is 8589934592 speedup is 1.00
Nov 19 15:40:33 migration finished successfuly (duration 00:03:43)

But the real deal is that by using ssh to transfer the rsync data costs a lot of cpu power and if we transfer the same file using rsync natively we reduce the cpu load on the hosts (a lot) and reduces the migration time even more. That would require configuring the rsync daemon and start it at boot time. Or do it the dirty way by NFS exporting and mounting the local image storages on the hosts (not preferred).
By doing this I transfer the same file as above in shorter time and offloads the cpu's on the hosts.
My test where I now used rsync over NFS the transfer time went down to 3 minutes and 10 seconds.

My test machines are some old Lenovo laptops connected to a very basic switch. So I guess that on some real modern server hardware the benefits would be even greater.

The obvious thing here is that using arcfour instead of blowfish reduces the time to migrate in half. the bonus is that it will also use a bit less CPU on the hosts.

And the other thing is that if it would be possible to implement a native rsync utilizing rsync daemon directly instead of transferring data over ssh will reduce the migration time even more and offload the CPU on the hosts a lot more which would be of great value.

I believe that this is a simple task to complete and the benefits of it are to great to ignore.

Here are the results and conclusion of my extensive testing and optimizations regarding the VM migration process.

This shows that with some minor changes we can radically enhance the speed of the migration process and reduce the CPU load of the hosts during the migrations.

If we can change the process to use the rsync daemon and and not use SSH as a transport we can reduce the CPU load even more and transfer data (VM images) even faster between the nodes in the cluster. This would also get us a step closer to actually do storage migrations of VM's running on shared storage (NFS) as well and not only for VM's on local storage.

As shown earlier

Using ssh cipher blowfish (the currently default proxmox coniguration) it takes 7min 24sec to migrate a VM between 2 hosts.
sent 8590983247 bytes received 31 bytes 19414651.48 bytes/sec
total size is 8589934592 speedup is 1.00
Nov 19 15:05:13 migration finished successfuly (duration 00:07:24)

And simply changing the ssh cipher to arcfour brings it down to 3min 43sec.
sent 8590983247 bytes received 31 bytes 38961375.41 bytes/sec
total size is 8589934592 speedup is 1.00
Nov 19 15:40:33 migration finished successfuly (duration 00:03:43)

And now with some more tweaking and optimizations we can bring it down to 2min 1sec. This is simply done by telling rsync to use MD4 instead of MD5 with the option --protocol=28. MD4 tends to be 50% faster then MD5 regarding the checksums and crypt.
sent 8590983264 bytes received 36 bytes 71891073.64 bytes/sec
total size is 8589934592 speedup is 1.00
Nov 20 16:00:39 migration finished successfuly (duration 00:02:01)

All three tests above utilize SSH to transfer VM images which puts extra load on the CPU of the hosts.

Now if we instead configure and use rsync daemon on the hosts and use what I call "native" rsync we offload the CPU usage and can gain even higher transfer speeds. This would show even better if you where to run these tests on better hardware than what I currently have.

Begin by creating the /etc/rsyncd.conf (this is a very simple config file). You could also enable auth users and utilize user password to secure the transfer even more. I have chosen to only allow my management/storage network instead.

rsyncd.conf
---------
max connections = 1
log file = /var/log/rsync.log
timeout = 300

[images]
comment = VM images path
path = /var/lib/vz/images
read only = no
list = yes
uid = root
gid = root
list = yes
hosts allow = 192.168.10.0/24
#auth users =
#secrets file = /etc/rsyncd.secrets
---------

You also have to enable rsync on the hosts by changing the file /etc/defaults/rsync and set RSYNC_ENABLE=true before you can start the rsync daemon with
#/etc/init.d/rsync start

Since md4 tends to be 50% or so faster than md5, running rsync with --protocol=28 offloads the CPU a bit but also increases the transfer speed.

Now when we transfer a vm image from one host to the other using
#time rsync -hSWr --protocol=28 --progress /var/lib/vz/images/102 192.168.10.22::images
building file list ...
2 files to consider
102/vm-102-disk-1.raw
8.59G 100% 100.20MB/s 0:01:21 (xfer#1, to-check=0/2)

sent 8.59G bytes received 32 bytes 104.13M bytes/sec
total size is 8.59G speedup is 1.00

real 1m21.874s
user 0m33.255s
sys 0m43.284s

The image file is now transfered at higher speed (utilizing my 1Gbit network to max) and it took only 1min 21sec to complete. This also comes with the huge bonus of not using SSH which results in a lot less CPU usage on the hosts as you can see in the attached pictures.

rsync -hSWr --protocol=28 --progress
is a shorter version of the same command as below.
#rsync --protocol=28 -h --progress --sparse --whole-file --recursive

Now this is what I would like to see as default in proxmox VE 2.0.

By doing this one could utilize dedicated nic's with 802.3ad (LAG) and transfer image files at really high throughput rates and shorten the migrations down to seconds (depending on the size of the image of course and you how fast you storage is). With a LAG of 4 1Gbit nics the above file transfer could possibly be done in about 20sec. If you have network cards that support jumbo frames (MTU 9000) then that would be good option to use as well.

Conclusion
Simply by changing some values in the default config files of proxmox VE we can reduce the migration from 7:24 to 2:01 (in my example above).
With a little bit more work and changing the way of transferring images to "native" rsync instead of relying on SSH we go from 7:24 to 1:21. Then we get much better performance and utilizing a lot less CPU on the hosts.

Dietmar, I hope I have been clear enough now and showed you in detail what it is I'm talking about regarding this specific issue.

Just noticed that the online migrations of VM's now also works better. Before it took 16sec to do online migration of my test VM. Now it's done in 10sec.
That is with the ssh ciper set to arcfour and using rsync option --protocol=28.

I wonder if online migrations would also benefit from using "native" rsync as well. Would be really nice if it could be done in less then 10 seconds...

e100 said:

My processors have AES-NI instructions. From what I have gathered the debian openssl does not contain the patches to support AES-NI.
I installed SSH from wheezy using apt pinning.
Then I installed libssl1.0.0_1.0.0e-2ubuntu4_amd64.deb package from Ubuntu 11.10 because it has the AES patches.
modprobe aes to load the AES-NI module and wow, the speed is different!

Code:

arcfour256
test-file                                                                                     100%  512MB 128.0MB/s   00:04    
test-file                                                                                     100%  512MB  85.3MB/s   00:06    
arcfour128
test-file                                                                                     100%  512MB 128.0MB/s   00:04    
test-file                                                                                     100%  512MB 102.4MB/s   00:05    
aes128-cbc
test-file                                                                                     100%  512MB 256.0MB/s   00:02    
test-file                                                                                     100%  512MB 256.0MB/s   00:02    
blowfish-cbc
test-file                                                                                     100%  512MB  85.3MB/s   00:06    
test-file                                                                                     100%  512MB  73.1MB/s   00:07    
cast128-cbc
test-file                                                                                     100%  512MB  85.3MB/s   00:06    
test-file                                                                                     100%  512MB  73.1MB/s   00:07    
aes192-cbc
test-file                                                                                     100%  512MB 256.0MB/s   00:02    
test-file                                                                                     100%  512MB 170.7MB/s   00:03    
aes256-cbc
test-file                                                                                     100%  512MB 170.7MB/s   00:03    
test-file                                                                                     100%  512MB 170.7MB/s   00:03

If you were using 1Gb Ethernet using AES-NI would reduce CPU usage and your Ethernet would be a bottleneck.

Click to expand...

Using 1Gb network you would max it out at about 100-105MB/s, What wire/network speed do you have? Or are these tests run locally? Also it is important to use large files containing actual data (not just a files filled with zeros dd if=/dev/zero...) when doing these tests 5-10GB. But this sure looks promising.
Still though using rsync without SSH would be the best choice if you are simply looking at performance and host CPU utilization.

dietmar said:

AFAIK rsync uses MD5 because MD4 is not not secure enough?

Click to expand...

Not secure enough? But why would you that extra security in transferring data on a secure management/storage network? None of the MDx are that secure anyway so should that really matter when we the goal should be to optimize this for this specific solution/product.
If you where to rsync over public networks, then yes. Doing it over SSH connections using what ever cipher you feel secure with and let rsync use it's default MD5 is a good idea. But as said in this scenario we absolutely don't require this and in my mind the default behavior should be changed.
As e100 suggested earlier. Add it to the manager where you could easily have some options to change this your self.
Options that says.
1. If you are using a secure management/storage network then simply rsync
2. If you are using unsecure management/storage network, choose cipher and protocol as desired.