Cluster Communication

[adamb]

I need some input. Should I look to have cluster communication on its own dedicated NIC's? On one of our clusters we saw a very slow failover of our gateway VM (5-10 minutes). It is a very busy gateway and all devices inhouse point to it.

I am thinking its because the VM uses the same NIC as the cluster management. Any suggestions or ideas on whats best? I appreciate the input!

 

[adamb]

Just did a bit of testing. I have two clusters.

Cluster #1.
2 x IBM x3550 M3's

Cluster #2
2 x IBM x3650 M4's

Testing with a inactive CentOS6 VM, the migration process on the M3's takes 12-15 seconds. On the M4's it takes 2-3 minutes. I am trying to figure out what the issue is. Looking at pveperf the numbers for the M4 are quite a bit better then the M3.


Cluster #1
pve-manager: 2.1-14 (pve-manager/2.1/f32f3f46)
running kernel: 2.6.32-14-pve
proxmox-ve-2.6.32: 2.1-74
pve-kernel-2.6.32-14-pve: 2.6.32-74
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-49
pve-firmware: 1.0-18
libpve-common-perl: 1.0-30
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-31
vncterm: 1.0-3
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-8
ksm-control-daemon: 1.1-1

Cluster #2
pve-manager: 2.1-14 (pve-manager/2.1/f32f3f46)
running kernel: 2.6.32-14-pve
proxmox-ve-2.6.32: 2.1-74
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-14-pve: 2.6.32-74
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-49
pve-firmware: 1.0-18
libpve-common-perl: 1.0-30
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-31
vncterm: 1.0-3
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-8
ksm-control-daemon: 1.1-1

 

[dietmar]

I am thinking its because the VM uses the same NIC as the cluster management. Any suggestions or ideas on whats best? I appreciate the input!

For HA it is better to use a separate network.

 

[adamb]

For HA it is better to use a separate network.

Sounds good I am going to get the networks separated. I don't feel this is the issue for the really slow migration though.

What exactly is moved over the wire when a live migration takes place? I can see heavy traffic when live migration takes place.

The Slower cluster reports the following.
Nov 16 08:58:06 starting migration of VM 103 to node 'fiosprox2' (10.80.12.128)
Nov 16 08:58:06 copying disk images
Nov 16 08:58:06 starting VM 103 on remote node 'fiosprox2'
Nov 16 08:58:07 starting migration tunnel
Nov 16 08:58:08 starting online/live migration on port 60000
Nov 16 08:58:14 migration status: active (transferred 447300640, remaining 5841100800), total 6308626432)
Nov 16 08:58:17 migration status: active (transferred 660875446, remaining 5556670464), total 6308626432)
Nov 16 08:58:20 migration status: active (transferred 891092375, remaining 5304893440), total 6308626432)
Nov 16 08:58:23 migration status: active (transferred 1072295639, remaining 5122392064), total 6308626432)
Nov 16 08:58:26 migration status: active (transferred 1308016577, remaining 4885762048), total 6308626432)
Nov 16 08:58:29 migration status: active (transferred 1512800247, remaining 4680765440), total 6308626432)
Nov 16 08:58:32 migration status: active (transferred 1744262037, remaining 4445511680), total 6308626432)
Nov 16 08:58:35 migration status: active (transferred 1964305342, remaining 4216971264), total 6308626432)
Nov 16 08:58:38 migration status: active (transferred 2202610803, remaining 3980230656), total 6308626432)
Nov 16 08:58:41 migration status: active (transferred 2428386518, remaining 3754086400), total 6308626432)
Nov 16 08:58:44 migration status: active (transferred 2649492737, remaining 3532816384), total 6308626432)
Nov 16 08:58:48 migration status: active (transferred 2861174181, remaining 3320483840), total 6308626432)
Nov 16 08:58:51 migration status: active (transferred 3089899064, remaining 3091156992), total 6308626432)
Nov 16 08:58:54 migration status: active (transferred 3306520158, remaining 2874380288), total 6308626432)
Nov 16 08:58:56 migration status: active (transferred 3494592318, remaining 2685390848), total 6308626432)
Nov 16 08:58:59 migration status: active (transferred 3707595353, remaining 2460864512), total 6308626432)
Nov 16 08:59:03 migration status: active (transferred 3925167030, remaining 2241990656), total 6308626432)
Nov 16 08:59:32 migration speed: 71.43 MB/s
Nov 16 08:59:32 migration status: completed
Nov 16 08:59:35 migration finished successfuly (duration 00:01:29)

On my cluster with fast migration I see the following when migrating.
Executing HA migrate for VM 100 to node proxmox2
Trying to migrate pvevm:100 to proxmox2...Success
TASK OK


Why is there such a difference in migration process between the two?

Im starting to think the best way to speed this up is to use 10GB nics.

 

[adamb]

Another quick question, is it bad practice to have the cluster management network on the DRBD network?

I would like to uprade to 10GB card for DRBD and move cluster management to this nic also. I currently have DRBD isolated on its own private network.

 

[tom]

you compare the wrong logs. HA migrate task log just shows that a migration is triggered by HA. check again, you should see two logs here.

 

[tom]

Another quick question, is it bad practice to have the cluster management network on the DRBD network?

yes, don´t do it. use a dedicated link for DRBD. and if you use HA, use redundant and dedicated network for cluster communication.

I would like to uprade to 10GB card for DRBD and move cluster management to this nic also. I currently have DRBD isolated on its own private network.

 

[dietmar]

What exactly is moved over the wire when a live migration takes place?

RAM content.

Why is there such a difference in migration process between the two?

Different RAM usage. Maybe a process inside the VM constantly change RAM (for example playing a video)?

 

[adamb]

RAM content.




Different RAM usage. Maybe a process inside the VM constantly change RAM (for example playing a video)?

That makes perfect sense as to why it took so long, our gateway VM has 50GB and is very dependent on ram.

Am I right in thinking that getting our cluster management network upgraded to 10GB would help speed up the process?

Do you see any reason to not have DRBD and cluster management on the same 10GB private network. This would be a direct 10GB <-> 10GB connection. No switches.

I am bit concerned with SSH becoming the bottle neck once I move to 10GB cards as it is only capable of a single thread. To make matter worse it looks like proxmox isn't using the newest version of openssl which support AES-NI. This drastically speeds up ssh when using AES as long as your processor supports.


One of my proxmox Nodes
root@fiosprox1:~# openssl version
OpenSSL 0.9.8o 01 Jun 2010

root@fiosprox1:~# openssl engine
(dynamic) Dynamic engine loading support


One of my VM's on the proxmox node, that has a newer version of openssl

[root@fiosweb2 ~]# openssl version
OpenSSL 1.0.0-fips 29 Mar 2010

[root@fiosweb2 ~]# openssl engine
(aesni) Intel AES-NI engine
(dynamic) Dynamic engine loading support

Looking at quite a few how to's, it seems there should be no issues running DRBD and Cluster Management on the same network.

 

[dietmar]

Do you see any reason to not have DRBD and cluster management on the same 10GB private network.

yes (DRBD produce much traffic)

I am bit concerned with SSH becoming the bottle neck

Yes, that is very likely.

 

[adamb]

yes (DRBD produce much traffic)

Yes, that is very likely.

Currently our DRBD operates with no issue over a 1GB connection. You think it could eat up the whole 10GB connection and kill cluster communcation?

Any ideas when openssl will be updated. What cipher is currently being used when a migration takes place?

I appreciate all the input guys!

 

[dietmar]

Currently our DRBD operates with no issue over a 1GB connection. You think it could eat up the whole 10GB connection and kill cluster communcation?

Yes, But AFAIK you can set BW limits with DRBD.

Any ideas when openssl will be updated.

I guess when we update to Debian wheezy.

What cipher is currently being used when a migration takes place?

The current default is blowfish (set inside /root/.ssh/config)

 

[adamb]

Yes, But AFAIK you can set BW limits with DRBD.



I guess when we update to Debian wheezy.



The current default is blowfish (set inside /root/.ssh/config)

Do I need to create that file? Doesn't look like it exists.

root@fiosprox2:~/.ssh# ls -ltrha
total 20K
-rw-r--r-- 1 root root 396 Oct 3 11:38 id_rsa.pub
-rw------- 1 root root 1.7K Oct 3 11:38 id_rsa
lrwxrwxrwx 1 root root 29 Oct 3 11:38 authorized_keys -> /etc/pve/priv/authorized_keys
-rw-r--r-- 1 root root 442 Oct 3 11:38 known_hosts
drwxr-xr-x 2 root root 4.0K Oct 3 11:38 .
drwx------ 5 root root 4.0K Nov 27 17:22 ..

 

[tom]

you do not run the latest version? update and the file should be there.