Hey all. I have a two node cluster. Here is the hardware.
IBM x3650 M4's
E5-2680
150GB ram
Both nodes are identical. I noticed a slowdown in my openvpn performance on one of my VM's. Starting to dig in I noticed the openvpn process was running quite hard on cpu for the traffic it was moving. So I did a quick 7zip cpu bench mark inside the vm to see if cpu performances is degraded. Sure enough, it is. Here is the benchmark while its running on node2.
[root@fiosweb3 ~]# 7za b
7-Zip (A) 9.13 beta Copyright (c) 1999-2010 Igor Pavlov 2010-04-15
p7zip Version 9.13 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
RAM size: 29377 MB, # CPU hardware threads: 4
RAM usage: 850 MB, # Benchmark threads: 4
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 3128 284 1073 3043 | 41335 396 941 3729
23: 3284 292 1144 3346 | 40387 393 940 3695
24: 3239 293 1187 3483 | 39982 395 938 3709
25: 3283 307 1222 3749 | 35474 353 943 3336
----------------------------------------------------------------
Avr: 294 1156 3405 384 941 3617
Here it is while on node1. I also did some transfers over my openvpn tunnel and performance is where it should be while on node1.
[root@fiosweb3 ~]# 7za b
7-Zip (A) 9.13 beta Copyright (c) 1999-2010 Igor Pavlov 2010-04-15
p7zip Version 9.13 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
RAM size: 29377 MB, # CPU hardware threads: 4
RAM usage: 850 MB, # Benchmark threads: 4
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 7816 299 2545 7604 | 100794 367 2475 9094
23: 7780 292 2717 7927 | 96162 376 2338 8799
24: 8013 323 2671 8616 | 100217 390 2386 9297
25: 8683 333 2978 9915 | 101326 392 2429 9528
----------------------------------------------------------------
Avr: 311 2728 8515 381 2407 9180
As you can see quite the difference.
Here is are the same benchmarks done on the individual host.
Node #2 with issues
root@fiosprox2:~# 7zr b
7-Zip (A) 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30
p7zip Version 9.04 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,16 CPUs)
RAM size: 96658 MB, # CPU hardware threads: 16
RAM usage: 3402 MB, # Benchmark threads: 16
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 11909 1057 1095 11585 | 133079 1551 773 11999
23: 12649 1154 1116 12888 | 119539 1378 793 10933
24: 12875 1213 1140 13843 | 132002 1568 780 12243
25: 12708 1236 1173 14510 | 127842 1527 787 12020
----------------------------------------------------------------
Avr: 1165 1131 13206 1506 783 11799
Tot: 1336 957 12503
Node #1 without issues
root@fiosprox1:~# 7zr b
7-Zip (A) 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30
p7zip Version 9.04 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,16 CPUs)
RAM size: 96658 MB, # CPU hardware threads: 16
RAM usage: 3402 MB, # Benchmark threads: 16
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 26576 1047 2469 25853 | 268088 1155 2092 24173
23: 28447 1167 2484 28984 | 296376 1328 2040 27108
24: 24022 1062 2431 25828 | 272538 1217 2077 25279
25: 24917 1121 2538 28450 | 281164 1266 2088 26437
----------------------------------------------------------------
Avr: 1099 2480 27279 1242 2074 25749
Tot: 1170 2277 26514
I am unsure what in the world could be causing this issue. I have rebooted the node with the slow performance, but it still has the issue. I would think if something is up with the cpu it would either work or not. I appreciate any and all help on this one!
IBM x3650 M4's
E5-2680
150GB ram
Both nodes are identical. I noticed a slowdown in my openvpn performance on one of my VM's. Starting to dig in I noticed the openvpn process was running quite hard on cpu for the traffic it was moving. So I did a quick 7zip cpu bench mark inside the vm to see if cpu performances is degraded. Sure enough, it is. Here is the benchmark while its running on node2.
[root@fiosweb3 ~]# 7za b
7-Zip (A) 9.13 beta Copyright (c) 1999-2010 Igor Pavlov 2010-04-15
p7zip Version 9.13 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
RAM size: 29377 MB, # CPU hardware threads: 4
RAM usage: 850 MB, # Benchmark threads: 4
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 3128 284 1073 3043 | 41335 396 941 3729
23: 3284 292 1144 3346 | 40387 393 940 3695
24: 3239 293 1187 3483 | 39982 395 938 3709
25: 3283 307 1222 3749 | 35474 353 943 3336
----------------------------------------------------------------
Avr: 294 1156 3405 384 941 3617
Here it is while on node1. I also did some transfers over my openvpn tunnel and performance is where it should be while on node1.
[root@fiosweb3 ~]# 7za b
7-Zip (A) 9.13 beta Copyright (c) 1999-2010 Igor Pavlov 2010-04-15
p7zip Version 9.13 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)
RAM size: 29377 MB, # CPU hardware threads: 4
RAM usage: 850 MB, # Benchmark threads: 4
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 7816 299 2545 7604 | 100794 367 2475 9094
23: 7780 292 2717 7927 | 96162 376 2338 8799
24: 8013 323 2671 8616 | 100217 390 2386 9297
25: 8683 333 2978 9915 | 101326 392 2429 9528
----------------------------------------------------------------
Avr: 311 2728 8515 381 2407 9180
As you can see quite the difference.
Here is are the same benchmarks done on the individual host.
Node #2 with issues
root@fiosprox2:~# 7zr b
7-Zip (A) 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30
p7zip Version 9.04 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,16 CPUs)
RAM size: 96658 MB, # CPU hardware threads: 16
RAM usage: 3402 MB, # Benchmark threads: 16
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 11909 1057 1095 11585 | 133079 1551 773 11999
23: 12649 1154 1116 12888 | 119539 1378 793 10933
24: 12875 1213 1140 13843 | 132002 1568 780 12243
25: 12708 1236 1173 14510 | 127842 1527 787 12020
----------------------------------------------------------------
Avr: 1165 1131 13206 1506 783 11799
Tot: 1336 957 12503
Node #1 without issues
root@fiosprox1:~# 7zr b
7-Zip (A) 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30
p7zip Version 9.04 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,16 CPUs)
RAM size: 96658 MB, # CPU hardware threads: 16
RAM usage: 3402 MB, # Benchmark threads: 16
Dict Compressing | Decompressing
Speed Usage R/U Rating | Speed Usage R/U Rating
KB/s % MIPS MIPS | KB/s % MIPS MIPS
22: 26576 1047 2469 25853 | 268088 1155 2092 24173
23: 28447 1167 2484 28984 | 296376 1328 2040 27108
24: 24022 1062 2431 25828 | 272538 1217 2077 25279
25: 24917 1121 2538 28450 | 281164 1266 2088 26437
----------------------------------------------------------------
Avr: 1099 2480 27279 1242 2074 25749
Tot: 1170 2277 26514
I am unsure what in the world could be causing this issue. I have rebooted the node with the slow performance, but it still has the issue. I would think if something is up with the cpu it would either work or not. I appreciate any and all help on this one!