32 new servers, 10GE and Netapp for Proxmox and a MySQL and perf test questions.

[aaronrryan]

I just got a bunch of new hardware:
16x new servers (Dell R220 w/ 32GB RAM, 1TB 10k rpm SATA, Intel X520 10GE Nics)
16x new servers (Dell R220 w/ 32GB RAM, 512GB SSD SATA, Intel X520 10GE Nics)
2x used Cisco Nexus 5020 w/ 40 port 10GE each
1x used Netapp FAS3240 with 512GB Flash Cache and 24x 600GB SAS drives.

I plan on setting up 2 proxmox clusters:
one cluster with all SSD's (named ssdcld01 - 16)
the other with all 1TB 10krpm SATA (named hacld01 - 16)

The plan is to use the Netapp via NFS to store all the KVM machines for the hacldxx servers.
and the ssdcldxx servers will be running OpenVZ/LXC on local SSD.

I have some production mysql database servers that I'm running in master-slave config on bare metal, and I'm
considering moving them into a vm.

I'm fairly certain the ssdcldxx servers would be the best place to put the mysql servers, since the SSD performance would benefit mysql,
but I prefer the KVM snapshots on the netapp.

Has anyone done any comparisons of mysql on kvm vs. openvz/lxc, and local SSD vs NFS via 10GE?

I already have a test environment setup, so I'll be running some performance testing, but I was wondering if anyone else has done these tests,
and what their results were.

What tests did/would you run, I would assume:

• netperf
• iozone
• bonnie++
• any others?

Thanks for any input.

Aaron Ryan

 

[udo]

I just got a bunch of new hardware:
16x new servers (Dell R220 w/ 32GB RAM, 1TB 10k rpm SATA, Intel X520 10GE Nics)
16x new servers (Dell R220 w/ 32GB RAM, 512GB SSD SATA, Intel X520 10GE Nics)
2x used Cisco Nexus 5020 w/ 40 port 10GE each
1x used Netapp FAS3240 with 512GB Flash Cache and 24x 600GB SAS drives.

I plan on setting up 2 proxmox clusters:
one cluster with all SSD's (named ssdcld01 - 16)
the other with all 1TB 10krpm SATA (named hacld01 - 16)

The plan is to use the Netapp via NFS to store all the KVM machines for the hacldxx servers.
and the ssdcldxx servers will be running OpenVZ/LXC on local SSD.

I have some production mysql database servers that I'm running in master-slave config on bare metal, and I'm
considering moving them into a vm.

I'm fairly certain the ssdcldxx servers would be the best place to put the mysql servers, since the SSD performance would benefit mysql,
but I prefer the KVM snapshots on the netapp.

Has anyone done any comparisons of mysql on kvm vs. openvz/lxc, and local SSD vs NFS via 10GE?

I already have a test environment setup, so I'll be running some performance testing, but I was wondering if anyone else has done these tests,
and what their results were.

What tests did/would you run, I would assume:

• netperf
• iozone
• bonnie++
• any others?

Thanks for any input.

Aaron Ryan

Hi,
I don't have experiences with NFS...

About your config:
with mysql on localstorage you can't do live-migration and if one node die, all VMs (which use local storage) on that node are not restartable...

I would use drbd for that (with drbd8 (pve 3.x days) with drbd-resources between 2 nodes each).

About perf-testing:
iperf for network and fio for IO.

Udo

 

[hec]

Hey

You should use sio_ntap which is the tool for simulate IO.

Which Ontap release do you have?

You use the right nfs tunings?

24 disks are too low to get a good performance you need more disks also the 3240 is not real fast.

For databases create dedicated volumes and disable readahead on this volumes. FlashCache will speed up for reads but does not help you at writes.

You can also add SSDs to the aggregate this is called FlashPool and helps to improve performance?

Which Shelfs do you use DS14Mk4 or DS2246 or DS4246?

For NetApp use static PC and no LACP.

If you need NetApp support you can always contact me i work as systems engeneer for a NetApp service partner.

 

[aaronrryan]

We are using Clustered FAS3240 OnTap 8.2.3 7mode

1 FAS3240 has 1xDS2246 w/ 24x 600GB 2.5" SAS
the other the side has 3xDS2246.
They both are attached via single 10GE connections.

I also have a FAS3240 in development environment with 1xDS14mk4 w/ 14x300GB 15krpm drives also connect with 10GE.

I havent done any nfs tuning, just default settings so far.

I did a bunch of fio random read/writes here are the results:

Server Type

Proxmox Host to local SSD

Proxmox Host to 10GE NFS Netapp 24xSAS

Proxmox Host to 10GE NFS Netapp 72xSAS

Proxmox Host to 10GE NFS FreeNAS 2xSSD RAID0

Proxmox Host to Local 1TB 2.5" 7.2krpm SATA

KVM on local SSD virtio raw

KVM on local SSD IDE raw

KVM on 10GE NFS Netapp 72xSAS virtio raw

KVM on 10GE NFS Netapp 24xSAS virtio raw

KVM on 10GE NFS Netapp 24xSAS IDE raw

KVM on 10GE NFS FreeNAS 2xSSD RAID0 virtio raw

LXC on 10GE NFS Netapp 24xSAS

[TD="align: center"]fio
read
(iops)[/TD]
[TD="align: center"]fio
write
(iops)[/TD]

[TD="align: center"]71334[/TD]
[TD="align: center"]23817[/TD]

[TD="align: center"]10104[/TD]
[TD="align: center"]3369[/TD]

[TD="align: center"]14980[/TD]
[TD="align: center"]4995[/TD]

[TD="align: center"]6668[/TD]
[TD="align: center"]2222[/TD]

[TD="align: center"]347[/TD]
[TD="align: center"]115[/TD]

[TD="align: center"]32746[/TD]
[TD="align: center"]10920[/TD]

[TD="align: center"]3372[/TD]
[TD="align: center"]1124[/TD]

[TD="align: center"] 10779 [/TD]
[TD="align: center"] 3598 [/TD]

[TD="align: center"] 7814 [/TD]
[TD="align: center"] 2632 [/TD]

[TD="align: center"]314[/TD]
[TD="align: center"]104[/TD]

[TD="align: center"]3757[/TD]
[TD="align: center"]1263[/TD]

[TD="align: center"]5140[/TD]
[TD="align: center"]1729[/TD]

As reference, here is the fio command I ran for the test:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

 

[hec]

net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_mem = 3622080       4829440  7244160
net.ipv4.tcp_wmem = 40960        163840   41943040
net.ipv4.tcp_rmem = 40960        873800   41943040
net.ipv4.udp_mem = 363168       484224  726336
vm.swappiness=0
vm.dirty_background_ratio = 3
vm.dirty_ratio = 40
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
sunrpc.tcp_slot_table_entries = 128

put these settings to sysctl and rerun the tests also try to run the sio_ntap and show the results

do you use any mount options or default?

what does sysstat -u 1 show on the netapp while you run the tests?

i would recommend a 2nd 10G port per controller

 

[aaronrryan]

I'm using the default nfs mount options that proxmox uses, which is the following:
(rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=4046,mountproto=udp,local_lock=none)

I tried updating the sysctl and reran the tests, and the performance was better.

I re-ran some of the fio and added sio_ntap, and dd and got the following:

Server Type

Proxmox Host to local SSD

Proxmox Host to 10GE NFS Netapp 72xSAS

Proxmox Host to 10GE NFS Netapp 24xSAS

Proxmox Host to 1GE NFS Netapp 24xSAS

Proxmox Host to 10GE NFS FreeNAS 2xSSD RAID0

Proxmox Host to Local 1TB 2.5" 7.2krpm SATA

KVM on local SSD IDE raw

KVM on 10GE NFS Netapp 72xSAS virtio raw

KVM on 1GE NFS Netapp 24xSAS virtio raw

KVM on 10GE NFS Netapp 24xSAS IDE raw

KVM on 10GE NFS FreeNAS 2xSSD RAID0 virtio raw

LXC on 10GE NFS Netapp 24xSAS

OpenVZ on Local SSD

[TD="align: center"] Category [/TD]

[TD="align: center"] fio
read
(iops) [/TD]
[TD="align: center"] fio
write
(iops) [/TD]
[TD="align: center"] sio_ntap
(iops) [/TD]
[TD="align: center"] dd
4GB file
(MB/s) [/TD]

[TD="align: center"] Bare Metal [/TD]

[TD="align: center"] 71334 [/TD]
[TD="align: center"] 23817 [/TD]
[TD="align: center"] 6836 [/TD]
[TD="align: center"] 587 [/TD]

[TD="align: center"] Bare Metal [/TD]

[TD="align: center"] 14980 [/TD]
[TD="align: center"] 4995 [/TD]
[TD="align: center"] 9710 [/TD]
[TD="align: center"] 350 [/TD]

[TD="align: center"] Bare Metal [/TD]

[TD="align: center"] 10104 [/TD]
[TD="align: center"] 3369 [/TD]
[TD="align: center"] 8140 [/TD]
[TD="align: center"] 350 [/TD]

[TD="align: center"] Bare Metal [/TD]

[TD="align: center"] 7638 [/TD]
[TD="align: center"] 2547 [/TD]
[TD="align: center"] 5121 [/TD]
[TD="align: center"] 107 [/TD]

[TD="align: center"] Bare Metal [/TD]

[TD="align: center"] 6668 [/TD]
[TD="align: center"] 2222 [/TD]
[TD="align: center"] 1537 [/TD]
[TD="align: center"] 949 [/TD]

[TD="align: center"] Bare Metal [/TD]

[TD="align: center"] 347 [/TD]
[TD="align: center"] 115 [/TD]
[TD="align: center"] 388 [/TD]
[TD="align: center"] 152 [/TD]

[TD="bgcolor: #ffff00, align: center"] KVM-SSD [/TD]
[TD="bgcolor: #FFFF00"] KVM on local SSD virtio raw [/TD]
[TD="bgcolor: #ffff00, align: center"] 32746 [/TD]
[TD="bgcolor: #ffff00, align: center"] 10920 [/TD]
[TD="bgcolor: #ffff00, align: center"] 5483 [/TD]
[TD="bgcolor: #ffff00, align: center"] 539 [/TD]

[TD="align: center"] KVM-SSD [/TD]

[TD="align: center"] 3372 [/TD]
[TD="align: center"] 1124 [/TD]
[TD="align: center"] 2933 [/TD]
[TD="align: center"] 276 [/TD]

[TD="align: center"] KVM-NFS [/TD]

[TD="align: center"] 14350 [/TD]
[TD="align: center"] 4785 [/TD]
[TD="align: center"] 4172 [/TD]
[TD="align: center"] 402 [/TD]

[TD="bgcolor: #ffff00, align: center"] KVM-NFS [/TD]
[TD="bgcolor: #FFFF00"] KVM on 10GE NFS Netapp 24xSAS virtio raw [/TD]
[TD="bgcolor: #ffff00, align: center"] 9973 [/TD]
[TD="bgcolor: #ffff00, align: center"] 3326 [/TD]
[TD="bgcolor: #ffff00, align: center"] 4406 [/TD]
[TD="bgcolor: #ffff00, align: center"] 357 [/TD]

[TD="align: center"] KVM-NFS [/TD]

[TD="align: center"] 8440 [/TD]
[TD="align: center"] 2814 [/TD]
[TD="align: center"] 2218 [/TD]
[TD="align: center"] 107 [/TD]

[TD="align: center"] KVM-NFS [/TD]

[TD="align: center"] 314 [/TD]
[TD="align: center"] 104 [/TD]
[TD="align: center"] 412 [/TD]
[TD="align: center"] 252 [/TD]

[TD="align: center"] KVM-NFS [/TD]

[TD="align: center"] 3757 [/TD]
[TD="align: center"] 1263 [/TD]
[TD="align: center"] 1478 [/TD]
[TD="align: center"] 760 [/TD]

[TD="align: center"] LXC-NFS [/TD]

[TD="align: center"] 5140 [/TD]
[TD="align: center"] 1729 [/TD]
[TD="align: center"] 311 [/TD]
[TD="align: center"] 309 [/TD]

[TD="align: center"] OpenVZ-SSD [/TD]

[TD="align: center"] 8862 [/TD]
[TD="align: center"] 2944 [/TD]
[TD="align: center"] 15518 [/TD]
[TD="align: center"] 541 [/TD]

Here is the sio_ntap command I used: sio_ntap_linux 50 100 32k 4g 10 30 test -direct

I plan on using the 2 highlighted rows.
KVM on NFS for mysql servers, mainly for the snapshots. (high i/o master mysql servers will remain on bare metal, with slaves on NFS)
KVM on SSD for machines requiring high i/o for local file cache, where daily backups are ok (like load balanced servers)

And here is a sample of "sysstat -u 1" on the Netapp during the fio:

CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
56% 11006 14527 36265 53824 0 0 0 18s 81% 0% - 99%
67% 10405 13286 34675 57332 68160 0 0 18s 84% 40% H 99%
58% 10619 14082 34942 52600 82432 0 0 0s 82% 100% : 100%
58% 10872 13878 36241 52220 68763 0 0 20s 82% 100% : 99%
62% 11325 14773 37453 52776 80732 0 0 20s 82% 100% : 100%
57% 11079 14070 37001 50712 57888 0 0 21s 82% 100% : 99%
58% 11726 15316 38769 53596 0 0 0 21s 82% 26% : 100%
60% 12020 15223 40175 54088 24 0 0 20s 83% 0% - 100%

And here is a sample of "sysstat -u 1" on the Netapp during the sio_ntap:
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
84% 6033 400285 10833 26490 251957 0 0 2s 95% 96% M 89%
82% 4558 311007 8304 21555 298674 0 0 0s 95% 99% B 78%
89% 4452 305687 8180 21488 452610 0 0 0s 95% 100% B 89%
88% 4680 317363 8485 20594 383266 0 0 13 95% 100% B 84%
87% 5264 364832 9726 24783 318911 0 0 0s 91% 100% B 91%
76% 3353 220655 5942 17688 457799 0 0 0s 95% 88% 2 90%
75% 5162 353383 9444 26844 281017 0 0 1s 95% 89% M 95%
91% 5116 356852 9632 24389 414682 0 0 2s 95% 100% B 93%
85% 5082 348918 9322 21561 353494 0 0 1s 95% 100% B 88%
64% 3526 135313 34662 43186 301828 0 0 0s 94% 91% B 78%
49% 4170 73002 70523 69726 47481 0 0 1s 94% 11% M 83%
52% 4380 77124 73635 74040 165685 0 0 1s 94% 90% : 92%
39% 4948 88037 82332 76404 0 0 0 1s 94% 0% - 84%
55% 3880 68508 65070 64424 222088 0 0 1s 95% 97% M 88%
40% 4893 86312 82143 71212 9084 0 0 1s 94% 30% : 87%
55% 4052 69743 69728 63684 142228 0 0 13 95% 81% M 88%
43% 4592 79674 77438 68060 87424 0 0 13 94% 78% : 87%
50% 4096 70952 69996 62080 103544 0 0 13 95% 50% M 82%
44% 4561 77461 79328 66496 125924 0 0 13 95% 100% : 88%

 

[avladulescu]

net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_mem = 3622080       4829440  7244160
net.ipv4.tcp_wmem = 40960        163840   41943040
net.ipv4.tcp_rmem = 40960        873800   41943040
net.ipv4.udp_mem = 363168       484224  726336
vm.swappiness=0
vm.dirty_background_ratio = 3
vm.dirty_ratio = 40
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
sunrpc.tcp_slot_table_entries = 128

put these settings to sysctl and rerun the tests also try to run the sio_ntap and show the results

do you use any mount options or default?

what does sysstat -u 1 show on the netapp while you run the tests?

i would recommend a 2nd 10G port per controller

Might I make a comment on the vm.dirty_background_ratio and vm.dirty_ratio ?

I would see these value not being very appropriate for a hypervisor host, but more like vm.dirty_background_ratio = 10 and vm.dirty_ratio =5 as data should be flushed much faster to disk source and avoiding the piling'up on the hypervisor node.

As well, vm.swappiness=0, which it is not recommended to be totally turned off, but instead to use the value of 1 or 10, depending on the kernel version it is used on the hypervisor.

1 = Kernel version 3.5 and over: Minimum amount of swapping without disabling it entirely.
10 = This value is sometimes recommended to improve performance when sufficient memory exists in a system.

On the linux system, as I know and understand, there are some kernel threads which really make use of swap space (but not swapping aggregatively) more like putting very small chunks of data to it in order to speed things up.

These 3 settings are changing the benchmarks reports considerable (considering my past tests, when I was trying to find the best value to use on the guest server).