suggestions for shared storage for productive environment

[silbro]

Hello everybody

I've been using proxmox for a while now and I'm very happy with it. I've been using it with linux only and a freenas as a shared storage. I tested windows but it wasn't running good enough to actually use. I'm running Zimbra, bind, seafile, gameservers, teamspeak and apache servers which work absolutely perfect. This is all just for friends, family or myself so downtime isn't such a big issue. I tried out CEPH storage just because I was curious. When I tried it I had 4PCs with a 1Gig network and a few (maybe 8 in total) HDDs to test. The speed of the CEPH with linux machines was rather slow -> 40 - 60 MB/s. I have the following questions:

1) what shared storage is good for a productive environment? (there are so many options that I cannot decide what is actually good with proxmox and worth the money)
2) How many OSD for CEPH would I need to get nice performance where maybe a 10GB network is actually worth investing into? What speeds can I excpect when copying large files in linux and windows? What about I/Os, are they good enough? (yes i know it depends on the application
3) The idea is also to run Windows Servers. Do you think proxmox is good enough for Windows Terminal Servers and Windows SQL servers? (users up to 50 per server)
4) Are blade servers (or alike) working with proxmox?
5) I have a book where there are examples for proxmox builds, unfortunately I can't get all these hardware parts in my country. For example, there are only certain supermicro boards that I can get from vendors. Can you suggest me anything here? Maybe a hardware list or a build list that you have used and are satisfied with?

I'm sorry for asking so many questions, but I have invested a lot of time to find possibilities and just got lost. For example, EMC, NetAPP, Synology SAN, IBM SAN etc. I have no idea what is good and what isn't. I'd actually prefer to build my own systems (caus I like to do this and I have enough time If I have the option to use proxmox instead of hyper-v or vmware for Windows I definately would do so. For a productive environment I'd also buy proxmox support of course. The idea eventually is to spend around 20'000 (if necessary maybe even more) for hardware and have a HA over all hardware (including firewall and switches). For firewalls pfsense is a good and very affordable option. I'd like to run as many vms as possible of course, but if I'd have to give you a number I'd say probably around 30 Windows Terminal or SQL Servers and probably around 50 linux servers. Linux servers won't need much performance.

cheers and thanks for helping out!!

 

[nethfel]

A lot of this is my opinion, so take it or leave it

1) what shared storage is good for a productive environment? (there are so many options that I cannot decide what is actually good with proxmox and worth the money)

We use a combination of CEPH and NFS for our back end storage - mostly CEPH as VM storage and NFS as backup and ISO storage. I personally like CEPH. If you aren't running DB heavy VMs then Gluster is an option as well. NFS is an option, but really a bear to setup if you want to try to do HA with it.

2) How many OSD for CEPH would I need to get nice performance where maybe a 10GB network is actually worth investing into? What speeds can I excpect when copying large files in linux and windows? What about I/Os, are they good enough? (yes i know it depends on the application

In terms of when to move to a 10GB network - I can't really answer that. Currently we use 18 OSDs with 6 journal SSDs. The CEPH servers are independent of the VM servers. We also run 3 networks for our CEPH servers (if you just install CEPH using ProxMox, by default when you create the CEPH network, it generates a single network for both access and cluster data - you need to change this otherwise you'll be flooding your public network (the access network) with cluster traffic which can cause grief and slowdown). Our three networks for CEPH is as follows: 1 - 1x 1Gbe: Proxmox Management (just using the PVE web interface for managing ceph), 2 - 2x 1Gbe: Public/client network bonded via Balance ALB (could also bond via LACP), 3 - 3x 1Gbe: CEPH Cluster network bonded by Round Robin (using 3 vlans - I used RR as it gave better overall throughput which increased my write throughput on the CEPH cluster; RR didn't change read performance).

There will be a point in terms of the amount of OSDs and OSD Nodes where moving from 1Gb (even bonded) to 10Gb makes sense - I know I haven't reached it yet. I also know I'm not surprised you got terrible performance with so few OSDs, especially with no SSDs for the journals.

3) The idea is also to run Windows Servers. Do you think proxmox is good enough for Windows Terminal Servers and Windows SQL servers? (users up to 50 per server)

I think a lot of that will depend on how beefy your host servers are, how many resources you throw at the VMs and if you install all of the guest drivers and follow the best practices. I noticed when I first started playing with ProxMox, as an example - I just used IDE for my disks, my Windows VM performance was terrible compared to if I went through the process to use the virtio drivers.

4) Are blade servers (or alike) working with proxmox?

For my needs no. The blade servers I had the option to get (basically reclaim from them not getting used elsewhere) only had support for 2 Gbe connections per blade, which wasn't enough for what I wanted (1 for management, 2 for connecting to cluster public network and at least 2 for the VM guests, so my need for 5+ ethernet adapters killed my ability to use the blade servers). Now, if they had 10Gb on the blade servers it would have worked for me just fine.

5) I have a book where there are examples for proxmox builds, unfortunately I can't get all these hardware parts in my country. For example, there are only certain supermicro boards that I can get from vendors. Can you suggest me anything here? Maybe a hardware list or a build list that you have used and are satisfied with?

The sample hardware is just that - a sample of what you can run it on. We are running our VM Hosts on second hand retired Dell custom cloud servers found on EBay (Dell CS24-SC) w/ 2 drives in mirror for boot and 1 quad port ethernet add-on card (total 6 Gbe, 2 onboard, 4 on the card not including the OOB management port). You will need to choose hardware based on your requirements more than the samples or for what someone else uses. My VM needs are only 1-2 cores per VM and between 512 MB and 4 GB Ram per vm where as someone running a TS might need many more cores and a lot more RAM to support their operation depending on the amount of simultaneous clients. And then of course MSSQL is happy with as much RAM as you can throw at it. Most of my VMs don't hog a lot of resources, and with the 20 VMs I'm running across three nodes, I see an average of 10% CPU usage per node and ~50% memory usage.

I'm sorry for asking so many questions, but I have invested a lot of time to find possibilities and just got lost. For example, EMC, NetAPP, Synology SAN, IBM SAN etc. I have no idea what is good and what isn't. I'd actually prefer to build my own systems (caus I like to do this and I have enough time If I have the option to use proxmox instead of hyper-v or vmware for Windows I definately would do so. For a productive environment I'd also buy proxmox support of course. The idea eventually is to spend around 20'000 (if necessary maybe even more) for hardware and have a HA over all hardware (including firewall and switches). For firewalls pfsense is a good and very affordable option. I'd like to run as many vms as possible of course, but if I'd have to give you a number I'd say probably around 30 Windows Terminal or SQL Servers and probably around 50 linux servers. Linux servers won't need much performance.

You are right, there are a ton of options for storage (although the list starts to get smaller very quickly if you are looking for distributed/HA storage). My personal choice for where I work was to go with CEPH - although it may be a bit slower then some of the other options available, the built in redundancy and HA really sold me - plus the ease in expansion with (relatively) cheap hardware just made the solution right for us. Everyones needs are different - if you have enough hardware sitting around, I'd try to setup a larger ceph network and see what kind of performance you can squeeze out of it.

 

[brad_mssw]

I don't run Windows VMs, but I can't imagine the performance difference would be far off from HyperV or VMWare since the network and disk are both paravirtualized and all solutions use the hardware extensions provided by the CPU. Maybe I'm wrong, but I've been happy with Proxmox, but run primarily Linux systems.

We have 3 total combined Ceph + Proxmox nodes ourselves with dual 10GbE (bonded LACP) networking, and build our own systems based on Supermicro components. We also run OpenVSwitch which greatly eases network configuration.

Here's our parts list for each node if you're interested:

• Supermicro 2U Chassis with SAS3 expander 216BE2C-R920LPB: ​http://www.supermicro.com/products/chassis/2U/216/SC216BE2C-R920LP.cfm
• Supermicro X10 Xeon E5-2600 v3 Motherboard X10DRI-O: ​http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRi.cfm
• 2x Xeon E5-2630v3 8-core, 2.4GHz CPUs
• 8x 16GB DDR4-2133 ECC Registered RAM (128GB total)
• Intel Dual SFP+ 10Gbps NIC X520-DA2
• 2x Intel DC S3500 80GB SSD (boot drives raid 1)
• 4x Intel DC S3700 400GB SSD (fast VM storage raid 5)
• 4x WD 2.5" Velociraptor 10kRPM 1TB HD (bulk VM storage raid 5)
• LSI MegaRAID SAS 9361-8i Raid controller
• LSI Supercap LSICVM02 BBU for raid controller
• LSI BBU-BRACKET-05 pci-e slot mounting bracket for BBU
• 2x CPU heatsink SNK-P0048PS
• 2x 0.6m miniSAS HD cables

No we technically use Juniper EX 4300 switches in chassis cluster mode (aka stacked) with 10GbE modules and use SFP+ Direct Attach Cables to interconnect the nodes to the switches. However, if we had it to do over again, we'd probably buy 40GbE cards instead which are nearly the same cost, and just use Direct Attach cables between the nodes in a star topology, then use bonded 1GbE nics back to the core switches. The 40GbE physical interfaces as well as the bonded 1GbE ports would be attached to the same Open vSwitch virtual switch and let spanning tree do its thing (in theory only one node's 1GbE links would be active, the rest would route through the 40GbE cards). We suspect that would not only be a lot cheaper (as the core switches could be Juniper EX2200's for a fraction of the cost), but would probably also be a lot faster.

One other thing, you'll notice we use RAID with Ceph, that's actually not recommended...we did that mainly because we were scared of how hard it might be to replace a failed drive ... but you definitely take a performance it for that. But I still get 1GB/s disk transfers within the VMs so that was good enough for what we needed.

Oh, and I'd probably go with Samsung 845DC Pro drives these days if I had it to do over, they just weren't really available when we built this stuff.

 

[udo]

Hello everybody

I've been using proxmox for a while now and I'm very happy with it. I've been using it with linux only and a freenas as a shared storage. I tested windows but it wasn't running good enough to actually use. I'm running Zimbra, bind, seafile, gameservers, teamspeak and apache servers which work absolutely perfect. This is all just for friends, family or myself so downtime isn't such a big issue. I tried out CEPH storage just because I was curious. When I tried it I had 4PCs with a 1Gig network and a few (maybe 8 in total) HDDs to test. The speed of the CEPH with linux machines was rather slow -> 40 - 60 MB/s. I have the following questions:

1) what shared storage is good for a productive environment? (there are so many options that I cannot decide what is actually good with proxmox and worth the money)
2) How many OSD for CEPH would I need to get nice performance where maybe a 10GB network is actually worth investing into? What speeds can I excpect when copying large files in linux and windows? What about I/Os, are they good enough? (yes i know it depends on the application

Hi,
this is not easy to answer because it's depends on many things (CPU-Power of OSD-Nodes, Network-Stack, SSD-Cache, number of OSDs, Filesystem-parameters...).

3) The idea is also to run Windows Servers. Do you think proxmox is good enough for Windows Terminal Servers and Windows SQL servers? (users up to 50 per server)

Yes, why not? We switch all VMs from VMware to pve (Windos/Linux).

4) Are blade servers (or alike) working with proxmox?

I had an Sun Blade-Center running. Depends on your IO-Config and storage, because normaly you can't get much local storage with blades. For me such an Blade center is not flexible enough.

5) I have a book where there are examples for proxmox builds, unfortunately I can't get all these hardware parts in my country. For example, there are only certain supermicro boards that I can get from vendors. Can you suggest me anything here? Maybe a hardware list or a build list that you have used and are satisfied with?

I'm happy with ASUS Server boards (and after some Supermicro boards I hate them).

About shared storage: an external FC-Raid is an "cheap" thing to get god performance values (but of course an SPOF). I use DRBD also (between pairs of pve-nodes) and ceph.

Udo

 

[silbro]

@udo
I also have tested DRBD once, I just find it too much work compared to a shared storage that you can only mount

Why do you hate Supermicro boards? So far I only had good experience with them (apart from needing to update a bios because the new cpu wasnt yet suppored. But I didnt have an old CPU to update, haha :-(

 

[silbro]

Thanks a lot for your reply! I'm sorry I didn't respond earlier but I was just too busy this week.

@nethfel
I actually even tried ssds for journaling but it wasn't much better. Maybe I had the wrong ssd disks or did something wrong or didn't have enough osds (that I'm sure of ). What kind of ssds do you use for journaling? Is it better to use msata ssds for journaling (so I can have more hdds) and does it always need to be a raid1 ? How many times can I use an ssd for journaling (like if i have 9 hdds do i need 6 ssds in RAID1 for journaling or could I just use 2 in a raid1?)

For Windows I always used virtio drivers. It was better than everything else but still too slow (at least with local hdds or over a 1gig network and a free nas as nfs storage). But I really think I didn't have the right equipment because with a raid controller and sdds, Windows was extremely fast!

As lucky as I am, we just got 2 servers with many disks and about 4 more servers coming, so I can test around (old hardware a client doesnt need anymore, but still ok) We even have ssds available for journaling. Will really look forward to that!

 

[silbro]

@brad_mssw
thanks a lot for your hardware list! Very good tip on open vswitch, sounds very helpful and I will definatelly look into this subject. I didn't know about this.

 

[mir]

You could also try a Solaris based shared storage using ZFS. With the ZFS support in Proxmox you have support for clone and snapshot of KVM.
To give you an idea of performance this is some performance tests from my storage (zvol's published as iSCSI over Infiniband).

CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   377.803 MB/s
          Sequential Write :   105.317 MB/s
         Random Read 512KB :   266.057 MB/s
        Random Write 512KB :   109.987 MB/s
    Random Read 4KB (QD=1) :     8.106 MB/s [  1978.9 IOPS]
   Random Write 4KB (QD=1) :     7.162 MB/s [  1748.6 IOPS]
   Random Read 4KB (QD=32) :    28.032 MB/s [  6843.8 IOPS]
  Random Write 4KB (QD=32) :    24.708 MB/s [  6032.1 IOPS]

  Test : 100 MB [F: 1.0% (58.7/6141.0 MB)] (x5)
  Date : 2015/06/02 23:38:46
    OS : Windows 7 Enterprise SP1 [6.1 Build 7601] (x64)

 

[silbro]

@mir

thanks for the benchmark seems quiet ok! Will also take a look at that.

 

[udo]

@udo
I also have tested DRBD once, I just find it too much work compared to a shared storage that you can only mount

Why do you hate Supermicro boards? So far I only had good experience with them (apart from needing to update a bios because the new cpu wasnt yet suppored. But I didnt have an old CPU to update, haha :-(

Hi,
on one system I had some intermittent bugs - which was solved by an bios-update, but supermicro don't give information about which bugs are solved due bios-updates (they forgot also release-dates of bios-updates and so on - not really proffesional).
But they warn for bios-updates!! Should only installed if you be sure it's bios-related!! ROTFL

With other system I had different errors (die, after powercycle...).

Udo