ScaleIO on Proxmox

[MDL]

Hi,

We use EMC ScaleIO in a hyperconverged environment but not in Proxmox (yet) and find it literally shines in virtually every aspect which could be expected from a distributed storage system.

Has anyone tried to install and run the SDS natively on Proxmox hosts ?

Otherwise any idea if and how the client portion (SDC) can be integrated and used in the ProxMox Interface ?

Thanks.

 

[dietmar]

I would use Ceph instead because it is 100% open source.

 

[MDL]

Although we have some clients using Ceph for the reason you mentioned, we found ScaleIO to significantly outperform Ceph in terms of performance, manageability and simplicity.

So I deduct that it will not be possible to consume existing ScaleIO storage from Proxmox natively for the time being ?

Thanks.

 

[dgeist]

Given that the ScaleIO system is closed-source and could not be "integrated" into the proxmox environment, there's no reason that it has to be incompatible. I'm looking into trying the same thing based on experiences with CEPH's complexity and the shortfalls of its "one product does everything" strategy (i.e. I only need block storage..please make things just do block storage fast).

As ScaleIO presents block images as a mounted point in the filesystem, they should be able to be treated similarly to an NFS mount providing the same feature (including locking, live migration, etc.) The one hangup I can see is that it doesn't currently support systemd-based linux flavors (which jessie and Ubuntu 15.04 and later are, if I recall). It's probably hack-able for init purposes, which is what I plan to do to see how well it'll operate.

Dan

 

[tom]

As its closed source, you should contact the vendor of ScaleIO - only these guys can do the integration (obviously), our code and the way to get integrated is open and showed with other technologies as examples.

The majority of the Proxmox VE community love open source. They moved away from closed source software on the virtualization level (vmware, microsoft, ...) and the interest to move back to an US based company on the storage level for this user group is very limited, as there are already several open source alternatives.

 

[dgeist]

Hi, Tom. I don't think anyone would expect Proxmox or any other purely open source product to adopt support for a closed product. EMC has done a fairly good job with the product as a steward after its purchase making scaleio MORE accessible to the linux community. I've been a FOSS user/advocate since 1994, and as much as I really do like ceph, it is not the best for everything...but it is good for most things.

I get that proxmox uses ceph and drives its adoption for improvement of HA and storage capability. I'm simply suggesting that it's possible to use something else WITH it and how it could be done, not that you support it. If that kind of discussion is not welcome here, please let me know.

 

[tom]

Discussion is welcome here.

But as a potential ScaleIO customer, tell EMC that you want to use it with Proxmox VE - A discussion without the vendor will not lead to an integration or supported and stable setup.

I never used ScaleIO but almost all and especially commercial vendors are telling that are the fastest. So if there are for example 100 vendors/project, 99 % are just wrong .

 

[blackpaw]

My 2-cents, I'm getting good results with gluster (3.7.11), significantly faster than ceph for me and intgrated with proxmox

 

[spirit]

About ceph performance tuning,you can read this slides
http://events.linuxfoundation.org/s...opportunities of Open source storage_v6_0.pdf

tuning config at the end.

1000000 iops randread 4K
100000 iops randwrite 4k

(I'm reaching around same performance with 3nodes - 24 osd (intel s3610)
we can expect more write performance in coming releases, with the next bluestore format for the osd. (No more double write because no more journal)

 

[MDL]

Thank you all for your comments.

Unfortunately, IT environments are often not only about open source. Various products for various needs.

Altough we love Proxmox, it is rare that one can realistically step in an organization and say "guys, forget all your existing storage implementation and shift to CEPH in order to fit Proxmox".

ScaleIO (storage services) can easily be installed on many platforms, in mixed environments, where even Windows resources can contribute (yes, ScaleIO also installs on Windows too). The server part is not really the issue here.

EMC provides a client driver (plugin called SDC) for hosts to consume ScaleIO distributed storage and it is very easy to install it on Linux hosts. It would "just" (I know...) be a matter of integrating it to the Proxmox interface in order to make make the cluster(s) aware of the available ScaleIO resources.

In Datacenter -> Storage -> Add, we have the choice between LVM, NFS, ZFS, iSCSI etc... Are these plugins usually made available by contributors or by Proxmox developers ?

We have tried Gluster on Proxmox a while back but stumbled on some considerable stability issues and did not have time then to pursue further. However we still believe it deserve serious consideration and will be looking into it again soon.

From our experience, ScaleIO installation and use are a breeze, performance is awesome and very importantly, monitoring out of the box is feature rich and very deep, which cannot be said in aggregate for most open source storage solutions found natively on Proxmox, thus limiting for a lot of use cases.

Another two cents on transport: the prime bottleneck is often the networking for storage and RMDA (iSER / SRPt / NFS over RDMA etc.) support and awareness in the management console would improve peformance and latency tremendously while reducing CPU load significantly. Not a trivial issue in hyperconverged environments; but this is another debate...

 

[mir]

It would "just" (I know...) be a matter of integrating it to the Proxmox interface in order to make make the cluster(s) aware of the available ScaleIO resources.

If you sponsor a ScaleIO storage box to one of the devs you might could convince them to implement this integration into Proxmox.

 

[PigLover]

I think the biggest issue with Scaleio is EMCs lack of support for systemd & apt based package management - both required for simple integration to Debian. If they would support this then integration into Proxmox would be trivial - at least no harder than integrating other storage platforms that, while they might be FOSS, are not really controlled by ProxMox (including Ceph, Gluster, NFS, etc).

In other words - if the install was clean building the integration plugins would be reasonable. Unfortunately I can find no clear roadmap from EMC for support of apt based package management and/or systemd based inits.

Would be great to have yet another storage option. Unfortunately the baseline required to support remains in the hands of EMC.

 

[dgeist]

P.L., these were my thoughts as well. "Integration" in the true sense of the word is unlikely due to points made above, but happy co-existence and symbiosis seems pretty straightforward as another posix-type storage option for block images. The .apt support is there in the free downloads (just not broadcasted). Also, adding systemd support isn't actually very hard. I plan on writing some PoC .service files for the daemons when I get a few spare cycles to try things out in a somewhat-supportable install model.

Dan

 

[iptgeek]

The scaleio bots regularly troll me on twitter when I mention Ceph.. I did get info from an EMC rep - scaleio costs about 25,000 euro for less than 10TB of usable storage. its not a very compelling pricing model.

 

[PigLover]

ScaleIO is available on a "free and frictionless" basis, per their EULA, as long as you want it on an unsupported basis. Not really much different than how competing products like Ceph is available.

If you want support, however, I'm not sure Inktank/Redhat pricing for Ceph is much lower than what you see from EMC.

I guess with Ceph you have some level of "community" support available that might be better than what you'd see for ScaleIO. But really, its not much better. In both cases - if you are comfortable with no formal "support" its free. If you want a supported license they are very expensive.

 

[iptgeek]

My understanding is that its free to download to test / run in non production for unlimited time. If you want to run in production that is another thing all together.