New cluster - storage solution

dignus

Renowned Member
Feb 12, 2009
154
10
83
For a new cluster we are investigating storage solutions. Cluster(s) will be 20 nodes in size, so nothing local and had to do failover and such.
Current clusters running on ultra-low-latency nvme, so ceph, NFS and zfs over iscsi ruled out.

Been speaking to various suppliers this week. Some of them offer a solution not natively integrating with proxmox, some of them. Well.. one does: blockbridge.

Does anyone have experience running on blockbridge, since they seem to be the only guys to have a native integration with promox?

@bbgeek17 you are not allowed to comment, although I do respect your dozens of helpful comments on various storage related topics :)
 
Last edited:
@dignus - we have been a Blockbridge customer since 2016 after having tried just about every solution imaginable with our various hypervisor platforms (VMware, KVM on OpenStack, and now KVM on Proxmox). Every solution had their issues, except for Blockbridge. We have continued using Blockbridge for all of these use cases in many production systems across many Blockbridge clusters, and continue to expand (I think we're up to 5 BB clusters now?).

Reliability is THE most important part of a storage system, with stability, support, maintenance, and performance/throughput/latency coming in close seconds. I can honestly say that Blockbridge checks the boxes on all of these, which is honestly a pretty tough ask, and something no other vendor has been able to do (we have tried at least 5 other vendors in our history, some with horrific outcomes).

We have been using Proxmox with the NVMe-oF protocol with their driver in Proxmox to one of our Blockbridge clusters (two "complexes) with 20 servers so far, and haven't had any issues at all. Extensive reliability and performance testing has been done since we are incredibly sensitive to any issues whatsoever. Blockbridge has tuned the system to work exactly the way we want it, down to the Mellanox NIC settings, while understanding the switch configuration we have.

Troubleshooting is always tough, and Blockbridge has always been willing to assist more than we expect. For example, we have seen the case where multiple switches cascaded between servers and the Blockbridge system produced a noticeable latency difference - compared to servers and Blockbridge systems connected to the same switch. Blockbridge actually performed some latency measures for us to understand just how important port-to-port latency on switches can be! :) For example, Mellanox versus FS switches have a pretty significant port-to-port latency difference (Mellanox being half that of the cheaper FS switches), causing QD1 IOPS to be halved.

I would be happy to answer any questions about our environment (or at least what I can say without violating our internal privacy policies).
 
  • Like
Reactions: davemcl
@dignus We have been a Blockbridge customer for just about a year now and are in the process of installing our second set of storage units in another datacenter. So far things have worked flawlessly. The folks at Blockbridge are great to work with in terms getting setup and long term support. Thankfully we've only had one instance to reach out to them for support and they were extremely helpful even when the issue turned out to be a kernel bug and not an issue related to storage at all. The Blockbridge servers also phone home and are monitored by them and if there are any problems, they are available 24x7 to address them. We're a small organization and don't have a dedicated storage administrator so having Blockbridge there to backstop us is helpful.

Because their driver directly integrates Blockbridge storage with Proxmox, we've rarely had to login to the Blockbridge units for anything. The performance and reliability has been outstanding. For us the solution has been set it and forget it.

I am also happy to answer any questions you might have relating to our environment or our Blockbridge setup.
 
@dignus Our team had been using Ceph before, and we were constantly dealing with slow read and write speeds. But since we switched to Blockbridge, everything has been running so much faster delivering great IOPS performance as well.
What I really appreciate about Blockbridge is its flexibility. We're able to customize our storage setup to fit our specific needs, and the system can easily adapt as our workloads change, it also supports live cluster to cluster migration which proxmox has introduced recently. Plus, the user interface is clean and easy to navigate, so managing our storage environment is a breeze.
Another big plus is the Blockbridge team themselves. They're incredibly knowledgeable and responsive, always willing to help us troubleshoot any issues that come up. It's clear that they really care about their customers and are dedicated to providing the best possible service.

I would say go for it.
 
Thanks for the replies! One thing I do wonder is how the compression performs in a real life scenario and what the compression gains are.

Reason for asking: currently we have a storage solution that supports trim and thus we have only 65% of allocated storage in use in the storage system. I understand BlockBridge doesn’t support trim, but compression might help in this case.
 
Thanks for the replies! One thing I do wonder is how the compression performs in a real life scenario and what the compression gains are.

Reason for asking: currently we have a storage solution that supports trim and thus we have only 65% of allocated storage in use in the storage system. I understand BlockBridge doesn’t support trim, but compression might help in this case.
Hi,
You are right on the part, the trim is not supported. But, I rather felt it was never required because normally what trim does is it clears off white space and gives it for reuse. In BB, their thin provisioning only stores the blocks with data in it. If its a whitespace, it simply does not consider it used. Hence, you don’t need to trim anymore. In fact we used to trim each of our VMs in ceph but now not a single trim is required. Also, we have not tested compression yet but it depends on what type of data are you storing and accordingly you might be able to save around 20% more i guess.
 
I tried CEPH on the proxmox machines, it was not good. Since I have a dedicated blockbridge storage, there are no problems at all. Low latency, high performance.
They have scheme with only 2 node , full replication , one VM voter for quorum which is perfect for me.
I use it with lvm over iscsi.
 
no offence but these Serversare no high performance. they wrote on the website to one storagebox 3 million IOPs for round about 20 Drives. this is the cheapest enterprise hardware like netapp, dell, ibm etc. buy cheap shit and charge a lot. how i know ? i built a server myself, with the goal only one bottleneck which sets the overall speed. my harddrive, and only ONE of them, hat 10 million IOPs. that is high performance.

dont get me wrong - the complete equiptment is highly adusted to proxmox and works very well, but far away from real high performace
 
Last edited:
@dignus

Another +100 for Blockbridge. Absolutely hands down the best storage solution that we have come across for Proxmox. It just works - really nicely integrated with PVE and the blockbridge team are really excellent to deal with - nothing is ever too much trouble. Really happy to chat off forum, PM me if you think that would be helpful.
 
Last edited:
Hi @pille99 ,
No offense taken. 10 million IOPS is fantastic! That would require 16+ of the top-end, latest generation, Gen-4 micron 9400s running at maximum load. You have a supercomputer, for sure!

If suitable for datacenter use, we'll look into adding your configuration to our list. Please DM me some information (I want to avoid taking over the thread).
  • Are you measuring sustained write IOPS without drive caching?
  • Is the storage fully redundant (i.e., supports drive loss)?
  • Does the configuration support power loss without data loss?
  • What drives are you using?
And to clarify, we don't sell hardware. The platforms on our site are configurations that we recommend for performance, reliability, and global accessibility of replacement parts. Customers are free to choose and supply their hardware.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: dignus
i built a server myself, with the goal only one bottleneck which sets the overall speed. my harddrive, and only ONE of them, hat 10 million IOPs. that is high performance.
It is POSSIBLE to do what you're after on your own.

If you dont have any delivery deadlines, can solve all problems on your own in a timely manner, have reliable and tested failover/failback, and have the budget to replace/wait for all the missteps during design and implementation. In other words, if your time has no value and there's no real cost to downtime.

the reason commercial solutions costs as much as they do (well, discounting sales overhead) is because the vendor takes all that away from you and onto themselves. Folks who run storage for real can't afford to spend their entire professional life fixing holes and putting out fires in production- also, its very useful to blame the vendor when shit goes sideways and your job isnt in jeopardy ;)
 
No offense taken. 10 million IOPS is fantastic! That would require 16+ of the top-end, latest generation, Gen-4 micron 9400s running at maximum load. You have a supercomputer, for sure!

If suitable for datacenter use, we'll look into adding your configuration to our list. Please DM me some information (I want to avoid taking over the thread).

it was the goal, to create a real high performance Server. i think i archived it. but the main mission was: a server (in this case a cluster with 10 hardware servers) which doesnt need an aircondition. 2 years ago, i finished a test, 24 hours on 23 degrees (running only 5 servers). from the numbers my configuration is multiplied 11 times faster as a 2.5 mil Netapp config (found a PDF file on netapp). in my tests (i used esx with vsan) copy a 5 gb file was almost not shown up anymore - maybe 1 or 2 secs (windows). i also use a different hardware configuration and not server, network storage. the project is frozen because i need an investor and its difficult to find (i am not a sales person).
but i dont want to flame the threat. btw: a proper configuration contains multiple parts, the storage is just one.
dont get me wront - the mentioned storage is, what the comments confirmed, highly adjusted to proxmox, which makes it a good product.
 
Last edited:
What is the ballpark cost of the Blockbridge software licensing on its own? Or does one have to invest in a BlockBridge server of some sort?
 
What is the ballpark cost of the Blockbridge software licensing on its own? Or does one have to invest in a BlockBridge server of some sort?
Actually, the product is software and you purchase your own hardware (through them). Their supported platforms are either Dell or Supermicro hardware https://www.blockbridge.com/platforms/ and then you license their software.

We purchased a couple systems through Blockbridge. They will spec out the systems based on your requirements, order the equipment for you (cheaper than you probably can), configure and test it then ship it off to you to rack and stack.

Their Blockbridge driver is loaded onto your hypervisors and once connected into the API, allows you to configure your storage without leaving the Proxmox GUI. You can of course do much more logging into the management interface but we have founds ourselves rarely doing so.

They license the software based on TB of storage in use and it includes support and management. I'd reach out to them to chat if you are interested. They've been really good to deal with and easy to talk to.
 
  • Like
Reactions: davemcl
ejmerkel was faster than me and said most everything I was going to say. :)

A couple extra notes: The storage appears just like any other storage device in Proxmox. The configuration is done via the /etc/pve/storage.cfg file, just like any other storage, where you indicate an authentication key, protocol to use, etc.

We use both the Web UI and the CLI where appropriate, but both can do the same operations in most cases.

We have both SuperMicro and Dell equipment, although the Dell equipment is typically superior in terms of Engineering as well as remote management features, but expect to pay quite a bit more for Dell. You are probably going to end up with 2 x 10-bay or 12-bay servers (front-loading) in a 1U form factor, or 2 x 24-bay servers in a 2U form factor. You also need another machine for a witness/vote node so Blockbridge can maintain a quorum. We use VMs for vote nodes in a highly-available cluster.

Blockbridge supports multiple "complexes" per server, which is essentially a full instance of the Blockbridge software that is dedicated to a given set of SSDs. There is a minimum TB quantity per complex you must pay for, so if you want multiple complexes, you need to be sure you need the amount of storage required. A very dense system that has insanely high performance can be achieved with multiple complexes.

Hope that helps!
 
  • Like
Reactions: davemcl
So you basically run blockbridge on any x86 Server with nvme/ssd drives and connect the server simply over iscsi to Proxmox?
Or do they use their "own" special protocol similar to iscsi?

Another question comes into my mind, everyone is talking about the great performance, compared to any other solution, but in my findings i found iscsi a lot slower as fcoe in terms of latency.
But that was some time ago tho, fcoe is probably dead by now xD

And a screenshot of the Proxmox blockbridge part of the GUI would be really appropriated, tho you can mask all stuff that's not meant for public :)

First time here that i hear blockbridge, it's probably interesting for my company either.
Just license per TB of Storage makes me a bit of headaches, because that makes only sense for really fast nvme Storage, not so much for big hdd Storage pools.
Im just saying, because using 2 different Storage types let's say 1x nvme blockbridge and 1x iscsi freenas, doesn't look that beautiful in the eyes of an admin :)

Thanks very much for this topic, very interesting informations!
 
> So you basically run blockbridge on any x86 Server with nvme/ssd drives and connect the server simply over iscsi to Proxmox?

Blockbridge performs the install on pre-qualified systems due to the amount of tuning they do to get the absolute best performance. You can imagine the differences between servers in terms of processor architecture, number of chiplets, number of cores per chiplet, bus speed, dual veruss single processor, as well as NIC support. They have tunings for Mellanox NICs, for example, but have done some tunings for Intel as well. It makes a huge difference in the latency when everything is set correctly for the given workload.


> Or do they use their "own" special protocol similar to iscsi?

Thankfully, there are no proprietary protocols, custom kernel drivers, or specialized networks (i.e., FC, RDMA). They support iSCSI/TCP and NVMe/TCP. Having native kernel support simplifies maintenance and PVE upgrades. Both protocols are speedy. I run both. NVMe/TCP has lower latency.


> Another question comes into my mind, everyone is talking about the great performance, compared to any other solution, but in my findings i found iscsi a lot slower as fcoe in terms of latency.
> But that was some time ago tho, fcoe is probably dead by now xD

Their iSCSI implementation is quite fast, so I definitely wouldn't qualify it as "slow", but NVMe/TCP is best for latency. I've always been a big fiber channel fan, and we still have a bunch of fiber channel switches running in our environment with fiber channel HBAs. It just works and runs forever with super low latency, but I never did this with FCoE. In any case, you are right that fiber channel is essentially dead. :( NVMe-oF is the latest low-latency technology that has so many benefits from large queue counts to low latency command execution that it is hard to beat.


> And a screenshot of the Proxmox blockbridge part of the GUI would be really appropriated, tho you can mask all stuff that's not meant for public :)

There are a lot of pieces to the GUI. Note that they also have a CLI that can perform all of the functions too. I would suggest contacting Blockbridge for a demo if you wanted to see the GUI, but it would also give you an opportunity to meet the team, discuss your workloads and environment, and get first-hand responses from the company.


> First time here that i hear blockbridge, it's probably interesting for my company either.

They have been around for a long time - but have been in somewhat of a niche market since everyone wants everything for free - and chooses Ceph (which is dog slow). You get what you pay for, unfortunately.


> Just license per TB of Storage makes me a bit of headaches, because that makes only sense for really fast nvme Storage, not so much for big hdd Storage pools.

Blockbridge is a flash-only system, so you can't use HDDs with it. We use Ceph for HDD storage - which is what it is good at (not SSDs). For object storage, Ceph isn't great in terms of efficiency (neither performance nor storage requirements), so we have been looking around for alternatives, even if we have to go with a commercial solution.


> Im just saying, because using 2 different Storage types let's say 1x nvme blockbridge and 1x iscsi freenas, doesn't look that beautiful in the eyes of an admin :)

True, and understandable, but the two storage types (SSD and HDD) are very different beasts. I wouldn't suggest iSCSI for hard drive storage - I would use Ceph with RBD or if you don't have a huge environment, TrueNAS (or FreeNAS). We have had issues with legacy equipment working with TrueNAS, so had to use older FreeNAS virtual appliances. It just sits there and works fine, but you don't want to manage 100's of TiBs with it.
 
> So you basically run blockbridge on any x86 Server with nvme/ssd drives and connect the server simply over iscsi to Proxmox?

Blockbridge performs the install on pre-qualified systems due to the amount of tuning they do to get the absolute best performance. You can imagine the differences between servers in terms of processor architecture, number of chiplets, number of cores per chiplet, bus speed, dual veruss single processor, as well as NIC support. They have tunings for Mellanox NICs, for example, but have done some tunings for Intel as well. It makes a huge difference in the latency when everything is set correctly for the given workload.


> Or do they use their "own" special protocol similar to iscsi?

Thankfully, there are no proprietary protocols, custom kernel drivers, or specialized networks (i.e., FC, RDMA). They support iSCSI/TCP and NVMe/TCP. Having native kernel support simplifies maintenance and PVE upgrades. Both protocols are speedy. I run both. NVMe/TCP has lower latency.


> Another question comes into my mind, everyone is talking about the great performance, compared to any other solution, but in my findings i found iscsi a lot slower as fcoe in terms of latency.
> But that was some time ago tho, fcoe is probably dead by now xD

Their iSCSI implementation is quite fast, so I definitely wouldn't qualify it as "slow", but NVMe/TCP is best for latency. I've always been a big fiber channel fan, and we still have a bunch of fiber channel switches running in our environment with fiber channel HBAs. It just works and runs forever with super low latency, but I never did this with FCoE. In any case, you are right that fiber channel is essentially dead. :( NVMe-oF is the latest low-latency technology that has so many benefits from large queue counts to low latency command execution that it is hard to beat.


> And a screenshot of the Proxmox blockbridge part of the GUI would be really appropriated, tho you can mask all stuff that's not meant for public :)

There are a lot of pieces to the GUI. Note that they also have a CLI that can perform all of the functions too. I would suggest contacting Blockbridge for a demo if you wanted to see the GUI, but it would also give you an opportunity to meet the team, discuss your workloads and environment, and get first-hand responses from the company.


> First time here that i hear blockbridge, it's probably interesting for my company either.

They have been around for a long time - but have been in somewhat of a niche market since everyone wants everything for free - and chooses Ceph (which is dog slow). You get what you pay for, unfortunately.


> Just license per TB of Storage makes me a bit of headaches, because that makes only sense for really fast nvme Storage, not so much for big hdd Storage pools.

Blockbridge is a flash-only system, so you can't use HDDs with it. We use Ceph for HDD storage - which is what it is good at (not SSDs). For object storage, Ceph isn't great in terms of efficiency (neither performance nor storage requirements), so we have been looking around for alternatives, even if we have to go with a commercial solution.


> Im just saying, because using 2 different Storage types let's say 1x nvme blockbridge and 1x iscsi freenas, doesn't look that beautiful in the eyes of an admin :)

True, and understandable, but the two storage types (SSD and HDD) are very different beasts. I wouldn't suggest iSCSI for hard drive storage - I would use Ceph with RBD or if you don't have a huge environment, TrueNAS (or FreeNAS). We have had issues with legacy equipment working with TrueNAS, so had to use older FreeNAS virtual appliances. It just sits there and works fine, but you don't want to manage 100's of TiBs with it.
Hey @nightowl, thank you a lot for answering all my questions very precisely and satisfying.

I will get in contact with blockbridge, for more detailed info about pricing and hardware.

Thank you again!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!