[SOLVED] Shared NVMe storage OR local NVMe Ceph?

[Razva]

Hello,

I'm starting a new cluster and I would apreciate your opinion on this. Should I use a central storage server plus compute machines (similar with OpenStack) or should I use local NVMe and create a Ceph cluster? Obviously everything will be connected via 10 Gbps.

My target is to create something that's fast and flexible at the same time.

Thank you,
Razvan

 

[ph0x]

You cannot really have both. At least not in the best quality. Ceph is flexible but not exactly fast (although fast enough in my opinion). Local storage is fast but not very flexible. As soon as you have to reboot the storage, everything stands.

 

[Razva]

You cannot really have both. At least not in the best quality. Ceph is flexible but not exactly fast (although fast enough in my opinion). Local storage is fast but not very flexible. As soon as you have to reboot the storage, everything stands.

I know, hence this thread. I'm planning to use this for a public cloud. What's your recommendation, and why?

 

[ph0x]

If you would only have one storage afterwards, I would go for Ceph. Otherwise you can never reboot the storage without a downtime of everything.
If you have a HA storage, I would go for iSCSI or NFS. Like in a real SAN.

 

[Razva]

If you would only have one storage afterwards, I would go for Ceph. Otherwise you can never reboot the storage without a downtime of everything.
If you have a HA storage, I would go for iSCSI or NFS. Like in a real SAN.

If you could choose between HA storage and Ceph, which would you choose, and why?

For the HA storage would you use Ceph, or something else?

 

[ph0x]

If the number of nodes isn't limited and you are concerned about performance, I would go for a separate setup of compute and storage nodes.
Unfortunately, I'm not really experienced in the SAN environment. But there are also other cluster file systems around. I'm a Ceph fan but that doesn't mean that there maybe are more suitable solutions.
If the number of nodes is limited, I would go for hyperconverged Ceph and try to optimize latency and bandwidth with appropriate hardware.

 

[Razva]

If the number of nodes is limited, I would go for hyperconverged Ceph and try to optimize latency and bandwidth with appropriate hardware.

What "appropriate hardware" are you referring to, except 10 Gbps NICs?

 

[ph0x]

10 Gbps is the lowest that you should go with Ceph. The higher, the better, so maybe you could think about 25 Gbps or even FC solutions, for the sake of low latency.
Then, a lot of RAM and CPU cores since Ceph can be hungry.
Ceph provides a hardware requirements document for such considerations.

As a reference, I run Ceph with three 8-core Xeon D SoC boards and 10 Gbps ethernet. It's reliable and totally okay in terms of performance for home use. Don't know if I would rent that out, though.

 

[bbgeek17]

Ceph is a solution that's well-integrated and supported by PVE. As Object storage that was adapted to serve block, it also comes with pretty significant baggage in terms of latency. As @ph0x has said - it's fast enough and functional for some environments. Is it the best from a performance standpoint? No.

As with most Distributed Storage solutions taking a node offline will affect performance and availability. Doing maintenance will require tight coordination and patience. Nevertheless, if you master it - it will work.

As far as I know, we are the only SAN solution that provides iSCSI block storage with a fully supported (by us) Proxmox driver. In addition to storage provisioning, the driver also gives you Snapshot functionality. The alternative solutions that are built into Proxmox are "ZFS over iSCSI" and "LVM on top of iSCSI". Neither one comes with HA, that's something you have to build yourself. ZFS allows snapshots, LVM does not for shared storage.

In summary: If you want to provide your customers with storage that responds in under 100 microseconds (i.e., NVME latencies) - going with the appropriate SAN would be the right way.

As for your network plans: if you can build your solution with at least 25Gbit that would also help you. At this point, 25Gbit is the new 10 in terms of price/economy.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Razva]

As far as I know, we are the only SAN solution that provides iSCSI block storage with a fully supported (by us) Proxmox driver. In addition to storage provisioning, the driver also gives you Snapshot functionality. The alternative solutions that are built into Proxmox are "ZFS over iSCSI" and "LVM on top of iSCSI". Neither one comes with HA, that's something you have to build yourself. ZFS allows snapshots, LVM does not for shared storage.

The only issue with this is that it's not open-source nor affordable. Or maybe I'm wrong?

 

[bbgeek17]

If having everything in your environment to be Open-Source is your hard requirement, than Blockbridge would not be a fit. The solution is not geared for Home/Lab environment, but rather towards Service Provides who intend to run their offering as a business. We definitely want our customers to be competitive in the market while offering the fastest storage to their end-users.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ADI91]

Hello everyone,

Sorry but not speaking english, this is a google translation.

To meet your need, personally, we use NFS on Synology HA NAS with 10Gbps network cards.

What are your performance needs? And for what purpose will the VMs be used?

NVMEs in local storage would be an unnecessary expense, an SSD in SATA will suffice because if the virtual hard disks are on the NAS no need to put NVME. The performance will not change.

After, to have the top in cheap performance, the easiest would be to have 2 NVME disks in RAID 1 with the VMs on the Host with a replication on a backup Host.

Advantages are:
- cheap
- the top in terms of performance (NVME SAMSUNG runs at around 3000 MBps in reading and 2500 MBps in writing).
- fault tolerance of NVME disks thanks to RAID 1
- fault tolerance of the clustered host with replication

The downside is that the replication can only be done every 5 mins minimum, so a loss of data of 5 minutes is possible in the event of a switch to another host