Ceph cache tier alternative

[timsergeeff]

Hi!

I’m planning to migrate from TrueNAS to Ceph because I want node-based redundancy. I was planning to use HDDs with an SSD cache tier on top to boost performance, since I work with heavy file sequences and want to saturate my 10 Gbit network. However, I found that the cache tier is deprecated, and I can’t find what the best practice is nowadays for building a fast storage cluster based on HDDs for cost efficiency while using SSDs for speed.

ChatGPT suggested creating two pools (one for SSDs and one for HDDs) and building a special service that would move files automatically based on last access date. I’m not sure this is the right suggestion because why would I need to create a separate program to do this? Is there a native way to achieve this? Or am I thinking about it incorrectly?

In my case, I’m planning to create a three-node setup and want to have several network disks (SMB, NFS) to use for containers and Windows machines for work. However, this is purely for organizational purposes — I don’t want to maintain separate network disks for SSDs and HDDs.

Any help would be appreciated

 

[ucholak]

search: Ceph DB/WAL on SSD
https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster#pve_ceph_osds

 

[timsergeeff]

search: Ceph DB/WAL on SSD
https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster#pve_ceph_osds

Thank for help!

I've found that but in terms of lack of practice with ceph i cant tell for my self is it fast enought? I mean this is just db and meta not actual files on ssds. Of cource i understand that i wiil not have pure ssd performance since i want hybrid solution but at least will it enought for saturating 10gbit connection with that? I think it will not because actual files will be at hdds as i see. And however if that is only one solution: how much ssd volume do i need is hdd for now is 80 tb raw?

 

[ucholak]

Thank for help!

I've found that but in terms of lack of practice with ceph i cant tell for my self is it fast enought? I mean this is just db and meta not actual files on ssds. Of cource i understand that i wiil not have pure ssd performance since i want hybrid solution but at least will it enought for saturating 10gbit connection with that? I think it will not because actual files will be at hdds as i

depends on type of operations/data, number and speed of hdds.
practically i only have full flash setups.

see. And however if that is only one solution: how much ssd volume do i need is hdd for now is 80 tb raw?

if i remember correctly: db 10% + wal 1%

 

[oz1cw7yymn]

Hi!

I'm the biggest proponent of ceph, it's really an amazing product and technology. Multi-node redundancy, self-healing and rebalancing is really fantastic.

Please note though that it has a very different performance profile than other storage solutions I've worked with. For individual clients, I'd say it's never going to be as fast as a SAN or NAS with the same storage hardware. But since it's multinode and for ceph aware clients, the aggregate performance across a large number of clients can easily surpass what you can squeeze out of a single point of failure "legacy" SAN or NAS.

But that requires ceph aware clients, using for example ceph native rbd or cephfs. That allows clients to receive data directly from all nodes rather than just one. But if you're using SMB or NFS, you will still need those single-point of failure gateways, narrowing the possible performance bandwidth down to those nodes with gateways.

So it all boils down to what is important in your use-case. If you want to use a low number of clients, if you have a low number of nodes, if you want to use NFS or SMB, it's going to be very hard to get maximum performance out of ceph. A rule of thumb in my head is consider (for single clients) a ceph cluster with just SATA SSDs as performant as a NAS/SAN with just HDDs, or a ceph cluster with nvmes like a NAS/SAN with just SSDs, essentially one generation back.

If you still want that sweet, sweet multi-node redundancy, put the DB/WAL on SSDs as recommended. Also consider making separate pools with different rules using HDDs and SSD, placing slow data (backups etc) on HDDs and boot disks on SSDs, for example. Those pools can also have different redundancy rules, including replication or erasure coding. There are a lot of configuration available and just throwing everything in one big bucket is not going to give you the results you want. That's just what it is to work with ceph. Which I'm still a huge proponent of!

 

[timsergeeff]

depends on type of operations/data, number and speed of hdds.

I use my storage for exr image seqences because i work in cgi industry (usualy 2-3 shots in parallel so this is like files of 10mb*150 per shot) and to run my containers whitch like my dnss websites, pipline intsuments etc.

if i remember correctly: db 10% + wal 1%

thanks will note that.

Please note though that it has a very different performance profile than other storage solutions I've worked with. For individual clients, I'd say it's never going to be as fast as a SAN or NAS with the same storage hardware. But since it's multinode and for ceph aware clients, the aggregate performance across a large number of clients can easily surpass what you can squeeze out of a single point of failure "legacy" SAN or NAS.

for me personaly i see that bandwith of maybe 5gbe per client is ok, and i planning maybe to 10 client+vms+containers.

So it all boils down to what is important in your use-case. If you want to use a low number of clients, if you have a low number of nodes, if you want to use NFS or SMB, it's going to be very hard to get maximum performance out of ceph. A rule of thumb in my head is consider (for single clients) a ceph cluster with just SATA SSDs as performant as a NAS/SAN with just HDDs, or a ceph cluster with nvmes like a NAS/SAN with just SSDs, essentially one generation back.

I have a little solution right there! I'm planning to do ceph+ cephfs (for smb and nfs)+ kubernetes for containers on each node. So i will have smb and nfs gateways in a conteiners and will have several replicas of gateways (i think 3 so each node will have its own gateway) so in theory i'll be able to satutare all bandwith per node. Am I rigth? This maybe i missunderstood the concept of smb sharing for ceph/cephfs, pleace help)

using HDDs and SSD, placing slow data (backups etc) on HDDs and boot disks on SSDs, for example. Those pools can also have different redundancy rules,

this is the qestion! May i able to combine ssds and hdds in a single (for user view) disk and move hot data and cold data automaticaly? GPT suggested that i need to write my own service to do in on schedule but i feel like this is strange. But maybe not strange since nowadays guys can have all nvme storage so speed of a drives is not a problem.

I understand about wal/db but it means that i wiil need only aproximately 9tb of nvme for db/wal but I still wonder if it any good comparing to storing files directy on ssds. Thats why I'm trying to combine ssds with hdds- cost and hardware i've alredy have)