ceph: 5 nodes with 16 drives vs 10 nodes with 8 drives

[PabloUserT38]

Hi,

I'm designing a ceph cluster for our VFX studio. We have about 32 artists seats and I need high sequential read and write speeds, not so much IOPS. I will use whatever it takes to put the best possible hardware inside each node, but I have to decide now if I go with many nodes with fewer disks or viceversa, for the same number of total drives.

Any advice will be greatly appreciated.

 

[alexskysilk]

some specifics would be helpful.

1. I'm assuming you intend to use cephfs (as opposed to rbd over iscsi.)
2. what is the minimum required bandwidth per connection? (what bitrate are you working with?)
3. what is the interconnect to the clients?
4. do you have access to native cephfs clients, or will you need a network file system (eg nfs, cifs)?
5. what is the largest number of SIMULTANEOUS users?

 

[PabloUserT38]

Thank you for your reply. Here are the answers to your questions:

1. I'm assuming you intend to use cephfs (as opposed to rbd over iscsi.)

Actually, I'd love to use a block device, but I'm still new to ceph and don't know if it's possible to share it over iSCSI with all the workstations (like you can't share a single iSCSI drive with many workstations). Let's assume the fastest available path for sharing.

2. what is the minimum required bandwidth per connection? (what bitrate are you working with?)

Only a handful of workstations require really high bandwitch, in the realm of 1 GB/s simultaneously (as of today; but the plan is going up to 4GB/s in the future). The rest are fine with around 500 MB/s or even less. It's a rare occurrence that everyone is pulling data at the exact same time from the server we currently use (a Symply FibreChannel Stornext 100 TB appliance), but of course the idea is to improve the speeds with time.

3. what is the interconnect to the clients?

10GbE over fiber. Eventually we can go to 25GbE or even start with it right away. We might even put 100GbE in some workstations in the future. We're buying a 32 port 100GbE switch to manage the whole thing.

4. do you have access to native cephfs clients, or will you need a network file system (eg nfs, cifs)?

I can put whatever I want in every workstation. As I said, I'd like to use block access, but anything will do as long as the speed is maintained.

5. what is the largest number of SIMULTANEOUS users?

We currently have 4 workstations requiring 1GB/s concurrently. I'd like to reach up to 4GB/s for these at some point. Another 12 workstations need at least 500 MB/s. The rest will be fine with whatever speed they get. I don't know if it's possible to segmentate the system like this. To sum it up, 16 workstations will be using the server cluster concurrently most of the time, with occasional access from the other 16.

EDIT: Just FYI, most of the workstations are Windows based, but a couple are using Linux. In the "low speed tier" we have a few Macs too. Also, I'm willing to put whatever NVMe cache is needed in every node of the cluster.

 

[alexskysilk]

Actually, I'd love to use a block device, but I'm still new to ceph and don't know if it's possible to share it over iSCSI with all the workstations (like you can't share a single iSCSI drive with many workstations). Let's assume the fastest available path for sharing.

You can it would would exactly like it does now for your fcal/stornext solution. logically speaking, there is no difference between a fiber channel block target and an iscsi target.

I can put whatever I want in every workstation. As I said, I'd like to use block access, but anything will do as long as the speed is maintained.

hold your horses hoss. cephfs is only native to linux. are your clients linux workstations? otherwise, block will work- especially since you already have a san solution in place. you may want to run this idea by your stornext sales rep as I dont know how their licensing will apply.

As this discussion is really not proxmox related, I'd be happy to discuss further in DM.

 

[PabloUserT38]

How is this "really not proxmox related"? I'm trying to build a Proxmox cluster! My only question is if it's better for my use case to go with 5x16 or 10x8 (node x drives). Not that I don't want to DM you, but I think this would help not just me, but anyone interested by such an essential question when building a Proxmox cluster with Ceph.

 

[jdancer]

Best practice for Ceph is lots of nodes with fewer OSDs versus fewer nodes with more OSDs.

This is to spread out the I/O load.

In order to avoid split-brain issues and have quorum, you'll want an odd number of nodes.

 

[alexskysilk]

Proxmox is a hypervisor product. storage features are secondary to its function. you wouldnt use proxmox for your intended usecase as it lacks any tools for that.

 

[PabloUserT38]

Best practice for Ceph is lots of nodes with fewer OSDs versus fewer nodes with more OSDs.

This is to spread out the I/O load.

Thank you for an answer to the point.

 

[PabloUserT38]

You can it would would exactly like it does now for your fcal/stornext solution. logically speaking, there is no difference between a fiber channel block target and an iscsi target.

Actually there is. They are completely different protocols and their requirements differ. Knowing them, using a single drive as an iSCSI target is a stupid proposition, so I should have said "you shouldn't" instead of "you can't".

cephfs is only native to linux

Ceph does provide a Windows client for block access, so there's no need to use iSCSI. This much I learned on the internet in the last few days. (Here: https://cloudbase.it/ceph-on-windows-part-1/). Unfotunately MacOS is not supported yet.

I really don't know why you made all those questions in the first place if you think Ceph issues are out of place in a Proxmox forum. If this thread had been my only contact with Proxmox I would run away as fast as possible.