Ceph hardware

[casalicomputers]

Hello,
could you share your working and tested configuration (DELL anyone?) for use with Ceph?
I need a 3-nodes general purpose configuration which offers good performance (~10-20 vms).
Details are welcome (hdd/ssd models, benchmarks, problems encountered.... and so on).

In addition I have a few questions:
1) Is 10GbE really mandatory for storage network? What about 4x1Gb NIC bonding in round-robin with VLAN separation or maybe InfiniBand? Any experience to share?
2) Is it safe to have OSD journal on external SSD? What if it suddenly breaks? Will I lose all the backed OSDs as well?
3) Is Ceph like ZFS about RAID controllers? Does it need to manage disks by itself or it can make proper use of the controller cache? Could it be a good practice to mirror OSDs or SSDs used for journaling ?

Thanks,
M.

 

[Melanxolik]

How many do you want have virtual machines?
What about configurations this virtual machines?

I have bad experience with 200 virtual machines and 30 osd drives. Now we are using only ssd drives on ceph cluster.

 

[casalicomputers]

hi melanxolik,
thanks for sharing.

I don't have a specific workload, but just need to get an idea on how to dimensionate a pve + ceph cluster in a scalable way.

A common environment could be ~10 vms (domain controllers, smb file servers, mail servers and database servers (oracle, mysql), web servers, etc...).

how many osd ssd have you got and how many servers? how are they performing?

 

[Melanxolik]

we have over 7 nodes with:
Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
10gbit/s network adapter
at this moment we are have 10ssd for 800gb by intel, and running over 200 customers virtual servers

All ssd connected to ceph cluster and we have some trouble, when cluster rebuild.
First if you have a lot of drives, you must connect them to raid controller, after that you should create osd over raid array.
Why you ask me?
Because if ceph cluster start rebuilding, performance very strong degraded, even you have only ssd in osd... That's why you must use raid controller with raid storage.

We did test in customers vm, and have next results: by disk 10-15k random write, 12k random read, 1k liner write.
300Mb/s write and read from virtual drive.


That's small result my experience with proxmox and ceph.

other my cluster has 15nodes and external storage system, very terrible performance. Proxmox+ceph on every node more better and more dangers for saving data.

 

[markusd]

First if you have a lot of drives, you must connect them to raid controller, after that you should create osd over raid array.
Why you ask me?
Because if ceph cluster start rebuilding, performance very strong degraded, even you have only ssd in osd... That's why you must use raid controller with raid storage.

You have a raid 1 / 5 / 10 and use this as osd?
i think, this is not the way ceph wants to work and is not recommended.
There are severeal tunings make such things like rebuilding good work

Markus

 

[Melanxolik]

No, we only think about introduceing to us project.
Yes, I know, other way ceph you can change rebuilding parameters, but I think when you have a lot of drives, your storage always will rebuild.
But, If you use raid controller and array storages it will be next situation with rebuilding system

 

[casalicomputers]

So it seems that the overall Ceph performance is not that great as one may think...

I mean: can I compare eg. 3-node ceph cluster with 2 osd (15k SAS 600gb) + 1 jornaling ssd per each node against a dedicated FC/iSCSI SAN with the same amount and kind of disks? I think the SAN would win here in term of performance... you may argue that the SAN could be a SPOF but nowadays they got redundant PSUs and controllers......

What do you think?

 

[udo]

...
All ssd connected to ceph cluster and we have some trouble, when cluster rebuild.
First if you have a lot of drives, you must connect them to raid controller, after that you should create osd over raid array.
Why you ask me?
Because if ceph cluster start rebuilding, performance very strong degraded, even you have only ssd in osd... That's why you must use raid controller with raid storage.

Hi Melanxolik,
that's not completly right. Ceph normaly should not run on raided OSDs... To avoid (big) performance drop during rebuild, you should tune your ceph.conf (the default is for safety - fast rebuild). Like this:

osd max backfills = 1
osd recovery max active = 1
osd_disk_threads = 1 #disk threads, which are used to perform background disk intensive OSD operations such as scrubbing
osd_disk_thread_ioprio_class  = idle
osd_disk_thread_ioprio_priority = 7

If one drive fails and you want an faster rebuild, you can modify the values on the fly:

ceph tell osd.* injectargs '--osd_max_backfills 5'
ceph tell osd.* injectargs '--osd_recovery_max_active 5'

Udo

 

[KHosting]

we have over 7 nodes with:
Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
10gbit/s network adapter
at this moment we are have 10ssd for 800gb by intel, and running over 200 customers virtual servers

All ssd connected to ceph cluster and we have some trouble, when cluster rebuild.
First if you have a lot of drives, you must connect them to raid controller, after that you should create osd over raid array.
Why you ask me?
Because if ceph cluster start rebuilding, performance very strong degraded, even you have only ssd in osd... That's why you must use raid controller with raid storage.

We did test in customers vm, and have next results: by disk 10-15k random write, 12k random read, 1k liner write.
300Mb/s write and read from virtual drive.


That's small result my experience with proxmox and ceph.

other my cluster has 15nodes and external storage system, very terrible performance. Proxmox+ceph on every node more better and more dangers for saving data.

Please what type of SSD are you using for Journals and OSD?

 

[robhost]

Please what type of SSD are you using for Journals and OSD?

You need datacenter SSDs, here is a good start and a listing of some devices:
https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/

 

[KHosting]

Thanks @robhost. I am aware of that link.

I was actually asking @Melanxolik what SSD make he used for his specific results. It is interesting that he attained 300Mb/s write and read from virtual drive.

 

[Melanxolik]

In first time we using:
Device Model: INTEL SSDSC2BB800G6

but now, we try use two ssd from samsung:
Device Model: SAMSUNG MZ7LM960HCHP-00005

but, I think intel more better reliable than samsung.

 

[KHosting]

Thanks @Melanxolik. Am guessing that's for your OSD?

 

[Melanxolik]

yes