What kind of disk do your use? SSD for journal? How fast is it without extra journal disk?
Ceph large file (images) resync
- Thread starter jinjer
- Start date
What kind of disk do your use? SSD for journal? How fast is it without extra journal disk?
I have 8 Sata HDD for OSDs. Journal is on the same OSD. No separate SSD. I was told in CEPH IRC that when large number of OSDs are used, it is a goo didea to put journal on same disk instead of SSD. I only have 8 OSDs now, but it is going to grow very soon with some addition of big data proxmox VM.
Last edited:
I originally setup my CEPH cluster with 16 OSDs with journals on the OSD disks. Writes were horrible. Dedicating one disk in each node for journal and having only 12 OSD performed better.
I am confident that an SSD journal would perform even better. Reading over CEPH mailing list archives I got the impression that a single SSD can act as a journal for a few OSDs but having too many journals on a single SSD hurts performance.
I am confident that an SSD journal would perform even better. Reading over CEPH mailing list archives I got the impression that a single SSD can act as a journal for a few OSDs but having too many journals on a single SSD hurts performance.
Are there any recommendations to the size of dedicated journal node?
How much data/traffic is generate while doing journaling?
How much data/traffic is generate while doing journaling?
Yes normal SATA disk with a partition for each journal.
Performance is not great but is better than having journals on the OSDs.
With the price of 16g and 32g SSD being so cheap I was thinking of using one ssd journal per OSD disk.
But I am still stuck with read speeds being limited, if I cannot fix this I cannot use CEPH.
Maybe that aio issue you suspect exists is also reducing CEPH read performance.
Performance is not great but is better than having journals on the OSDs.
With the price of 16g and 32g SSD being so cheap I was thinking of using one ssd journal per OSD disk.
But I am still stuck with read speeds being limited, if I cannot fix this I cannot use CEPH.
Maybe that aio issue you suspect exists is also reducing CEPH read performance.
It did not seem to me that the specs of SSD mattered that much when we are talking about large number of OSDs lets say 36 or more of them. Even with the fastest SSD out there performance will not get much benefit after certain point.
My personal preference though is Journal on OSDs even though it hurts performance a little. It eliminates extra layer of RAID for Journal SSDs. With Journal on OSD, if one hdd fails i only lose data for that one. If SSD/SSDs Journal fails, i lose entire host of OSDs and longer downtime.
Following a trail left by e100, i came across some articles where i found some tweaking that i can still do to the CEPH Cluster. Going to try them and see where my performance goes.
Proxmox VM seem to have slightly better performance when they are on CephFS. Somewhat promising. None of the CephFS VM crashed so far.
You get 180MB/s per VM, and that is the speed of a single disk - isn't that good enough?
Yep. Tried all Cache and finally settled down with writeback. That seem to give the best performance for RBD VMs.
Below is a comparison of the same VM on CephFS and RBD. I ran several benchmarks on each and had about the same numbers. Cache enabled.
VM on CephFS:
VM on RBD:
VM on CephFS:
VM on RBD:
I have not played around Infiniband yet. Does anybody know if i can use a 20Gbps Infiniband card with a 10Gbps Infiniband switch?
A 1Gbps Ethernet card works with lower bandwidth LAN Switch. Does the same logic applies to Infiniband?
Yes 20gbps any 10gbps work together, I have used a few 20g cards all my switches are 10g
You can connect two servers directly without a switch but you need to install opensm on one or both of the nodes.
Just wanted to confirm that increasing the number of OSDs does increase the performance of CEPH Cluster. I went from 6 OSDs to 10 OSDs and the increase of performance was immediate without changing anything else.