New All flash Ceph Cluster

[adamb]

We are looking to build out a all flash ceph cluster and I wanted to get some opinions. We would use proxmox with enterprise subscriptions. This cluster would be dedicated to ceph with no VM's.

Leaning towards a 8 node setup using the supermicro 2113S-WTRT as the chassis with a AMD epyc cpu and 256G of ram.

We are thinking of using the Micron 5210 7.68TB SSD for OSD's along with the Micron 9300 MAX 3.2TB for the OSD journal. Right now the plan is to have 1x 9300 max per 6 OSD's. Does anyone have any experience with these drives? Or possibly a better configuration?

Most of it is based on this white paper.

https://www.redhat.com/en/resources/st-micron-ceph-storage-amd-brief-f7698-201903

 

[adamb]

I think my biggest concern is the Micron 5210 endurance. Open to other brands, also looking at the Micron 5300 Pro which has a much better DWPD.

Our writes will be primarily 4k-8k and our read/write work load should be in the 50/50 range.

Doesn't really look like many others have anything in the 7.68TB range.

 

[Klaus Steinberger]

I do not understand why you want a separate Journal for a full flash deployment. if your journal SSD fails you loose immediately all 6 OSD's depending on it. Also the write load on the journal will be 6 times the OSD's

compare: 0.8 DWPD to 3 DWDP -> so if you are really hitting the DWDP Limit, your journal will be dying first

and 0.8 DWPD on a 7.68 TB means you have to write 6.14 Tbyte per day in each of them to hit the limit

If your write rates are really that high, then better invest in SSD's with higher DWDP Rates and drop the journal

 

[adamb]

I do not understand why you want a separate Journal for a full flash deployment. if your journal SSD fails you loose immediately all 6 OSD's depending on it. Also the write load on the journal will be 6 times the OSD's

compare: 0.8 DWPD to 3 DWDP -> so if you are really hitting the DWDP Limit, your journal will be dying first

and 0.8 DWPD on a 7.68 TB means you have to write 6.14 Tbyte per day in each of them to hit the limit

If your write rates are really that high, then better invest in SSD's with higher DWDP Rates and drop the journal

I am seeing quite a few examples out there were they are using cheaper SATA SSD's for OSD and high end NVMe for journal/WAL/DB. Micron specially has a great example of this setup and looks really solid (Link is in my OP)

I have changed my config a bit more. I am now going to use 1x Micron 7300 MAX 1.6TB NVMe for 2x OSD's.

All our writes are small IO 4k-8k, but we also read heavily. Its proprietary but I would consider it very similar to most database engines.

 

[sg90]

I am seeing quite a few examples out there were they are using cheaper SATA SSD's for OSD and high end NVMe for journal/WAL/DB. Micron specially has a great example of this setup and looks really solid (Link is in my OP)

I have changed my config a bit more. I am now going to use 1x Micron 7300 MAX 1.6TB NVMe for 2x OSD's.

All our writes are small IO 4k-8k, but we also read heavily. Its proprietary but I would consider it very similar to most database engines.

Do the MAX not come in smaller sizes? 1.6TB is massively overkilled for WAL/DB especially for 2 OSD's.

 

[adamb]

Do the MAX not come in smaller sizes? 1.6TB is massively overkilled for WAL/DB especially for 2 OSD's.

They do have 800G versions as well. What is typically the size ratio for the WAL/DB device?

 

[sg90]

They do have 800G versions as well. What is typically the size ratio for the WAL/DB device?

Curently it goes 3/30/300GB max, but depends on the size of OSD, so 2 OSD should be more than fine.

 

[adamb]

Curently it goes 3/30/300GB max, but depends on the size of OSD, so 2 OSD should be more than fine.

From what I am reading it should be 4% of the block device size. So if I am using 7.68TB drives that would be roughly 307G. Ill probably shoot for 2x 350G partitions for the DB of each OSD and the rest can be for WAL. The 800G drives definitely make a bit more sense. Appreciate the input!