ceph question

[mada]

Hello,

Planing to setup small ceph but i do care much for i/o speed

my setup will be following

3 x U2
72GB RAM
SSD SM863 for Proxmox installation
3 or 4 x 4tb enterprise drives. planned to add up to 5 of them later
2 x 120GB MyDigitalSSD BPX 80mm (2280) M.2 PCI as journal
Dual 10 Gbit Intel X540-T2
10 Gbit Layer L2 Switch

Do missing something? is that better using 1 or 2 PCI as Journal?

 

[udo]

Hi,
are you sure, that your journal device is the right choice?

Have you do an test? See also here: https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/

Udo

 

[alexskysilk]

observation 1: 7200 RPM spinners are never going to be "fast" but it really does depend on what you're trying to accomplish. Why are you spec'ing hardware before defining your workload?

 

[mada]

Hi,
are you sure, that your journal device is the right choice?

Have you do an test? See also here: https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/

Udo

So the MyDigitalSSD BPX bad choice? Samsung SSD 950 PRO cost around $200 or more per piece

 

[mada]

observation 1: 7200 RPM spinners are never going to be "fast" but it really does depend on what you're trying to accomplish. Why are you spec'ing hardware before defining your workload?

KVM Storage my reason to use the SATA enterprise Drives that i need huge of amount of space which SSD can't offer for cheap price.

 

[udo]

So the MyDigitalSSD BPX bad choice? Samsung SSD 950 PRO cost around $200 or more per piece

Hi,
I would prefer enterpris SSD as journal - otherwise the chance to buy the journal-disks twice is not low.

Udo

 

[mada]

Hi,
I would prefer enterpris SSD as journal - otherwise the chance to buy the journal-disks twice is not low.

Udo

So 1 x Samsung 960 PRO Series - 512GB PCIe NVMe would be better than 2 of those?

i also see SanDisk FS1-004-320-CS-0001 Fusion IO Drive 320GB ?

 

[udo]

So 1 x Samsung 960 PRO Series - 512GB PCIe NVMe would be better than 2 of those?

i also see SanDisk FS1-004-320-CS-0001 Fusion IO Drive 320GB ?

Hi,
960 PRO is named pro - but not classified as datacenter ssd... but 1200 TBW sounds not bad.
Depends on your workload (the samsung DC SSD SM825 has 7000 TBW).

Udo

 

[mada]

Hi,
960 PRO is named pro - but not classified as datacenter ssd... but 1200 TBW sounds not bad.
Depends on your workload (the samsung DC SSD SM825 has 7000 TBW).

Udo

So from what i understand that the journal will works as SSD cache for my SATA OSD so i need fasted i can get and with more enterprise version?

The SM825 just SATA is there is better as PCI instead? as Journal

and what is better using LSI 9210-8i HBA to connect all 8 bays or just using Motherboard ports?

 

[mada]

Any help would be appreciate it

if possible recommended such PCIE for Journal i can go with 2 x Journal one PCIE and one SATA drive. i did some google but got nowhere i checked https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/ however those card like Netlist EV3 16GB & Intel P3700 400GB they can't be found to bought!

The ceph will be for KVM Production Storage so it can be a lot of write and read.

 

[alexskysilk]

KVM Storage my reason to use the SATA enterprise Drives that i need huge of amount of space which SSD can't offer for cheap price

KVM is the what, not the how much. not really relevant in and of itself.

How may VMs? What OS will you be running on them, and how much OS partition space do the need? what sort of data do you need kept triple replicated, and is it really the best way to access it? How much of your data would be just as well served on a NAS?

Just because you bought a hammer, doesnt make all your problems nails.

 

[mada]

KVM is the what, not the how much. not really relevant in and of itself.

How may VMs? What OS will you be running on them, and how much OS partition space do the need? what sort of data do you need kept triple replicated, and is it really the best way to access it? How much of your data would be just as well served on a NAS?

Just because you bought a hammer, doesnt make all your problems nails.

i will start with 3 node and looking for something 100VM or 150VM maybe more depend on how much load i do get. i looking to do about 70TB of space . for what i understand replicated 2 will be fine with me .

 

[udo]

for what i understand replicated 2 will be fine with me .

Hi,
never ever use replica 2 with 70TB data except you like datalost!

It's has good reasons why ceph change the recommendation years ago from replica 2 to replica 3!

Udo

 

[mada]

Hi,
never ever use replica 2 with 70TB data except you like datalost!

It's has good reasons why ceph change the recommendation years ago from replica 2 to replica 3!

Udo

Thanks very much for letting me know i will do it as 3 however i still have issue to know what kind of Journal need to use for my case.

 

[mada]

How about PCIE SM961 as Journal? It seems enterprise!

 

[Jarek]

Are You going to use ceph luminous with bluestore?
If yes, journaling device isn't necessary.