touche! hahahahawell, unless you like to spend 3 times the cost of the hardware you need to have inferior performance, that was a given from the start
Understanding ceph performance and scaling
- Thread starter crembz
- Start date
I know this is an old thread but I am curious...
How would you potentially increase the IOPS or performance by spending 3x?
If you had the perfect 3 node hardware, what would it look like?
I'm guessing the biggest drawback is that writes have to be committed to all 3 nodes.
Is there a way to use a cache device to acknowledge writes locally before they get synced over to the rest of the ceph fs?
Here's what I'm thinking...
Dual Xeon 4GHz 20-core CPUs.
IT-mode flashed H310 - SAS break-out cables (12Gb).
Eight SAS Enterprise Endurance SSDs.
Four small NVMe on Gen3x16 pcie.
SAS for Data on Ceph....and if possible RockDB?
Four $mall NVMe's in Mirrors (R10) for the initial PVE install and improved host page cache "writeback"?
I'm very new to proxmox so go easy on me lol. Still learning but I come from a FreeNAS background.
Any help is appreciated. Looking to host a clustered MSSQL DB for an enterprise environment.
I'm thinking it may be better to have the SQL on non-ceph storage, with periodic FS replication?
How would you potentially increase the IOPS or performance by spending 3x?
If you had the perfect 3 node hardware, what would it look like?
I'm guessing the biggest drawback is that writes have to be committed to all 3 nodes.
Is there a way to use a cache device to acknowledge writes locally before they get synced over to the rest of the ceph fs?
Here's what I'm thinking...
Dual Xeon 4GHz 20-core CPUs.
IT-mode flashed H310 - SAS break-out cables (12Gb).
Eight SAS Enterprise Endurance SSDs.
Four small NVMe on Gen3x16 pcie.
SAS for Data on Ceph....and if possible RockDB?
Four $mall NVMe's in Mirrors (R10) for the initial PVE install and improved host page cache "writeback"?
I'm very new to proxmox so go easy on me lol. Still learning but I come from a FreeNAS background.
Any help is appreciated. Looking to host a clustered MSSQL DB for an enterprise environment.
I'm thinking it may be better to have the SQL on non-ceph storage, with periodic FS replication?
Last edited:
im not an expert on this but from what i've read on this forum so far
* don't create a 3 node ceph cluster for production, use 5 nodes ++
* ceph performance depends on amount of nodes, the local ssd's speed and network speed
* don't host vm's with databases on ceph or zfs if you really need them to be high performance
* do the database HA on the database level, not on the hypervisor level
* use 25G or faster nics for ceph and inter-node communications if possible
* use LACP for network redundancy
hardware depends on your needs but in general if you can
* use pcie gen5 nvme ssd's (direct attached backplane)
* don't use hba / raid controller if you can avoid it
* with pcie gen5 ssd's you don't need any log / cache / zil devices for zfs
* for best vm performance use mirrors / vdevs (raid 1 / 10 equivalents) instead of erasure coding raid modes (raid 5 / 6 equivalents)
just a few points
* don't create a 3 node ceph cluster for production, use 5 nodes ++
* ceph performance depends on amount of nodes, the local ssd's speed and network speed
* don't host vm's with databases on ceph or zfs if you really need them to be high performance
* do the database HA on the database level, not on the hypervisor level
* use 25G or faster nics for ceph and inter-node communications if possible
* use LACP for network redundancy
hardware depends on your needs but in general if you can
* use pcie gen5 nvme ssd's (direct attached backplane)
* don't use hba / raid controller if you can avoid it
* with pcie gen5 ssd's you don't need any log / cache / zil devices for zfs
* for best vm performance use mirrors / vdevs (raid 1 / 10 equivalents) instead of erasure coding raid modes (raid 5 / 6 equivalents)
just a few points
Can you elaborate on that? I dont see why 5 nodes should be prefeared over 3.
Thanks!
Three is the absolute minimum. As soon as anything "bad" happens with one node you are degraded. And you stay degraded as there is no space for self-healing...
Also: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/
Last edited: