touche! hahahahawell, unless you like to spend 3 times the cost of the hardware you need to have inferior performance, that was a given from the start
Understanding ceph performance and scaling
- Thread starter crembz
- Start date
I know this is an old thread but I am curious...
How would you potentially increase the IOPS or performance by spending 3x?
If you had the perfect 3 node hardware, what would it look like?
I'm guessing the biggest drawback is that writes have to be committed to all 3 nodes.
Is there a way to use a cache device to acknowledge writes locally before they get synced over to the rest of the ceph fs?
Here's what I'm thinking...
Dual Xeon 4GHz 20-core CPUs.
IT-mode flashed H310 - SAS break-out cables (12Gb).
Eight SAS Enterprise Endurance SSDs.
Four small NVMe on Gen3x16 pcie.
SAS for Data on Ceph....and if possible RockDB?
Four $mall NVMe's in Mirrors (R10) for the initial PVE install and improved host page cache "writeback"?
I'm very new to proxmox so go easy on me lol. Still learning but I come from a FreeNAS background.
Any help is appreciated. Looking to host a clustered MSSQL DB for an enterprise environment.
I'm thinking it may be better to have the SQL on non-ceph storage, with periodic FS replication?
How would you potentially increase the IOPS or performance by spending 3x?
If you had the perfect 3 node hardware, what would it look like?
I'm guessing the biggest drawback is that writes have to be committed to all 3 nodes.
Is there a way to use a cache device to acknowledge writes locally before they get synced over to the rest of the ceph fs?
Here's what I'm thinking...
Dual Xeon 4GHz 20-core CPUs.
IT-mode flashed H310 - SAS break-out cables (12Gb).
Eight SAS Enterprise Endurance SSDs.
Four small NVMe on Gen3x16 pcie.
SAS for Data on Ceph....and if possible RockDB?
Four $mall NVMe's in Mirrors (R10) for the initial PVE install and improved host page cache "writeback"?
I'm very new to proxmox so go easy on me lol. Still learning but I come from a FreeNAS background.
Any help is appreciated. Looking to host a clustered MSSQL DB for an enterprise environment.
I'm thinking it may be better to have the SQL on non-ceph storage, with periodic FS replication?
Last edited:
im not an expert on this but from what i've read on this forum so far
* don't create a 3 node ceph cluster for production, use 5 nodes ++
* ceph performance depends on amount of nodes, the local ssd's speed and network speed
* don't host vm's with databases on ceph or zfs if you really need them to be high performance
* do the database HA on the database level, not on the hypervisor level
* use 25G or faster nics for ceph and inter-node communications if possible
* use LACP for network redundancy
hardware depends on your needs but in general if you can
* use pcie gen5 nvme ssd's (direct attached backplane)
* don't use hba / raid controller if you can avoid it
* with pcie gen5 ssd's you don't need any log / cache / zil devices for zfs
* for best vm performance use mirrors / vdevs (raid 1 / 10 equivalents) instead of erasure coding raid modes (raid 5 / 6 equivalents)
just a few points
* don't create a 3 node ceph cluster for production, use 5 nodes ++
* ceph performance depends on amount of nodes, the local ssd's speed and network speed
* don't host vm's with databases on ceph or zfs if you really need them to be high performance
* do the database HA on the database level, not on the hypervisor level
* use 25G or faster nics for ceph and inter-node communications if possible
* use LACP for network redundancy
hardware depends on your needs but in general if you can
* use pcie gen5 nvme ssd's (direct attached backplane)
* don't use hba / raid controller if you can avoid it
* with pcie gen5 ssd's you don't need any log / cache / zil devices for zfs
* for best vm performance use mirrors / vdevs (raid 1 / 10 equivalents) instead of erasure coding raid modes (raid 5 / 6 equivalents)
just a few points
