Not specific to Ceph, but for one cluster I went the used Mellanox route, SN2410's. Dirt cheap on ebay, relatively, very low latency. Needed 2 sets of these, instead I bought 6. 2 hot spares for 2 sets is enough redundancy :)
48x25 & 8x100 gbit each.
Currently we're running a PBS server containing 10 Samsung PM983 7.68 TB NVME drives. But.. we're coming to the point where we need to upgrade. Performance is excellent, but limited storage space. I have no extra bays available for more drives...
Sure. We found out one core in the ipsec endpoint (pfsense) on one side was running at 100% load and was limiting the transfer speed. After enabling MSS clamping (preventing fragmentation) and Asynchronous Cryptography (use multiple cores for...
Little ashamed to say the issue was found and was not in PBS. The ipsec tunnel endpoints had some issues. Now that these are resolved we can completely fill the gbit connection.
We've been using PBS for over a year now, it meets all our needs, love it. However - somehow thee offsite sync is way slower than expected. We can't really find any obvious bottlenecks.
Benchmark for the source host:
Time per request: 6515...