ceph public MTU from 1500 to 9000

[traveltrex]

Hi there,

currently we are running a 4-node PVE cluster with ceph. From Best Practice recommendations, we split up the ceph traffic for cluster and public and put it on different switches, network cards and networks. Now as our network is growing, we want to change the mtu to 9000 (as the current used switch cannot be used for jumbo frames, as not all connected peers are able to use jumbo frames). I have some questions for this

1. Is it really necessary? I know it might be a weird question, but i read that unless we dont have any problems, its not actually necessary.
2. What would be the best strategy for switching from 1500 to 9000?
3. Is it still best practice to split up public and cluster networks? Right now we are using 25gbit/s fc connections on a separate switch using balance-xor l3+4 for ceph cluster and its not even utilizing 50% of the bandwith.

Thank you for any advice!
Patrick

 

[Francois Baillargeon]

Honestly, it's more trouble than worth, we had zillions of times when "Configured correctly " did not pass pass the

ping -M do -s 8972 [destinationIP]

test

All moderns switch ASICS can do line speed with regular frames, and most NICs have hardware tcp offload of some sort nowadays (or a fast enough cpu)

At Croit we do not separate it anymore since the past five years, as sometimes if one of the two break OSDs can end up kind of confused about getting to the MON, and we sell a Ceph appliance so we have a few hundreds of sites already deployed.

 

[traveltrex]

Thank you for your advice. We will keep the MTU 1500 then.

 

[Francois Baillargeon]

Thank you for your advice. We will keep the MTU 1500 then.

Adding to that, it raises latency a bit too to have bigger frames, testing would give you the real answer if you are unsure.