Ceph cluster network is used for replication traffic only. All traffic from clients, MON, MDS, OSD flows through the public network. I would bond both 2.5 nics in an LACP LAG to two stacked/MLAG switches and set both public and cluster networks as vlans through the bond. That would get you the...