Hyper-converged 100Gbe MTU settings and best practices....

[johnnydatacenter]

Hello,

We recently built a 3-node PVE hyper-converged cluster w/ ceph and I was wondering about the following:

• For MTU size on interfaces, does it matter where that is applied? I believe I've found that on linux bridge it is unnecessary as it inherits it from the bond, but what about the physical interface? Currently config example is attached.
• We're using MLAG across two switches - what should the hash policy be?
• On our network switches - we're using FS - should the port-channels be set to a higher MTU than the individual ports?
• What kind of speeds can we likely expect with a bonded 200Gb connection between nodes? Migrating VMs is pretty quick but I still see the speeds at around 2.2 Gb/s, so I'm just curious.

Just FYI, we're all L2, and are trunking all VLANs to the primary bond and bridge (bond 101 and vmbr0) - we tag the VMs themselves at the network adapter.

Appreciate anyone's input on the MTU and hashing policy.