Traffic separation and nics - how to plan the network?

[rubenhauser]

Hi, we're building an 4 node PVE cluster with NVME Ceph storage.

Available Nics: We have several nics available:

• Nic1: 2 x 10G + 2 x 1G
• Nic2: 2 x 10G
• Nic3: 2 x 100G

Traffic/Networks: Now we need (I think) the following traffic separations:

• PVE Management
• PVE Cluster & Corosync
• Ceph (public) traffic > 2 x 10G Bond1 (MLAG)
• Public VM & Migration traffic > 2 x 10G Bond2 (MLAG)
• Ceph (internal) cluster traffic > 2 x 100G Bond3 (MLAG)

Question now: What to do best with the remaining 2 x 1G ports?

Bond/Bridge: All 10/100G ports should use Linux Bond to use MLAG. The Public VM & Migration traffic should also use a Linux bridge. Whats with the rest? No Bridge?

Thanks!

 

[gurubert]

My 2c:

Do not use the 100G network for the Ceph cluster network but instead for the public network. No need for a separate cluster network here.

Use 2x 10G for Proxmox management and VM migration.

Use the other 2x 10G for VM guest traffic.

Use the remaining 1G ports for additional corossnc rings.