Network Recommendation 8 node cluster

[Haider Jarral]

Hello Experts,

I have read tons of blogs, posts, forums but I still can't get my head around setting networking for my following setup. I have following physical nodes

4 X Super Micros
Each has 4X1G and 2X10G

4 X HP Proliant 360 G7
Each has 4X1G and 2X10G

3 X Dell R730
Each has 2X10G, 2X1G, 2X10G

My requirements are to have

Redundant 1G for my VMs which will get Public IP
Redundant 10G for OSD Replication
Redundant 1G for providing a private IPs to VM which do not require Public IP

This is what I have in mind so far

LACP Bond 2X1G for VMs publics IPs
LACP Bond for 2X10G for Cluster Communication
LACP Bond for 2X1G for Private IPs

I have previously used same public and private network without a problem for 4 node setup. Will it be ok to use same for 8 node setup as well ? I have read a lot on having a separate coronsync at least 1G but I don't seem to have spare NIC port for it.

Is it really must to segregate public and private network for such setup ? If I don't have the NIC for it can I do it via vlans over 2x10G LACP Bond.


The other way I can think of is using 10G without LACP bonds but then I lose my redundancy.

Please provide some pointers to ease the situation.

 

[Alwin]

Corosync needs at least one independent link (no other network traffic) to provide low and stable latency. Ceph needs to run on the 10 GbE, otherwise it will be to slow, especially during recovery.

In general, separate cluster communication (corosync), storage and client traffic, from each other onto separate physical links.

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_requirements
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_precondition
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_requirements_3

 

[czechsys]

2x10G LACP for ceph backend
2x 1G (HP,SM) 2x10G (Dell) LACP for VMs public and nonpublic

corosync:
1x 1G for corosync dedicated 1G switch, secondary corosync can be other 1G on other dedicated or on LACP for ceph/VMs
the unused 1G can be used for secondary corosync or added to any 1G LACP