Networking on new Cluster

[DrillSgtErnst]

Soo I am going through the manual and I have some questions.

5.7.1 states:
The network should not be used heavily by other members, ideally corosync runs on its own network. Do not use a shared network for corosync and storage (except as a potential low-priority fallback in a redundant configuration).

5.7.2 further says:
When creating a cluster without any parameters the corosync cluster network is generally shared with the Web UI and the VMs and their traffic.

Sooo I do have some questions.
I have 4 Nodes
pve 1 initially set IP vmbr0 10.100.3.1
pve 2 initially set IP vmbr0 10.100.3.2
pve 3 initially set IP vmbr0 10.100.3.3
pve 4 initially set IP vmbr0 10.100.3.4

For Cluster Creation I would like to add another NIC to every Server
Let's say
pve 1 vmbr1 10.0.0.1
pve 2 vmbr2 10.0.0.2
pve 3 vmbr3 10.0.0.3
pve 4 vmbr4 10.0.0.4

Soo I would like to use 10.0.0.x as default corosync (Link1) and 10.100.3.x as backup. BUT 10.0.0.x is only connected to itself, so I don't have interfereces from outside. They're like an Island.
I want to use vmbr0, respective 10.100.3.x for WebUI and Internettraffic of the VM.
I do fear locking myself out of the UI with this.
As far as I understand the Proxmox Host will use the bridge with set Gateway for Internettraffic, right? Is the UI also hosted on the bridge with the set gateway?
Inter-VM Communication will also go through the Public-Link, right? So if two machines talk to each other, the Public Link will be the bottleneck, correct? Now I would bond 2x1GBit for Public-VM Traffic, but this would mean, that my VMs can only communicate with 2GBit shared between all. With the Ceph Storage it would be possible to saturate 2 GBit fairly easy (e.g. copying files from one VM to another). Can I make Proxmox route the Inter-VM traffic different from the Internettraffic? I would have 1 40GB IB available. Even with the about 50% performance loss from ib over eth on PVE 20GB inter-VM Connect would be fine. I just think Inter-VM traffic limited to 2 GBit would be kinda slow for the entirety of this cluster.

Further I will use Ceph
2 56GB Connect-X3 Adapters in 802.3ad mode as Link1
1 40GB ConnectX Infiniband as Link2
For Ceph I am kinda sure it's allright.


I am thankful for every advice

On a secong note, I could use the IB for Migration, since this needs much performance. But I luckily have a 10GB Switch and some cards lying here.. I could insert these cards, so the Public Interface could get up to 20GB (LACP). The Internet would be Connected to that Switch with 1GBit, thats enough, but the VMs would talk with high speed.
Well...
Would it possibly work to bond eno1 & 2, and ib1
balance alb would allow dynamic load balancing, would this work, or will these two networkpartitions rather have other backdraws?

 

[aaron]

Hey, I try to answer what I can.

Soo I would like to use 10.0.0.x as default corosync (Link1) and 10.100.3.x as backup. BUT 10.0.0.x is only connected to itself, so I don't have interfereces from outside. They're like an Island.
I want to use vmbr0, respective 10.100.3.x for WebUI and Internettraffic of the VM.
I do fear locking myself out of the UI with this.

No, the web UI will listen on all interfaces.

As far as I understand the Proxmox Host will use the bridge with set Gateway for Internettraffic, right? Is the UI also hosted on the bridge with the set gateway?

Traffic is being routed the way IP routing works. If there is a better route than the gateway, that one is used. In most cases this is an interface which as the destination network configured. The ip route can give you an idea which routes are set (automatically).

Inter-VM Communication will also go through the Public-Link, right? So if two machines talk to each other, the Public Link will be the bottleneck, correct?

Depends on a few things. If the two VMs are on the same node, then the traffic never leaves the bridge. If the VMs are on different nodes then yes, the traffic will pass through the configured bridge out to the physical network.

Can I make Proxmox route the Inter-VM traffic different from the Internettraffic?

Regular IP routing comes into play here again. You would need to configure a second network for the VMs and use that for the VMs to talk to each other. I don't have any experience with IB but I suspect that you can use it as a normal NIC right? You would create another bridge with the IB NIC as bridge port. This second bridge is used for the additional NIC each VM will get. Then each VM will have two networks configured. One to access the internet with the default gateway and the other, internal one.

In general, the more NIC you have the better as you can split up the traffic into different subnets.

For example a very ideal setup could look like this:

2x 1GBit -> Corosync (two links for redundancy)
1x 1GBit -> Internet
1x 10GBit (or faster) -> Ceph public
1x 10GBit (or faster) -> Ceph cluster
1x 10GBit (or faster) -> Migration
1x 10GBit (or faster) -> Backup target (make sure to configure it on the IP in this subnet to force traffic here)
1x 10GBit (or faster) -> Inter VM traffic

Each network needs a different IP subnet.

Some networks should be bonded for redundancy, increasing the number of NICs needed. In reality, it is rarely possible to have that many NICs and certain compromises have to be made.


I hope this helps.

 

[DrillSgtErnst]

Thanks. That reply is very helpful.
The only Problem is, that I didn't want to use two NIC in one Server. I would like my servers to only see one NIC. But I will try some things out.

I will Update this topic by the time I'm done