[SOLVED] Network configuration on a Cisco UCS Blade

[babis430]

Hello everyone,

We would like to use Proxmox on a Cisco UCS B200 M-Series Blade. We've already installed the latest Proxmox version on the Host.
From the network side on the switch we've configured two LACPs which has 4 ports each. With the VLANs 9 for management and 201, 4092 for data. This has also been configured on the Cisco Blade Center.
How must this be configured within Proxmox, in order to avoid any networking issues (e.g. Loops)?
Our plan was to configure first one port (eno2) on the blade with the management network and put that on the bridge (vmbr0) afterwards we'd put every VLAN on all Ports (eno1 - eno4) and put all ports in the bridge vmbr0 (see attached image).
How would you configure it?


Thanks

 

[weehooey]

On a Cisco UCS, you will not use or need LACP for the bonds. Look at something like “balance-alb”.

Each of the four LAG member ports will go into a bond together. For example, if enoX are in a LAG and ensX are in a LAG, enoX will go to bond0 and ensX will go to bond1.

If all your traffic is tagged, do not put your host IPs on the ports or vmbrX. You will want to use VLAN tagged interfaces. E.g. vmbr0.9, vmbr1.201, vmbr1.4092.

More info: PVE Network Config

To be safe from loops, only deliver each VLAN over one LAG/bond — not both. This will help to prevent mistakes that will cause loops.

 

[babis430]

Hello @weehooey,

Thanks.

We were thinking of using one vmbrX for the VLANs 9, 201 and 4092 . So it would look like this:

VLAN 9: Port 0 => bond0 (Mode active-backup) => vmbr0 => Host Management
VLAN 201: Port 1 => bond1 (Mode balance-rr) => vmbr1 => Data Network
VLAN 4092: Port 2 => bond2 (Mode balance-rr) => vmbr2 => Data Network

 

[weehooey]

Hello @weehooey,

Thanks.

We were thinking of using one vmbrX for the VLANs 9, 201 and 4092 . So it would look like this:

VLAN 9: Port 0 => bond0 (Mode active-backup) => vmbr0 => Host Management
VLAN 201: Port 1 => bond1 (Mode balance-rr) => vmbr1 => Data Network
VLAN 4092: Port 2 => bond2 (Mode balance-rr) => vmbr2 => Data Network

Hi @babis430

We may have some confusion. My understanding from your original post is you presented eight vNICs from your UCS fabric interconnect to the blade server.

• eno1
• eno2
• eno3
• eno4
• ens210
• ens211
• ens610
• ens611

Is this correct? Are these 10 Gbps connections?

Also, I believe you said your three VLANs are available on all eight vNICs, is this correct?

In your last post, you mentioned three ports, three bonds, and three bridges, each with different bond modes. I have some questions about that:

• Are you using the Cisco UCS fabric interconnect or do you have a different network configuration?
• Why have you selected round-robin and active-backup?
• Why three bridges?
• Are the two data networks for VM/CT traffic or are they for host traffic?

Also, are you doing testing right now and planning to use more blades in a cluster? Or are you just planning to run PVE on just one blade?

Let me know. Cheers.

 

[babis430]

Hello @weehooey,

I'm sorry for the confusion on my part.
We will not utilize all eight network ports. At the moment we will use only 5 of these 10G ports:

VLAN 9: One port with the configuration Active-Backup. Here we will put the port in the active state and nothing in the backup slot.
VLAN 201: Two ports with the configuration balance-RR.
VLAN 4092: Two ports with the configuration balance-RR.

All the VLANs are tagged on the LACP configs between the Switches and the Cisco Fabric Interconnects. Which are also available on the vNICs on the Blade.

1. We are using the Cisco FI as a central networking switch for all Blades.
Between the FI and our switch, we have the two aforementioned LACPs in my first post. Between the FI and the Blade we thought of using the config above.

2. We chose Active-Backup for VLAN 9, because we would only use one network port for the management network. Would that cause an issue? Otherwise we could put in an other port as well.
As for the Round-Robin configs, we decided to go for it, because we thought that this would be more suited for the connection between the Blades and the FI. We can't configure LACP between these two instances. Could that cause a loop in any step of the config? Blade <=> FI <=> Switch?

3. The logic for the three bridges was to completely separate the networks from each other. What we also thought was to only create two bridges: Separating management and data networks.

4. The two data networks are for VMs.

Right now we're at the planing and testing phase. Our goal would be to install Proxmox on a couple Blades and use them in a cluster.
At this time these blades have VMware ESXi installed and are in a vCenter environment. But we want to remove a couple of these Hosts and create the new Proxmox environment.

Attached you will also find a plan of my thoughts.

I hope that my thoughts are not confusing.

Thank you very much for your help!


Regards,
Babis430

 

[weehooey]

Ah! That is much clearer. Thank you for the additional info.

I would do something like the following -- keeping in mind that I only know what you have provided, and there are likely things about your environment that I do not know. You will need to decide how to use this information.

One port => vmbr0 for host traffic (VLAN 9)

Four ports => bond1 > vmbr1 for all VM traffic
- bond1 set to balance-alb
- skip bond0 to reserve it for when you create a bond for the host traffic

SDN VLAN Zone on vmbr1 with two VNets (one for each 201 and 4092).

The logic and notes:

• You could put all five ports into one bond and bridge, which will work fine. However, I prefer to stay with the old rule of powers of 2 for the number of LAG members.
• Leave the host VLAN 9 without the complexity of the bond until you have another port you can use. You can convert to a bond later without hassle.
• In most cases, there is no advantage to having the four data ports on separate bridges. Bundle them and allow the VMs to share and balance over the whole pipe.
• The SDN feature is fantastic. Use it to manage the VM's VLANs. Once you have all your blades in Proxmox and need to make a network change, you will thank me
• I prefer using the balance-alb mode if the hardware supports it. Cisco UCS does. Sending alternating packets on different links has always seemed an unnecessary risk (i.e. packet reordering). You will likely be fine with either, so it's not a big deal to use balance-rr.

Some additional comments:

• You did not mention storage. Is that what you are saving the other three ports for?
• I like to see at least two host VLANs. One routable (internet access) and one isolated (i.e. no internet access). Make the isolated VLAN the primary Corosync network and the routable network as the backup Corosync network (with the web GUI and ssh traffic).

Let me know if you have any questions.

 

[babis430]

Hello @weehooey,

thank you for your support!
For Storage we would like to use the remaining ports in a bond.
As for the Corosync, we'll take a look for the setup and try configuring it.

Thank you very much!


Regards,
Babis430