Networking best practice

[M Anon]

Hi,

We're testing PVE as an alternative to ESXi and just trying to wrap our heads around networking in PVE vs ESX.

In ESX, we have 2 x 10Gbps ports (non LACP) connected to a vSwitch which has 3 vmkernel nics (1 for management, 1 for storage and 1 for vMotion) and then multiple port groups (each a different vlan). We simply attach each VM's nic to a portgroup.

I believe that to replicate the above in PVE, we need to create 1 VLAN-aware vmbr with no IP assigned, then create subinterfaces (vmbr0.10 for management with an IP assigned, vmbr0.20 for storage with an IP assigned, vmbr0.30 for phones with no IP assigned, vmbr0.40 for servers with no IP assigned, etc). We then connect each VM's nic to a sub-interface and then specify the VLAN tag. Is this correct?

And more importantly, is this best practice? I've read the admin guide that says PVE prefers LAGs/LACP which requires switch config (not a problem, ours support it), whilst ESX uses NIC teaming (equivlanet to balance-tlb in PVE I think) which does not require any config on the switch side.

Is there any best practice document out there we can take a look at?

 

[david_tao]

Following information is not best practice for PVE networking, just FYR.
And in the PVE document "https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration", it's mentioned about:

• If your switch supports the LACP (IEEE 802.3ad) protocol, then we recommend using the corresponding bonding mode (802.3ad). Otherwise you should generally use the active-backup mode.
• For the cluster network (Corosync) we recommend configuring it with multiple networks. Corosync does not need a bond for network redundancy as it can switch between networks by itself, if one becomes unusable.

 

[M Anon]

Following information is not best practice for PVE networking, just FYR.
And in the PVE document "https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration", it's mentioned about:

• If your switch supports the LACP (IEEE 802.3ad) protocol, then we recommend using the corresponding bonding mode (802.3ad). Otherwise you should generally use the active-backup mode.
• For the cluster network (Corosync) we recommend configuring it with multiple networks. Corosync does not need a bond for network redundancy as it can switch between networks by itself, if one becomes unusable.View attachment 83596

Wow! Thanks for the infographic. Not sure where you got that from but it certainly helps visualize.

This is basically how we have it configured now (except that vmbr1 is connected to bond0 because we don't have a separate nic for it).

Thanks

 

[david_tao]

Hi Anon:
You are welcome, the picture is draw by myself, use for in-project knowledge transfer with customer.. ^_^

 

[Tom123]

@david_tao
Was there a reason why you had a bridge between the management and the network device or can you leave it out?

 

[bofh]

@m

Wow! Thanks for the infographic. Not sure where you got that from but it certainly helps visualize.

This is basically how we have it configured now (except that vmbr1 is connected to bond0 because we don't have a separate nic for it).

Thanks

so forget the concepts of vmware, these are mostly vmware specific. proxmox is using barebones linux networking concepts, naming and schemes.

while it will work to make subinterfaces of your bridge interface you should do it the other way around.

so one or multiple bridges are used only for vms and container
the bridge has an interface. that can be physical (nic, bond) or a subinterface (vlan)


proxmox has in its core 4 networks.
management
vm
corosync
transfer
(new replication to make it 5)

you should at least logical (VLAN) seperate these. psychical seperation can be done but dont nessesarly has to be (keep in mind corosync needs low latency so max saturation phzsical might be an issue)


then you have the choice for vlans. you could either make multiple bridges and assign each a vlan via sub interface vlan as a bridge device
or passtrough vlans and set the vlan on a vm level (lots of choices)

and thats just the basics.

if you wanna go full fancy lookup SDN, here you can do a lot more things, specially with host seperation etc.
qinq routing, fabrics etc..


most important thing of all is to forget about vmware conventions.
it will get confusing trying to find equivalent there

 

[david_tao]

@david_tao
Was there a reason why you had a bridge between the management and the network device or can you leave it out?

Thank for your challenge. Yes, I agreed with you, the bridge (vmbr1) in the picture can take it away if only use for management.