Proxmox networking best practices (aka How to setup networking right this time)

[hanley.development]

For a HA system with 3 nodes, is it a fair assessment that can both use CEPH and NAS storage would work in this setup?

1 10GB nic for Storage (VLAN 10)
2 Bonded 2.5GB nic for Web interface and VM usage
2 Bonded 1GB nic for cluster (VLAN 1)

Rather than make this look like Pepe Silvia conspiracy wall I drew the first lines, but would be repeated for each node in the cluster.

 

[gurubert]

It depends on your workload and the type of devices you use for Ceph. A single 10G NIC can be full quite fast.

 

[hanley.development]

The 10gb nic would be connected to a 10 gb switch for ceph. Would that work for migration or does that require another nic?

 

[gurubert]

It would work. It depends on what your requirements are for performance.

 

[hanley.development]

My requirements are
1) Fast HA migration
2) redundant networks for clustering
3) redundant networks for Web and vm traffic
4) Backup storage to NAS
5) CEPH replication

The main question is splitting up all of these into networks that has the best performance and secondly would this be the correct way to achieve this using this in my network config:

auto lo
iface lo inet loopback


#Cluster
auto en05
iface en05 inet manual

auto en06
iface en06 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves en05 en06
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4

auto vmbr1
iface vmbr1 inet static
        address 192.168.1.153/24
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1


#Web/ VM Network
auto enp86s0
iface enp86s0 inet manual

auto enp87s0
iface enp87s0 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp86s0 enp87s0
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4

auto vmbr0
iface vmbr0 inet manual
        address 192.168.0.153/24
        gateway 192.168.0.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4092
        
#storage/migration
auto en07
iface en07 inet manual
        
auto vmbr2
iface vmbr2 inet manual
        address 192.168.10.153/24
        bridge-ports en07
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 10

 

[hanley.development]

I guess what I don't understand is that at the device level (bridge) there is no way to set the VLAN ID. Should they run all under the same subnet or different subnets? Is there a better way of separating those networks from each other?

 

[gurubert]

You only create bridge interfaces where you want to attach VMs and have VM network traffic.

If you want to have bond1 in a different VLAN than bond0 then the switch has to be configured to switch this VLAN untagged to the ports for bönd1.

If you need multiple VLANs vmbr1 then you can make it VLAN aware and the switch has to tag them.

 

[hanley.development]

Ah.... that makes sense so vmbr2 (incase i need to connect something to storage) and vmbr0 should be my only bridges. Then on the actual switches tag Bond1 connections with vlan 1 and vmbr2 with vlan 10.