Hello!
So we're planning to migrate our vCenter cluster over to Proxmox, and right now we've got a test cluster setup using hardware that's pretty close to what we intend to order. We'd like to use SDN on our new Proxmox cluster, and based on everything I've read in the docs for Proxmox, OpenVSwitch, FRRouting, as well as Reddit, Discord, etc. It seems like the correct option.
We've got 3 physical nodes in our cluster, each with 4 NICs across 2 PCIE cards (aside from a dedicated IPMI). We'd like to bond the interfaces so that eth0card1 + eth0card2 are bond0, and eth1card1 + eth1card2 are bond1.
We'd be trunking these bonds, the switch has LACP support and is expecting those interfaces bonded in that way. At the switch level there's only 3 VLANs allowed, let's call them 15, 16, and 17. We'd like to have 15 be where corosync and cluster management happen. This would be on bond0. We'd like for VLAN 16 to be our development net for our VMs, and also on bond0. Finally we'd like our Ceph storage network to be VLAN17, on bond1 by itself due to the bandwidth requirements.
I've tried a lot of different configurations, especially referencing this wiki page: https://pve.proxmox.com/wiki/Open_vSwitch but alas, none have worked. Below is a sanitized version of the last attempted OVS configuration. I'd appreciate any help/guidance on what might be going on here, and how to resolve it.
So we're planning to migrate our vCenter cluster over to Proxmox, and right now we've got a test cluster setup using hardware that's pretty close to what we intend to order. We'd like to use SDN on our new Proxmox cluster, and based on everything I've read in the docs for Proxmox, OpenVSwitch, FRRouting, as well as Reddit, Discord, etc. It seems like the correct option.
We've got 3 physical nodes in our cluster, each with 4 NICs across 2 PCIE cards (aside from a dedicated IPMI). We'd like to bond the interfaces so that eth0card1 + eth0card2 are bond0, and eth1card1 + eth1card2 are bond1.
We'd be trunking these bonds, the switch has LACP support and is expecting those interfaces bonded in that way. At the switch level there's only 3 VLANs allowed, let's call them 15, 16, and 17. We'd like to have 15 be where corosync and cluster management happen. This would be on bond0. We'd like for VLAN 16 to be our development net for our VMs, and also on bond0. Finally we'd like our Ceph storage network to be VLAN17, on bond1 by itself due to the bandwidth requirements.
I've tried a lot of different configurations, especially referencing this wiki page: https://pve.proxmox.com/wiki/Open_vSwitch but alas, none have worked. Below is a sanitized version of the last attempted OVS configuration. I'd appreciate any help/guidance on what might be going on here, and how to resolve it.
Rich (BB code):
auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth2 inet manual
iface eth1 inet manual
iface eth3 inet manual
auto bond0
iface bond0 inet manual
ovs_type OVSBond
ovs_bonds eth0 eth3
ovs_other_config bond-mode 802.3ad
ovs_other_config lacp=active
auto bond1
iface bond1 inet manual
ovs_type OVSBond
ovs_bonds eth2 eth1
ovs_other_config bond-mode 802.3ad
ovs_other_config lacp=active
#VLAN 15
auto br15
iface br15 inet static
address XXX.XXX.XX.XXX/24
ovs_type OVSBridge
ovs_ports port15_0 port15_1
#ovs_vlan_mode trunk
#ovs_trunks 15
auto port15_0
iface port15_0 inet manual
ovs_type OVSPort
ovs_bridge br15
ovs_bonds bond0
ovs_options "tag=15"
auto port15_1
iface port15_1 inet manual
ovs_type OVSPort
ovs_bridge br15
ovs_bonds bond1
ovs_options "tag=15"
#VLAN 16
auto br16
iface br16 inet static
address XX.XXX.XX.XXX/24
ovs_type OVSBridge
ovs_ports port16_0 port16_1
#ovs_vlan_mode trunk
#ovs_trunks 16
auto port16_0
iface port16_0 inet manual
ovs_type OVSPort
ovs_bridge br16
ovs_bonds bond0
ovs_options "tag=16"
auto port16_1
iface port16_1 inet manual
ovs_type OVSPort
ovs_bridge br16
ovs_bonds bond1
ovs_options "tag=16"
#VLAN 17
auto br17
iface br17 inet static
address XXX.XXX.XX.XXX/27
ovs_type OVSBridge
ovs_ports port17_0 port17_1
#ovs_vlan_mode trunk
#ovs_trunks 17
auto port17_0
iface port17_0 inet manual
ovs_type OVSPort
ovs_bridge br17
ovs_bonds bond0
ovs_options "tag=17"
auto port17_1
iface port17_1 inet manual
ovs_type OVSPort
ovs_bridge br17
ovs_bonds bond1
ovs_options "tag=17"