Proxmox Cluster with OVS and OOB Management

[paul477]

Hi,

I try to build a 3 node Proxmox Cluster.
The administrative interfaces shall connect to a 1Gb Switch with a "normal" vmbr0 switch on the nodes.
All clustertraffic and communications between virtual machines shall run via 10Gb Links that form a ring structure. RSTP shall manage where the ring has its open link.
I intend to have one network/vlan for cluster communications, one for ceph communications and others for traffic between the test machines running on the cluster.
structure as depicted in the appended drawing.
Config is derived from https://pve.proxmox.com/wiki/Open_vSwitch Example 4, with the main difference of the separate switch for administration, and no redundancy so far for the administration links.

Configuration of node proxmox02:

auto lo
iface lo inet loopback

iface enp10s0f0 inet manual

iface enp10s0f1 inet manual

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto enp7s0f0
iface enp7s0f0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr2
#       ovs_mtu 9000
        ovs_options other_config:rstp-enable=true other_config:rstp-path-cost=150 other_config:rstp-port-admin-edge=false other_config:rstp-port-auto-edge=false other_config:rstp-port-mcheck=true

auto enp7s0f1
iface enp7s0f1 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr2
#       ovs_mtu 9000
        ovs_options other_config:rstp-enable=true other_config:rstp-path-cost=150 other_config:rstp-port-admin-edge=false other_config:rstp-port-auto-edge=false other_config:rstp-port-mcheck=true

#PVE Cluster Com
auto vlan11
iface vlan11 inet static
        address 192.168.172.2
        netmask 255.255.255.0
        ovs_type OVSIntPort
        ovs_bridge vmbr2
#       ovs_mtu 9000
        ovs_options tag=11

# Ceph cluster comm
auto vlan22
iface vlan22 inet static
        address 192.168.182.2
        netmask 255.255.255.0
        ovs_type OVSIntPort
        ovs_bridge vmbr2
#        ovs_mtu 9000
        ovs_options tag=22


auto vmbr0
iface vmbr0 inet static
        address 172.19.3.130/16
        gateway 172.19.1.1
        bridge-ports enp10s0f0
        bridge-stp off
        bridge-fd 0
#admin net

auto vmbr1
iface vmbr1 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0

auto vmbr2
iface vmbr2 inet manual
        ovs_type OVSBridge
        ovs_ports enp7s0f0 enp7s0f1 vlan11 vlan22
#       ovs_mtu 9000
        up ovs-vsctl set Bridge ${IFACE} rstp_enable=true other_config:rstp-priority=32769 other_config:rstp-forward-delay=4 other_config:rstp-max-age=6
        post-up sleep 10

Problem: I get no pings between the proxmoxes between their cluster IPs.
tcpdump sees ARP requests from all nodes on the hardware NIC but only the requests from the local machine on the vlan interface.

Any ideas what is wrong with this setup?

Regards

Paul.

 

[huntah]

I have exactly the same problem.
The VLANs over OVS Bridge are not working. I cannot ping between PVE hosts. However on vmbr1 (10Gbit OVS Bridge) i can ping the host without a problem.
Did you manage to resolve your problem?
Here is my post about the problem:
https://forum.proxmox.com/threads/ovs-inport-vlan-ceph-10gbit.106971/

 

[paul477]

sorry for the delayed writing, but I'm currently dealing with different hardware problems: build up a new kitchen after having moved 500km to the beloved city of Dresdeen/Saxony/Germany. And "my" Ukrainian refugees and their challenges. I'll come back to this possibly next week.

Sry for the moment

Paul.