OVS Bridge Full Mesh with VLAN support

mnovi

New Member
Mar 9, 2021
18
0
1
47
Hello,

I'm using OVS configured for supporting 3 node cluster with full mesh connection (it's basically copied from wiki and it works perfectly). The problem that I'm facing now is separation of networks for different intention - I have an iSCSI share on one of the server nodes and I'd like to have one VLAN network dedicated to the storage access only (along with MTU=9000 if possible, whereas other networks would still have MTU=1500). The second VLAN network would be intended for communication between server nodes and the third VLAN network for communication between VMs. So far I corrected initial config from mesh, but it does not work at all. I changed it in different ways, but so far no success. This is what I have for now:

Code:
auto lo
iface lo inet loopback

iface ens20 inet manual

auto vmbr0
iface vmbr0 inet static
        address  192.168.2.51
        netmask  255.255.240.0
        gateway  192.168.2.1
        bridge_ports ens20
        bridge_stp off
        bridge_fd 0

auto ens18
iface ens18 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr1
    ovs_options other_config:rstp-enable=true other_config:rstp-path-cost=150 other_config:rstp-port-admin-edge=false other_config:rstp-port-auto-edge=false other_config:rstp-port-mcheck=true tag=1 vlan_mode=native-untagged trunks=2,3,4
    ovs_mtu 9000

auto ens19
iface ens19 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr1
    ovs_options other_config:rstp-enable=true other_config:rstp-path-cost=150 other_config:rstp-port-admin-edge=false other_config:rstp-port-auto-edge=false other_config:rstp-port-mcheck=true tag=1 vlan_mode=native-untagged trunks=2,3,4
    ovs_mtu 9000

auto vmbr1
iface vmbr1 inet static
    ovs_type OVSBridge
    ovs_ports ens18 ens19 vlan2 vlan3 vlan4
    up ovs-vsctl set Bridge ${IFACE} rstp_enable=true other_config:rstp-priority=32768 other_config:rstp-forward-delay=4 other_config:rstp-max-age=6
    post-up sleep 10

auto vlan2
iface vlan2 inet static
    address 10.15.15.50/24
    ovs_type OVSIntPort
    ovs_bridge vmbr1
    ovs_options tag=2

auto vlan3
iface vlan3 inet static
    address 10.15.16.50/24
    ovs_type OVSIntPort
    ovs_bridge vmbr1
    ovs_options tag=3
    ovs_mtu 9000

auto vlan4
iface vlan4 inet static
    address 10.15.17.50/24
    ovs_type OVSIntPort
    ovs_bridge vmbr1
    ovs_options tag=4


I am not sure if having VLANs in such configuration is even possible (RSTP) or I am doing something wrong with the specific part of config. The networks in vlan2, vlan3 and vlan4 would only exist in this cluster and not outside of it ...

- vlan2 = cluster traffic
- vlan3 = storage/iSCSI traffic
- vlan4 = VMs traffic

When I'm trying to ping from one server node to the other, I'm getting error Destination Host Unreachable. When looking at the traffic with tcpdump I see ARP packages are correctly tagged, but they simply don't appear on the other server node. How can I debug this?

Any help would be greatly appreciated. Thank you in advance!
 
I'm still trying to resolve this problem. Does anyone have some idea what should I check in order to come closer to the root of the problem?
 
I'm still trying to resolve this problem. Does anyone have some idea what should I check in order to come closer to the root of the problem?
 
I read many times these two wiki articles:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
https://pve.proxmox.com/wiki/Open_v...RSTP.29_-_1Gbps_uplink.2C_10Gbps_interconnect
hoping to achieve a simplified version without the two-switch stack from the above article.

It doesn't matter how often I tried, I was never able to communicate between my nodes; thus, couldn't bring anything online.

I could reach my three nodes from outside of the cluster. My nodes could ping everything on my LAN and reach the Internet but couldn't ping amongst themselves.

LLDP would confirm that the NICs were wired as expected. Spanning tree had all the correct blocks.

Switch settings (VLAN, rspt, jumbo frames, etc) were in place.

All seemed right but I was always missing the secret ingredient to make it work.

At the moment, I run this configuration:
https://packetpushers.net/proxmox-ceph-full-mesh-hci-cluster-w-dynamic-routing/
until I can find the article that will tell me what I'm missing :)

JEC
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!