corosync use the bridge ip instead of the ip set inthe configuration

[benoitc]

I have setup corosyn cover 2 network both handled in a distinct vlan (1 per nic) 10 on eno1 and 20 on eno2 and I was trying to give more priority to the VLAN 10 but I noticed that it's actually synchronizing over bridge instead of the VLAN 10 . Any idea what could be the reason ? What did I miss? How can we debug corosync?

in the switch I see the following log:

more /etc/corosync/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.10
    ring1_addr: 10.10.20.10
  }
  node {
    name: pve2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.11
    ring1_addr: 10.10.20.11
  }
  node {
    name: pve3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.10.12
    ring1_addr: 10.10.20.12
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: enki
  config_version: 5
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

# more /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

iface enx8cae4cdd0d4a inet manual

auto eno1.10
iface eno1.10 inet static
        address 10.10.10.10/24
#COROSYNC-A

auto eno2.20
iface eno2.20 inet static
        address 10.10.20.10/24
#COROSYNC-B

auto eno2.3161
iface eno2.3161 inet static
        address 10.10.130.10/24
#ISCSI-A

auto vmbr0
iface vmbr0 inet static
        address 10.10.1.10/24
        gateway 10.10.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

# pvecm status
Cluster information
-------------------
Name:             enki
Config Version:   5
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sat Feb 27 17:11:17 2021
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.1e07
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.10.10.10 (local)
0x00000002          1 10.10.10.11
0x00000003          1 10.10.10.12

[CODE=bash]Feb 27 15:02:45 pve1 systemd[1]: Starting Corosync Cluster Engine... Feb 27 15:02:45 pve1 corosync[8100]: [MAIN ] Corosync Cluster Engine 3.1.0 starting up Feb 27 15:02:45 pve1 corosync[8100]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlco Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] Initializing transport (Kronosnet). Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] totemknet initialized Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-g Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync configuration map access [0] Feb 27 15:02:45 pve1 corosync[8100]: [QB ] server name: cmap Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync configuration service [1] Feb 27 15:02:45 pve1 corosync[8100]: [QB ] server name: cfg Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync cluster closed process group servi Feb 27 15:02:45 pve1 corosync[8100]: [QB ] server name: cpg Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync profile loading service [4] Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync resource monitoring service [6] Feb 27 15:02:45 pve1 corosync[8100]: [WD ] Watchdog not enabled by configuration Feb 27 15:02:45 pve1 corosync[8100]: [WD ] resource load_15min missing a recovery key. Feb 27 15:02:45 pve1 corosync[8100]: [WD ] resource memory_used missing a recovery key. Feb 27 15:02:45 pve1 corosync[8100]: [WD ] no resources configured. Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync watchdog service [7] Feb 27 15:02:45 pve1 corosync[8100]: [QUORUM] Using quorum provider corosync_votequorum Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5] Feb 27 15:02:45 pve1 corosync[8100]: [QB ] server name: votequorum Feb 27 15:02:45 pve1 corosync[8100]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3] Feb 27 15:02:45 pve1 corosync[8100]: [QB ] server name: quorum Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] Configuring link 0 Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] Configured link number 0: local addr: 10.10.10.10, port=5405 Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] Configuring link 1 Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] Configured link number 1: local addr: 10.10.20.10, port=5406 Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 0) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [QUORUM] Sync members[1]: 1 Feb 27 15:02:45 pve1 corosync[8100]: [QUORUM] Sync joined[1]: 1 Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] A new membership (1.1e03) was formed. Members joined: 1 Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 3 has no active links Feb 27 15:02:45 pve1 corosync[8100]: [QUORUM] Members[1]: 1 Feb 27 15:02:45 pve1 corosync[8100]: [MAIN ] Completed service synchronization, ready to provide service. Feb 27 15:02:45 pve1 systemd[1]: Started Corosync Cluster Engine. Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] rx: host: 3 link: 1 is up Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] rx: host: 2 link: 1 is up Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] rx: host: 3 link: 0 is up Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] rx: host: 2 link: 0 is up Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397 Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] pmtud: PMTUD link change for host: 3 link: 1 from 469 to 1397 Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397 Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] pmtud: PMTUD link change for host: 2 link: 1 from 469 to 1397 Feb 27 15:02:47 pve1 corosync[8100]: [KNET ] pmtud: Global data MTU changed to: 1397 Feb 27 15:02:48 pve1 corosync[8100]: [QUORUM] Sync members[3]: 1 2 3 Feb 27 15:02:48 pve1 corosync[8100]: [QUORUM] Sync joined[2]: 2 3 Feb 27 15:02:48 pve1 corosync[8100]: [TOTEM ] A new membership (1.1e07) was formed. Members joined: 2 3 Feb 27 15:02:48 pve1 corosync[8100]: [QUORUM] This node is within the primary component and will provide service Feb 27 15:02:48 pve1 corosync[8100]: [QUORUM] Members[3]: 1 2 3 Feb 27 15:02:48 pve1 corosync[8100]: [MAIN ] Completed service synchronization, ready to provide service.[/CODE]

 

[Alwin Antreich]

VLAN 10 but I noticed that it's actually synchronizing over bridge instead of the VLAN 10

I don't see that in the logs.

Feb 27 15:02:45 pve1 corosync[8100]: [TOTEM ] Configured link number 0: local addr: 10.10.10.10, port=5405

Feb 27 15:02:45 pve1 corosync[8100]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 0)

 

[benoitc]

I don't see that in the logs.

Right, I think it's something I have to heck again. I did't see any messages exchanged when I was watching the switch. I will recheck. The main idea is to handle packets in priority on vlan 10 on this mikrotik switch.

Thanks for the answer anyway

 

[Alwin Antreich]

Right, I think it's something I have to heck again. I did't see any messages exchanged when I was watching the switch. I will recheck. The main idea is to handle packets in priority on vlan 10 on this mikrotik switch.

You could run corosync in active/active mode. This introduces more network traffic but has the advantage that no failover is needed, since the communication is done on both.