Multisite problem

[ilneofita]

Hi
I have 2 sites connected using the wireguard, SITEA the wireguard is on the router, the SITEB the wireguard is directly on PVE. I see the traffic using tcpdump but the two site sees the other down
Here the coresync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-garage
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.50.35
  }
  node {
    name: pve-hp
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.50.37
  }
  node {
    name: pve-ss
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.50.36
  }
  node {
    name: pve04
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 10.10.10.1
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: Casa
  config_version: 5
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

What I should check?
An example of tcpdup

09:43:04.206266 IP 10.10.10.1.5405 > 192.168.50.37.5405: UDP, length 112
09:43:04.212355 IP 192.168.50.37.5405 > 10.10.10.1.5405: UDP, length 112
09:43:04.255262 IP 192.168.50.37.5405 > 10.10.10.1.5405: UDP, length 112
09:43:04.256299 IP 10.10.10.1.5405 > 192.168.50.37.5405: UDP, length 112

If I restart a node on SITEA this what I see on SITEB

Oct 08 09:45:32 pve04 corosync[4425]:   [QUORUM] Sync members[1]: 4
Oct 08 09:45:32 pve04 corosync[4425]:   [TOTEM ] A new membership (4.13e50) was formed. Members
Oct 08 09:45:32 pve04 corosync[4425]:   [QUORUM] Members[1]: 4
Oct 08 09:45:32 pve04 corosync[4425]:   [MAIN  ] Completed service synchronization, ready to provide service.
Oct 08 09:45:41 pve04 corosync[4425]:   [KNET  ] link: host: 3 link: 0 is down
Oct 08 09:45:41 pve04 corosync[4425]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Oct 08 09:45:41 pve04 corosync[4425]:   [KNET  ] host: host: 3 has no active links
Oct 08 09:45:42 pve04 corosync[4425]:   [QUORUM] Sync members[1]: 4
Oct 08 09:45:42 pve04 corosync[4425]:   [TOTEM ] A new membership (4.13e5c) was formed. Members
Oct 08 09:45:42 pve04 corosync[4425]:   [QUORUM] Members[1]: 4
Oct 08 09:45:42 pve04 corosync[4425]:   [MAIN  ] Completed service synchronization, ready to provide service.
Oct 08 09:45:46 pve04 corosync[4425]:   [KNET  ] link: Resetting MTU for link 0 because host 3 joined
Oct 08 09:45:46 pve04 corosync[4425]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Oct 08 09:45:46 pve04 corosync[4425]:   [KNET  ] pmtud: Global data MTU changed to: 1317

 

[cheiss]

Hi,

what's the latency between those two sites?
Please note that the cluster networks requires a reliable network with <5ms latency - see also our documentation on requirements: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_cluster_network

Running clusters of the internet is highly discouraged and can break very easily.

 

[ilneofita]

Hi,

what's the latency between those two sites?
Please note that the cluster networks requires a reliable network with <5ms latency - see also our documentation on requirements: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_cluster_network

Running clusters of the internet is highly discouraged and can break very easily.

Thank you the latency was ~100ms that probably is the problem. I will delete the external node.

 

[cheiss]

You may be interested in the Proxmox Datacenter Manager, depending on your usecase: https://forum.proxmox.com/threads/proxmox-datacenter-manager-0-9-beta-released.171742/

This will allow you to get a centralized overview/management plane over all your nodes, without requiring them to be all part of the same cluster