Cluster nodes not fully communicating

K

krikey

Renowned Member
Aug 15, 2018
142
30
68
UK
I hope sopmeone can help?

I have two nodes in a cluster, both have 2 NICs. One WAN facing, the other LAN facing on their own local subnet 10.0.0.0/24. I can ping the other nodes on the local subnet no problems from the command line of each node.

Node 1 = 10.0.0.1
Node 2 = 10.0.0.2

In the GUI I can see both nodes and see a list of the running VMs, however, when I click on a VM or perhaps on the summary tab of another node, I get either a constantly spinning waiting icon or a "communication failure (0)" message.

Im thinking its corosync thats mis-configured in some way and I've included a copy of both corosync.conf files below:
Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node1
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.1
  }
  node {
    name: node2
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.0.2
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster001
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.2
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}
Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node1
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.1
  }
  node {
    name: node2
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.0.2
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster001
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.2
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

Copies of my /etc/hosts file look like this:

Code: 
127.0.0.1 localhost.localdomain localhost
1.2.3.4 node1.mydomainname.com node1
10.0.0.1 node1.mydomainname.com node1 pvelocalhost
10.0.0.2 node2.mydomainname.com node2

and

Code: 
127.0.0.1 localhost.localdomain localhost
1.2.3.5 node2.mydomainname.com node2
10.0.0.2 node2.mydomainname.com node2 pvelocalhost
10.0.0.1 node1.mydomainname.com node1

I've also temporarily disabled the PVE firewall thinking that this may have some bearing but it seems not.
 
I've configured NTP now for both nodes and this seems to be synced. Im also reaching out to the DC who provide the vLAN to ask if they support multicast on their vLANS.

I now have a different error message:

"No route to host (595)"

Oddly the node summary graphs all seem to be up to date so not suire whats being blocked specifically.
 
please provide the contents of `/etc/network/interfaces` and the output of `ip addr show`, `ip link show`, `ip route show` from both nodes
 
node 1
---------
Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address  x.x.x.x
        netmask  255.255.255.0
        gateway x.x.x.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address  10.0.0.1
        netmask  255.255.255.0
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0



1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether b0:83:fe:c3:f9:f1 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether b0:83:fe:c3:f9:f2 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b0:83:fe:c3:f9:f1 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/24 brd x.x.x.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::b283:feff:fec3:f9f1/64 scope link
       valid_lft forever preferred_lft forever
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b0:83:fe:c3:f9:f2 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::b283:feff:fec3:f9f2/64 scope link
       valid_lft forever preferred_lft forever
6: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr101i0 state UNKNOWN group default qlen 1000
    link/ether 46:3b:2d:c2:b9:de brd ff:ff:ff:ff:ff:ff
7: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:63:62:a2:e5:28 brd ff:ff:ff:ff:ff:ff
8: fwpr101p0@fwln101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether da:82:36:7c:92:33 brd ff:ff:ff:ff:ff:ff
9: fwln101i0@fwpr101p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP group default qlen 1000
    link/ether 7a:63:62:a2:e5:28 brd ff:ff:ff:ff:ff:ff
10: tap101i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 8e:6d:2f:db:32:1a brd ff:ff:ff:ff:ff:ff
11: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 32:8f:f1:d4:40:e7 brd ff:ff:ff:ff:ff:ff
13: veth107i0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:ca:90:90:0b:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
15: veth109i0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:cc:25:c3:74:d8 brd ff:ff:ff:ff:ff:ff link-netnsid 1


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:c3:f9:f1 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:c3:f9:f2 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:c3:f9:f1 brd ff:ff:ff:ff:ff:ff
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b0:83:fe:c3:f9:f2 brd ff:ff:ff:ff:ff:ff
6: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr101i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 46:3b:2d:c2:b9:de brd ff:ff:ff:ff:ff:ff
7: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 7a:63:62:a2:e5:28 brd ff:ff:ff:ff:ff:ff
8: fwpr101p0@fwln101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether da:82:36:7c:92:33 brd ff:ff:ff:ff:ff:ff
9: fwln101i0@fwpr101p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP mode DEFAULT group default qlen 1000
    link/ether 7a:63:62:a2:e5:28 brd ff:ff:ff:ff:ff:ff
10: tap101i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 8e:6d:2f:db:32:1a brd ff:ff:ff:ff:ff:ff
11: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 32:8f:f1:d4:40:e7 brd ff:ff:ff:ff:ff:ff
13: veth107i0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:ca:90:90:0b:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
15: veth109i0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:cc:25:c3:74:d8 brd ff:ff:ff:ff:ff:ff link-netnsid 1

default via x.x.x.x dev vmbr0 onlink
x.x.x.x/24 dev vmbr0 proto kernel scope link src x.x.x.x
10.0.0.0/24 dev vmbr1 proto kernel scope link src 10.0.0.1

node2
---------
Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address  x.x.x.x
        netmask  255.255.255.0
        gateway  x.x.x.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet static
        address  10.0.0.2
        netmask  255.255.255.0
        bridge_ports eno2
        bridge_stp off
        bridge_fd 0


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether c8:1f:66:bb:9c:86 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether c8:1f:66:bb:9c:87 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether c8:1f:66:bb:9c:86 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/24 brd x.x.x.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ca1f:66ff:febb:9c86/64 scope link
       valid_lft forever preferred_lft forever
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether c8:1f:66:bb:9c:87 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::ca1f:66ff:febb:9c87/64 scope link
       valid_lft forever preferred_lft forever
6: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr102i0 state UNKNOWN group default qlen 1000
    link/ether fe:78:32:55:aa:2f brd ff:ff:ff:ff:ff:ff
7: fwbr102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:48:70:37:c1:56 brd ff:ff:ff:ff:ff:ff
8: fwpr102p0@fwln102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether ca:40:94:98:48:1d brd ff:ff:ff:ff:ff:ff
9: fwln102i0@fwpr102p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UP group default qlen 1000
    link/ether 62:48:70:37:c1:56 brd ff:ff:ff:ff:ff:ff
13: veth106i0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP group default qlen 1000
    link/ether fe:2b:05:7a:d8:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 1
14: fwbr106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e6:f7:fa:d2:cf:04 brd ff:ff:ff:ff:ff:ff
15: fwpr106p0@fwln106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether d2:6d:7b:b9:10:f4 brd ff:ff:ff:ff:ff:ff
16: fwln106i0@fwpr106p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP group default qlen 1000
    link/ether e6:f7:fa:d2:cf:04 brd ff:ff:ff:ff:ff:ff
18: veth105i0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:75:da:1f:da:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether c8:1f:66:bb:9c:86 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000
    link/ether c8:1f:66:bb:9c:87 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether c8:1f:66:bb:9c:86 brd ff:ff:ff:ff:ff:ff
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether c8:1f:66:bb:9c:87 brd ff:ff:ff:ff:ff:ff
6: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr102i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:78:32:55:aa:2f brd ff:ff:ff:ff:ff:ff
7: fwbr102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 62:48:70:37:c1:56 brd ff:ff:ff:ff:ff:ff
8: fwpr102p0@fwln102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ca:40:94:98:48:1d brd ff:ff:ff:ff:ff:ff
9: fwln102i0@fwpr102p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UP mode DEFAULT group default qlen 1000
    link/ether 62:48:70:37:c1:56 brd ff:ff:ff:ff:ff:ff
13: veth106i0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:2b:05:7a:d8:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 1
14: fwbr106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e6:f7:fa:d2:cf:04 brd ff:ff:ff:ff:ff:ff
15: fwpr106p0@fwln106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether d2:6d:7b:b9:10:f4 brd ff:ff:ff:ff:ff:ff
16: fwln106i0@fwpr106p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP mode DEFAULT group default qlen 1000
    link/ether e6:f7:fa:d2:cf:04 brd ff:ff:ff:ff:ff:ff
18: veth105i0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:75:da:1f:da:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0


default via x.x.x.x dev vmbr0 onlink
x.x.x.0/24 dev vmbr0 proto kernel scope link src x.x.x.x
10.0.0.0/24 dev vmbr1 proto kernel scope link src 10.0.0.2
 
hmm - the interfaces look ok - however I noticed that you have the node names twice in the host file - once with the 'wan' and once with the 'lan' ips - maybe change the ones on the 'wan' side to something unique?

does the log (journalctl) or dmesg give any hints to what's not working?
 
I'll check journalctl or dmesg, but ive just been told that ip multicast is not supported on the vLAN that my nodes use. Could this be the issue?
 
After adjusting /etc/pve/corosync.conf accordingly i restarted the services as suggested, but still no connectivity. Should I restart each node?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back