one of the 4 nodes does not see one other, cannot migrate to that one

[Nakata]

Hello,
i'm experiencing strange situation. I have 4 nodes freshly installed - node05 node07, node 08, node 09. Each see each other.
Only exception is, that node05 cannot see node09. as on attachment node-problem.jpg
Do you have idea, what should be wrong, what i could focus on ?
Same with migration, cannot migrate from node05 to node09. as on other attachments. Other way from node09 to node05 migraton works (as node09 can see node05)
it neither works via cli:
root@node05:~# qm migrate 11600 node09
no such cluster node 'node09'

EDIT: I can be logged in to any of 4 nodes, and migrate from node05 to node09 is not working


Each node can log to others via ssh

FYI, installed versions
node05 - pve-manager/7.0-11/63d82f4e

node07 - pve-manager/7.0-8/b1dbf562 (this is different one, but 07 has no problem to see each other)

node08 - pve-manager/7.0-11/63d82f4e

node09 - pve-manager/7.0-11/63d82f4e

node05 have also one storage more, but i'm trying migrate from local-lvm, not from ssd
part of strorage.cfg

dir: local
path /var/lib/vz
content snippets
shared 0

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images


lvmthin: local-lvm-ssd
thinpool lvm2-ssd
vgname pve2-ssd-vg
content rootdir,images
nodes node05

Thank you for any hint
RH

 

[jsterr]

Hello!

Please post your:

• /etc/pve/corosync.conf
• /etc/hosts from each host
  

Please keep all hosts up to date:

node07 - pve-manager/7.0-8/b1dbf562 (this is different one, but 07 has no problem to see each other)

 

[Nakata]

Hello,
it sems that this is the main reason, "pvecm status" on node05 shows:
Unable to get node address for nodeid 4: CS_ERR_NOT_EXIST

cat /etc/hosts:
root@node05:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.20.30.105 node05.xxxxx.net node05

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

---
root@node07:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.20.30.107 node07.xxxxx.net node07

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

---
root@node08:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.20.30.108 node08.xxxxx.net node08

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
---
root@node09:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.20.30.109 node09.xxxxx.net node09

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts



cat /etc/pve/corosync :

root@node05:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: node05
nodeid: 2
quorum_votes: 1
ring0_addr: 10.20.30.105
}
node {
name: node07
nodeid: 1
quorum_votes: 1
ring0_addr: 10.20.30.107
}
node {
name: node08
nodeid: 3
quorum_votes: 1
ring0_addr: 10.20.30.108
}
node {
name: node09
nodeid: 4
quorum_votes: 1
ring0_addr: 10.20.30.109
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: cluster01
config_version: 4
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

----------------------
root@node07:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: node05
nodeid: 2
quorum_votes: 1
ring0_addr: 10.20.30.105
}
node {
name: node07
nodeid: 1
quorum_votes: 1
ring0_addr: 10.20.30.107
}
node {
name: node08
nodeid: 3
quorum_votes: 1
ring0_addr: 10.20.30.108
}
node {
name: node09
nodeid: 4
quorum_votes: 1
ring0_addr: 10.20.30.109
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: cluster01
config_version: 4
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

---------------------------------

root@node08:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: node05
nodeid: 2
quorum_votes: 1
ring0_addr: 10.20.30.105
}
node {
name: node07
nodeid: 1
quorum_votes: 1
ring0_addr: 10.20.30.107
}
node {
name: node08
nodeid: 3
quorum_votes: 1
ring0_addr: 10.20.30.108
}
node {
name: node09
nodeid: 4
quorum_votes: 1
ring0_addr: 10.20.30.109
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: cluster01
config_version: 4
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}
------------------------

root@node09:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: node05
nodeid: 2
quorum_votes: 1
ring0_addr: 10.20.30.105
}
node {
name: node07
nodeid: 1
quorum_votes: 1
ring0_addr: 10.20.30.107
}
node {
name: node08
nodeid: 3
quorum_votes: 1
ring0_addr: 10.20.30.108
}
node {
name: node09
nodeid: 4
quorum_votes: 1
ring0_addr: 10.20.30.109
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: cluster01
config_version: 4
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

 

[jsterr]

Try to put all hosts in /etc/hosts on each node. So it should look like this on each node:
maybe restart of node(s) is neeeded im not sure.

root@node07:~# cat /etc/hosts

127.0.0.1 localhost.localdomain localhost
10.20.30.107 node07.xxxxx.net node07
10.20.30.105 node05.xxxxx.net node05
10.20.30.107 node08.xxxxx.net node07
10.20.30.108 node08.xxxxx.net node08
10.20.30.109 node09.xxxxx.net node09

 

[Nakata]

Id didnt help, but restart of the node did