After joining the cluster, the node IP is not accessible

[tangshuai]

I have a cluster consisting of two nodes, and due to high load, I want to expand a new node to join the cluster. Joining the cluster will display
Establishing API connection with host '10.203.5.45'
Login succeeded.
check cluster join API version
No cluster network links passed explicitly, fallback to local node IP '10.203.1.41'
Request addition of this node
Node 3 service reported an error
Normal node log display：Feb 19 12:23:55 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:24:05 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:24:15 TJ-node2 pvestatd[1998]: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:24:15 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:24:25 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:24:35 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:24:46 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:24:55 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:25:05 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:25:15 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:25:25 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:25:35 TJ-node2 pvestatd[1998]: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:25:35 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:25:45 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refusedFeb 19 12:25:55 TJ-node2 pvestatd[1998]: qemu status update error: metrics send error 'proxmoxve': failed to send metrics: Connection refused

 

[tangshuai]

Why is this

 

[fabian]

when you join a cluster, the joining node will take on the config of the cluster. it seems like in your case, that is not quite correct, as your joining node seems to have different storage layout and is not able to reach the configured metrics endpoint..

 

[tangshuai]

Two normal nodes formed a GlusterFS shared storage and mounted the cluster. However, when the third node was added, communication with the storage could not be established properly. But the node is displayed as green online in the cluster status, but the server is experiencing login issues and IP connectivity issues.

when you join a cluster, the joining node will take on the config of the cluster. it seems like in your case, that is not quite correct, as your joining node seems to have different storage layout and is not able to reach the configured metrics endpoint..

Two normal nodes formed a GlusterFS shared storage and mounted the cluster. However, when the third node was added, communication with the storage could not be established properly. But the node is displayed as green online in the cluster status, but the server is experiencing login issues and IP connectivity issues.

 

[fabian]

the cluster status and glusterfs connectivity are not related at all.. it sounds to me like you have some network misconfiguration, but without more details about your setup it's hard to tell for sure.

 

[tangshuai]

New nodes joining the cluster display：

Cluster status：

However, node 3 cannot be managed by the cluster normally, and node 3 cannot be pinged

Node 1 Service Status：

 

[fabian]

please post
- /etc/pve/corosync.cfg
- "ip -a" from all nodes
- "/etc/network/interfaces" from all nodes

if you need to censor addresses, please do so consistently

 

[tangshuai]

node2
:
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: TJ-node1
nodeid: 2
quorum_votes: 1
ring0_addr: 10.203.5.45
}
node {
name: TJ-node2
nodeid: 1
quorum_votes: 1
ring0_addr: 10.203.5.34
}
node {
name: TJ-node3
nodeid: 3
quorum_votes: 1
ring0_addr: 10.203.1.41
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: TJ-PVE
config_version: 9
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

 

[tangshuai]

node1:
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: TJ-node1
nodeid: 2
quorum_votes: 1
ring0_addr: 10.203.5.45
}
node {
name: TJ-node2
nodeid: 1
quorum_votes: 1
ring0_addr: 10.203.5.34

}
node {
name: TJ-node3
nodeid: 3
quorum_votes: 1
ring0_addr: 10.203.1.41
}
}

quorum {
provider: corosync_votequorum
}


totem {
cluster_name: TJ-PVE
config_version: 9
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

 

[tangshuai]

node3: Node3：