Hello,
I'm running
proxmox-ve: 5.2-2 (running kernel: 4.15.17-1-pve)
pve-manager: 5.2-1 (running version: 5.2-1/0fcd7879)
pve-kernel-4.15: 5.2-1
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.2-pve5
on a couple of servers which I added into a cluster
Before cluster creation on all nodes were taken the following actions:
- same /etc/hosts on all servers
- date/time check
- apt-get update
I've created the cluster followed the official documentation at https://pve.proxmox.com/wiki/Cluster_Manager
When I add new node to the cluster again before add it to the cluster I check:
- /etc/hosts is the same
- date/time of the server
- run apt-get update
After I add new node/nodes to the cluster I see following happens:
- via web interface from random node I cannot see new added node
For example I have cluster from 5 nodes / all are seen via all web interface from all nodes /
node 1 - IP .1, node 2 - IP .2, node 3 - IP .3, node 4 - IP .4, node 5 - IP .5
When I open in the browser https://.1:8006 I am able to see all 5 nodes
When I open in the browser https://.2:8006 I am able to see all 5 nodes
When I open in the browser https://.3:8006 I am able to see all 5 nodes
When I open in the browser https://.4:8006 I am able to see all 5 nodes
When I open in the browser https://.5:8006 I am able to see all 5 nodes
here I add node 6 with IP .6 which is also visible via all web interfaces, node 7 with IP .7 which is also visible, but when I add node 8 with IP .8 I'm not abe to see node 8 from url address https://.4:8006
I've checked pvecm status and pvecm nodes on all nodes, all nodes are present BUT
when I check /etc/pve/.members on the node from which interface I am not able to see newly added node and I don't see newly added node in the node list.
And the real output: / only networking part is hidden - IP address x.x.x represents for example 192.168.0 network - all nodes are in the same network /
- node from which I am able to see newly added node via web interface
pvecm status
Quorum information
------------------
Date: Tue Oct 9 00:00:22 2018
Quorum provider: corosync_votequorum
Nodes: 7
Node ID: 0x00000001
Ring ID: 1/2656
Quorate: Yes
Votequorum information
----------------------
Expected votes: 7
Highest expected: 7
Total votes: 7
Quorum: 4
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 x.x.x.11 (local)
0x00000002 1 x.x.x.12
0x00000003 1 x.x.x.13
0x00000004 1 x.x.x.14
0x00000005 1 x.x.x.15
0x00000007 1 x.x.x.16
0x00000006 1 x.x.x.17
pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 x.x.x.11 (local)
2 1 x.x.x.12
3 1 x.x.x.13
4 1 x.x.x.14
5 1 x.x.x.15
7 1 x.x.x.16
6 1 x.x.x.17
.members
{
"nodename": "pve1",
"version": 32,
"cluster": { "name": "somecluster", "version": 9, "nodes": 7, "quorate": 1 },
"nodelist": {
"pve6": { "id": 7, "online": 1, "ip": "x.x.x.16"},
"pve1": { "id": 1, "online": 1, "ip": "x.x.x.11"},
"pve2": { "id": 2, "online": 1, "ip": "x.x.x.12"},
"pve3": { "id": 3, "online": 1, "ip": "x.x.x.13"},
"pve4": { "id": 4, "online": 1, "ip": "x.x.x.14"},
"pve5": { "id": 5, "online": 1, "ip": "x.x.x.15"},
"pve7": { "id": 6, "online": 1, "ip": "x.x.x.17"}
}
}
- node from which I am not able to see some of newly added node via web interface
pvecm status
Quorum information
------------------
Date: Tue Oct 9 00:06:40 2018
Quorum provider: corosync_votequorum
Nodes: 7
Node ID: 0x00000006
Ring ID: 1/2656
Quorate: Yes
Votequorum information
----------------------
Expected votes: 7
Highest expected: 7
Total votes: 7
Quorum: 4
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 x.x.x.11
0x00000002 1 x.x.x.12
0x00000003 1 x.x.x.13
0x00000004 1 x.x.x.14
0x00000005 1 x.x.x.15
0x00000007 1 x.x.x.16
0x00000006 1 x.x.x.17 (local)
pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 x.x.x.11
2 1 x.x.x.12
3 1 x.x.x.13
4 1 x.x.x.14
5 1 x.x.x.15
7 1 x.x.x.16
6 1 x.x.x.17 (local)
.members
{
"nodename": "pve7",
"version": 30,
"cluster": { "name": "somecluster", "version": 8, "nodes": 6, "quorate": 1 },
"nodelist": {
"pve1": { "id": 1, "online": 1, "ip": "x.x.x.11"},
"pve2": { "id": 2, "online": 1, "ip": "x.x.x.12"},
"pve3": { "id": 3, "online": 1, "ip": "x.x.x.13"},
"pve4": { "id": 4, "online": 1, "ip": "x.x.x.14"},
"pve5": { "id": 5, "online": 1, "ip": "x.x.x.15"},
"pve7": { "id": 6, "online": 1, "ip": "x.x.x.17"}
}
}
In members files of pve7 there is no entry for pve6 in the nodelist, I suppose that is the reason I am not able to see pve6 from pve7's web interface
From pve4's web interface I am also not able to see pve6 and again pve6 is missing in the nodelist from members file of pve4.
After that I'm adding pve8 to the cluster and I am able to see it from all pve's web interfaces.
The other:
Why when I run pvecm nodes I don't see node name from /etc/hosts, but I see IP addresses. This is not on all nodes, on 1 node I am able to see name of the node, but from other nodes when I run pvecm nodes I see instead of the node name, IP address. /etc/hosts are equal on all nodes.
When I remove 'faulthy' node / from which I am not able to see some of newly added nodes to the cluster / from the cluster and re-add it again - all nodes are visible and members file is fine. For removing the node I follow again https://pve.proxmox.com/wiki/Cluster_Manager
On 'faulthy' nodes I see difference in cluster version in members file and missing nodes.
It's not be a problem for me to reinstall all the servers and create the cluster again, but I would like to know why this happens and is it normal .
In the time of adding new node in the cluster I'm sure the network between all nodes is up and running, I've already created ping graphs between all nodes every 5 seconds and there is no network delay or network down issue.
It is just 1 from every 3 newly added nodes in the cluster is not presented in members files on all nodes, but only in some of the nodes. No errors during adding to the cluster, same output as other new added which are presented in members files on all nodes.
--
edit: Forgot to mention that cluster network is over 802.3ad layer3+4 mode bond instead of recommended:
"If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported."
I'm running
proxmox-ve: 5.2-2 (running kernel: 4.15.17-1-pve)
pve-manager: 5.2-1 (running version: 5.2-1/0fcd7879)
pve-kernel-4.15: 5.2-1
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.2-pve5
on a couple of servers which I added into a cluster
Before cluster creation on all nodes were taken the following actions:
- same /etc/hosts on all servers
- date/time check
- apt-get update
I've created the cluster followed the official documentation at https://pve.proxmox.com/wiki/Cluster_Manager
When I add new node to the cluster again before add it to the cluster I check:
- /etc/hosts is the same
- date/time of the server
- run apt-get update
After I add new node/nodes to the cluster I see following happens:
- via web interface from random node I cannot see new added node
For example I have cluster from 5 nodes / all are seen via all web interface from all nodes /
node 1 - IP .1, node 2 - IP .2, node 3 - IP .3, node 4 - IP .4, node 5 - IP .5
When I open in the browser https://.1:8006 I am able to see all 5 nodes
When I open in the browser https://.2:8006 I am able to see all 5 nodes
When I open in the browser https://.3:8006 I am able to see all 5 nodes
When I open in the browser https://.4:8006 I am able to see all 5 nodes
When I open in the browser https://.5:8006 I am able to see all 5 nodes
here I add node 6 with IP .6 which is also visible via all web interfaces, node 7 with IP .7 which is also visible, but when I add node 8 with IP .8 I'm not abe to see node 8 from url address https://.4:8006
I've checked pvecm status and pvecm nodes on all nodes, all nodes are present BUT
when I check /etc/pve/.members on the node from which interface I am not able to see newly added node and I don't see newly added node in the node list.
And the real output: / only networking part is hidden - IP address x.x.x represents for example 192.168.0 network - all nodes are in the same network /
- node from which I am able to see newly added node via web interface
pvecm status
Quorum information
------------------
Date: Tue Oct 9 00:00:22 2018
Quorum provider: corosync_votequorum
Nodes: 7
Node ID: 0x00000001
Ring ID: 1/2656
Quorate: Yes
Votequorum information
----------------------
Expected votes: 7
Highest expected: 7
Total votes: 7
Quorum: 4
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 x.x.x.11 (local)
0x00000002 1 x.x.x.12
0x00000003 1 x.x.x.13
0x00000004 1 x.x.x.14
0x00000005 1 x.x.x.15
0x00000007 1 x.x.x.16
0x00000006 1 x.x.x.17
pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 x.x.x.11 (local)
2 1 x.x.x.12
3 1 x.x.x.13
4 1 x.x.x.14
5 1 x.x.x.15
7 1 x.x.x.16
6 1 x.x.x.17
.members
{
"nodename": "pve1",
"version": 32,
"cluster": { "name": "somecluster", "version": 9, "nodes": 7, "quorate": 1 },
"nodelist": {
"pve6": { "id": 7, "online": 1, "ip": "x.x.x.16"},
"pve1": { "id": 1, "online": 1, "ip": "x.x.x.11"},
"pve2": { "id": 2, "online": 1, "ip": "x.x.x.12"},
"pve3": { "id": 3, "online": 1, "ip": "x.x.x.13"},
"pve4": { "id": 4, "online": 1, "ip": "x.x.x.14"},
"pve5": { "id": 5, "online": 1, "ip": "x.x.x.15"},
"pve7": { "id": 6, "online": 1, "ip": "x.x.x.17"}
}
}
- node from which I am not able to see some of newly added node via web interface
pvecm status
Quorum information
------------------
Date: Tue Oct 9 00:06:40 2018
Quorum provider: corosync_votequorum
Nodes: 7
Node ID: 0x00000006
Ring ID: 1/2656
Quorate: Yes
Votequorum information
----------------------
Expected votes: 7
Highest expected: 7
Total votes: 7
Quorum: 4
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 x.x.x.11
0x00000002 1 x.x.x.12
0x00000003 1 x.x.x.13
0x00000004 1 x.x.x.14
0x00000005 1 x.x.x.15
0x00000007 1 x.x.x.16
0x00000006 1 x.x.x.17 (local)
pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 x.x.x.11
2 1 x.x.x.12
3 1 x.x.x.13
4 1 x.x.x.14
5 1 x.x.x.15
7 1 x.x.x.16
6 1 x.x.x.17 (local)
.members
{
"nodename": "pve7",
"version": 30,
"cluster": { "name": "somecluster", "version": 8, "nodes": 6, "quorate": 1 },
"nodelist": {
"pve1": { "id": 1, "online": 1, "ip": "x.x.x.11"},
"pve2": { "id": 2, "online": 1, "ip": "x.x.x.12"},
"pve3": { "id": 3, "online": 1, "ip": "x.x.x.13"},
"pve4": { "id": 4, "online": 1, "ip": "x.x.x.14"},
"pve5": { "id": 5, "online": 1, "ip": "x.x.x.15"},
"pve7": { "id": 6, "online": 1, "ip": "x.x.x.17"}
}
}
In members files of pve7 there is no entry for pve6 in the nodelist, I suppose that is the reason I am not able to see pve6 from pve7's web interface
From pve4's web interface I am also not able to see pve6 and again pve6 is missing in the nodelist from members file of pve4.
After that I'm adding pve8 to the cluster and I am able to see it from all pve's web interfaces.
The other:
Why when I run pvecm nodes I don't see node name from /etc/hosts, but I see IP addresses. This is not on all nodes, on 1 node I am able to see name of the node, but from other nodes when I run pvecm nodes I see instead of the node name, IP address. /etc/hosts are equal on all nodes.
When I remove 'faulthy' node / from which I am not able to see some of newly added nodes to the cluster / from the cluster and re-add it again - all nodes are visible and members file is fine. For removing the node I follow again https://pve.proxmox.com/wiki/Cluster_Manager
On 'faulthy' nodes I see difference in cluster version in members file and missing nodes.
It's not be a problem for me to reinstall all the servers and create the cluster again, but I would like to know why this happens and is it normal .
In the time of adding new node in the cluster I'm sure the network between all nodes is up and running, I've already created ping graphs between all nodes every 5 seconds and there is no network delay or network down issue.
It is just 1 from every 3 newly added nodes in the cluster is not presented in members files on all nodes, but only in some of the nodes. No errors during adding to the cluster, same output as other new added which are presented in members files on all nodes.
--
edit: Forgot to mention that cluster network is over 802.3ad layer3+4 mode bond instead of recommended:
"If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported."
Last edited: