PVE4.1 3 Nodes Setup - Only local node shown in WebGUI

[thomas|i]

Hello,

we set up three nodes (with subscriptions) following the steps on:
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie
https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster
https://pve.proxmox.com/wiki/DRBD9

So far everything seems to be fine except that only the local-node is shown in the WebGUI.
(WebGUI of node#1 shows only node#1, GUI of node #2 only node#2, ...)

## pvecm status
Quorum information
------------------
Date: Fri Mar 18 18:03:13 2016
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 88
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.113.1 (local)
0x00000003 1 192.168.113.2
0x00000002 1 192.168.113.3

## pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 drbd-node01 (local)
3 1 drbd-node02
2 1 drbd-node03


## /etc/pve/.members
{
"nodename": "node01",
"version": 5,
"cluster": { "name": "pvec01", "version": 3, "nodes": 3, "quorate": 1 },
"nodelist": {
"drbd-node01": { "id": 1, "online": 1},
"drbd-node03": { "id": 2, "online": 1, "ip": "192.168.113.3"},
"drbd-node02": { "id": 3, "online": 1, "ip": "192.168.113.2"}
}
}


Did we miss something?

Thanks!


Edit:

## /etc/pve/nodes/node01/lrm_status
{"timestamp":1458321250,"results":{},"mode":"active","state":"wait_for_agent_lock"}
## /etc/pve/nodes/node02/lrm_status
{"timestamp":1458321252,"results":{},"mode":"active","state":"wait_for_agent_lock"}
## /etc/pve/nodes/node03/lrm_status
{"state":"wait_for_agent_lock","timestamp":1458321257,"mode":"active","results":{}}


=> wait_for_agent_lock ?

 

[dietmar]

Does it help if you clear the browser cache and reload the GUI?

 

[thomas|i]

I tried Chrome, IE, Firefox ... new Sessions with Clear Cache or even in privacy mode ... doesn't help.
Rebooting the nodes doesn't help neither

 

[wolfgang]

Hi,

can you send the output of
ls -ahl /etc/pve/nodes/

 

[thomas|i]

Sure!

root@node01:~# ls -ahl /etc/pve/nodes/
total 0
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 .
drwxr-xr-x 2 root www-data 0 Jan 1 1970 ..
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 node01
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 node02
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 node03
root@node01:~# ls -ahl /etc/pve/nodes/*
/etc/pve/nodes/node01:
total 1.5K
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 .
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 ..
-rw-r----- 1 root www-data 83 Mar 22 09:10 lrm_status
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 lxc
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 openvz
drwx------ 2 root www-data 0 Mar 18 14:14 priv
-rw-r----- 1 root www-data 1.7K Mar 18 14:14 pve-ssl.key
-rw-r----- 1 root www-data 1.8K Mar 18 14:14 pve-ssl.pem
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 qemu-server

/etc/pve/nodes/node02:
total 1.5K
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 .
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 ..
-rw-r----- 1 root www-data 83 Mar 22 09:10 lrm_status
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 lxc
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 openvz
drwx------ 2 root www-data 0 Mar 18 14:53 priv
-rw-r----- 1 root www-data 1.7K Mar 18 14:53 pve-ssl.key
-rw-r----- 1 root www-data 1.8K Mar 18 14:53 pve-ssl.pem
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 qemu-server

/etc/pve/nodes/node03:
total 1.5K
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 .
drwxr-xr-x 2 root www-data 0 Mar 18 14:14 ..
-rw-r----- 1 root www-data 83 Mar 22 09:10 lrm_status
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 lxc
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 openvz
drwx------ 2 root www-data 0 Mar 18 14:53 priv
-rw-r----- 1 root www-data 1.7K Mar 18 14:53 pve-ssl.key
-rw-r----- 1 root www-data 1.8K Mar 18 14:53 pve-ssl.pem
drwxr-xr-x 2 root www-data 0 Mar 18 14:53 qemu-server
root@node01:~#

 

[wolfgang]

What you get on your browser with this link?

https://192.168.113.3:8006/api2/html/cluster/resources

 

[thomas|i]

mh... 192.168.113.0/24 is the DRBD-Bonding Network, only available between the three Nodes.
We use different IPs for Management and Web-GUI access.

Using the 'management' IP I get:

{
"data" : [
{
"maxmem" : 50638344192,
"level" : "c",
"node" : "node03",
"id" : "node/node03",
"maxcpu" : 24,
"type" : "node",
"maxdisk" : 9707950080
},
{
"disk" : 1099785367552,
"storage" : "drbd1",
"maxdisk" : 1646046216192,
"type" : "storage",
"id" : "storage/node03/drbd1",
"node" : "node03"
},
{
"disk" : 1416892416,
"storage" : "local",
"type" : "storage",
"maxdisk" : 9707950080,
"id" : "storage/node03/local",
"node" : "node03"
}
]
}

 

[wolfgang]

Has your /etc/hosts the same ip as your cluster network?

 

[thomas|i]

node01 => /etc/hosts
127.0.0.1 localhost
192.168.113.1 drbd-node01 node01
192.168.113.2 drbd-node02 node02
192.168.113.3 drbd-node03 node03

node01 => ifconfig
bond0: 192.168.113.1 (DRBD-Bonding)
eth0: 192.168.41.150 (Management)

But I think we changed the /etc/hosts entrys during the setup :-/

 

[wolfgang]

After the change you reboot the server?

If not you have to do or minimum restart
pvedaemon
pveproxy
pvestatd

 

[thomas|i]

Yes, we rebooted all three nodes serveral times.


Maybe the Network-Setup is the reason for this issue?
We planned to use:
3x NIC (Linux-Bonding) for DRBD and the communication between the nodes (192.168.113.0/24)
1x NIC for management access (ssh, Web-GUI, 192.168.41.0/24)
1x NIC (Linux-Bridge) for the VMs network-access

 

[wolfgang]

The problem is you have 2 names for your nodes and this is not working.
corosync use drbd-node0x
and pve use node0x

 

[thomas|i]

Ok.
So I'm going to remove the 'drbd-nodeXX' names from /etc/hosts.

Do I have to manually edit the names in:
/etc/corosync/corosync.conf
/etc/pve/corosync.conf
/etc/pve/.members
/etc/pve/.version

?

After all: rebooting the nodes and maybe everything is ok?

 

[thomas|i]

I think it's solved... What I did:
- Stopped PVE (pve-cluster, pvedaemon, pveproxy)
- Deleted the 'drbd-nodeXX' entries from /etc/hosts
- Searched for those old names in /etc/
- Found /etc/corosync/corosync.conf and changed the names
- Rebooted all three nodes

Now on all three Web-GUIs all three nodes are shown (including the green status indicator).



## pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node01 (local)
3 1 node02
2 1 node03


## pvecm status
Quorum information
------------------
Date: Tue Mar 22 17:22:21 2016
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 132
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.113.1 (local)
0x00000003 1 192.168.113.2
0x00000002 1 192.168.113.3


## cat /etc/pve/.members
{
"nodename": "node01",
"version": 6,
"cluster": { "name": "pvec01", "version": 3, "nodes": 3, "quorate": 1 },
"nodelist": {
"node01": { "id": 1, "online": 1, "ip": "192.168.113.1"},
"node03": { "id": 2, "online": 1, "ip": "192.168.113.3"},
"node02": { "id": 3, "online": 1, "ip": "192.168.113.2"}
}
}