Two Node cluster, each node only sees itself "online" in GUI

[mstocker]

Everything appears to be functioning properly other than each node only sees itself as online (green). I can access the other node in the GUI but it does not display the VM names, only IDs, and its icon appears as red.

I have updated to the newest Proxmox release and rebooted the nodes several times. The nodes have 2x bonded 1gbps crossover for node communication and drbd. Multicast seem to be working fine tested with asmping.

Some information:

root@pm1:~# pveversion -v
pve-manager: 2.2-32 (pve-manager/2.2/3089a616)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.2-83
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-14-pve: 2.6.32-74
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-34
qemu-server: 2.0-72
pve-firmware: 1.0-21
libpve-common-perl: 1.0-41
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.3-10
ksm-control-daemon: 1.1-1

root@pm2:~# pveversion -vpve-manager: 2.2-32 (pve-manager/2.2/3089a616)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.2-83
pve-kernel-2.6.32-14-pve: 2.6.32-74
pve-kernel-2.6.32-17-pve: 2.6.32-83
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-34
qemu-server: 2.0-72
pve-firmware: 1.0-21
libpve-common-perl: 1.0-41
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.3-10
ksm-control-daemon: 1.1-1

root@pm1:~# cat /etc/hostname
pm1

(Domain name replaced with x's)

root@pm1:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
172.16.1.1 pm1.xxxxxxxxxxxxxxx pm1 pvelocalhost


# The following lines are desirable for IPv6 capable hosts
172.16.1.2 pm2.xxxxxxxxxxxxxxx pm2


::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

root@pm2:~# cat /etc/hostname
pm2

root@pm2:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
172.16.1.2 pm2.xxxxxxxxxxxxxxx pm2 pvelocalhost
172.16.1.1 pm1.xxxxxxxxxxxxxxx pm1
# The following lines are desirable for IPv6 capable hosts


::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

root@pm1:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M     12   2013-02-21 17:23:40  pm1
   2   M     20   2013-02-21 17:24:15  pm2

root@pm2:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M     20   2013-02-21 17:24:16  pm1
   2   M     20   2013-02-21 17:24:16  pm2

This is the only weird thing I notice, each node only shows itself in the .members file (I have restarted pvestatd):

root@pm1:~# cat /etc/pve/.members{
"nodename": "pm1",
"version": 0
}

root@pm2:~# cat /etc/pve/.members{
"nodename": "pm2",
"version": 0
}

Any help would be greatly appreciated!

 

[dietmar]

Does it help if you restart pve-cluster file system:

# service pve-cluster restart

What is the output of 'cat /etc/pve/.members' after that?

 

[mstocker]

Nothing changed:

root@pm1:~# service pve-cluster restart
Restarting pve cluster filesystem: pve-cluster.
root@pm1:~# cat /etc/pve/.members
{
"nodename": "pm1",
"version": 0
}

root@pm2:~# service pve-cluster restart
Restarting pve cluster filesystem: pve-cluster.
root@pm2:~# cat /etc/pve/.members
{
"nodename": "pm2",
"version": 0
}

 

[mstocker]

Re: Two Node cluster, each node only sees itself "online" in GUI [SOLVED]

Figured out the problem. For some reason /etc/pve/cluster.conf was missing on both nodes? I'm not sure how this could have happened? I created a new cluster.conf with the following contents:

<?xml version="1.0"?><cluster name="PMCLUST" config_version="2">


  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>


  <clusternodes>
  <clusternode name="pm1" votes="1" nodeid="1"></clusternode>
  <clusternode name="pm2" votes="1" nodeid="2"></clusternode>
  </clusternodes>


</cluster>

Restart pve-cluster and then /etc/pve/.members is as follows:

{"nodename": "pm2",
"version": 4,
"cluster": { "name": "PMCLUST", "version": 2, "nodes": 2, "quorate": 1 },
"nodelist": {
  "pm1": { "id": 1, "online": 1, "ip": "172.16.1.1"},
  "pm2": { "id": 2, "online": 1, "ip": "172.16.1.2"}
  }
}