Cluster running, but failing to connect to resource group manager

D

dallen1

New Member
Jun 28, 2012
6
0
1
I'm getting the following message for all my HA managed systems. I've tried restarting all the services as well as rebooting each node. It's not just migrating that causing the error - all HA commands have the same failure to connect to the resource manager. The weird thing is that everything regarding the cluster seems to be reporting as okay. Where do I go from here to get HA running again?


error message:
Code: 
Executing HA migrate for VM 112 to node pve-02
Trying to migrate pvevm:112 to pve-02...Could not connect to resource group manager
TASK ERROR: command 'clusvcadm -M pvevm:112 -m pve-02' failed: exit code 1


Here's the cluster info:


PVE Version info
Code: 
root@pve-01:/etc/pve# pveversion -v
proxmox-ve-2.6.32: 3.3-147 (running kernel: 2.6.32-37-pve)
pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-37-pve: 2.6.32-147
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-34-pve: 2.6.32-140
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.3-20
pve-firmware: 1.1-3
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-31
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-12
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1


root@pve-01:/etc/pve# ssh pve-02 pveversion -v
proxmox-ve-2.6.32: 3.3-147 (running kernel: 2.6.32-37-pve)
pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-37-pve: 2.6.32-147
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-34-pve: 2.6.32-140
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.3-20
pve-firmware: 1.1-3
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-31
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-12
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1


root@pve-01:/etc/pve# ssh pve-03 pveversion -v
proxmox-ve-2.6.32: 3.3-147 (running kernel: 2.6.32-37-pve)
pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-37-pve: 2.6.32-147
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-34-pve: 2.6.32-140
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.3-20
pve-firmware: 1.1-3
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-31
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-12
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
root@pve-01:/etc/pve#


PVECM status
Code: 
root@pve-01:/etc/pve# pvecm status
Version: 6.2.0
Config Version: 76
Cluster Name: CLUSTER
Cluster Id: 9300
Cluster Member: Yes
Cluster Generation: 1376
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2  
Active subsystems: 6
Flags: 
Ports Bound: 0  
Node name: pve-01
Node ID: 1
Multicast addresses: 239.192.36.120 
Node addresses: 10.10.164.9 


root@pve-01:/etc/pve# ssh pve-02 pvecm status
Version: 6.2.0
Config Version: 76
Cluster Name: CLUSTER
Cluster Id: 9300
Cluster Member: Yes
Cluster Generation: 1376
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2  
Active subsystems: 6
Flags: 
Ports Bound: 0  
Node name: pve-02
Node ID: 2
Multicast addresses: 239.192.36.120 
Node addresses: 10.10.164.10 


root@pve-01:/etc/pve# ssh pve-03 pvecm status
Version: 6.2.0
Config Version: 76
Cluster Name: CLUSTER
Cluster Id: 9300
Cluster Member: Yes
Cluster Generation: 1376
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2  
Active subsystems: 6
Flags: 
Ports Bound: 0  
Node name: pve-03
Node ID: 3
Multicast addresses: 239.192.36.120 
Node addresses: 10.10.164.11 
root@pve-01:/etc/pve#


/etc/pve/cluster.conf:
Code: 
<?xml version="1.0"?>
<cluster config_version="76" name="CLUSTER">
  <cman keyfile="/var/lib/pve-cluster/corosync.authkey"/>
  <fencedevices>
    <fencedevice agent="fence_ipmilan" ipaddr="10.10.164.19" lanplus="1" login="pve" name="node1" passwd="secret" power_wait="5" shell_timeout="300"/>
    <fencedevice agent="fence_ipmilan" ipaddr="10.10.164.20" lanplus="1" login="pve" name="node2" passwd="secret" power_wait="5" shell_timeout="300"/>
    <fencedevice agent="fence_ipmilan" ipaddr="10.10.164.18" lanplus="1" login="pve" name="node3" passwd="secret" power_wait="5" shell_timeout="300"/>
  </fencedevices>
  <clusternodes>
    <clusternode name="pve-01" nodeid="1" votes="1">
      <fence>
        <method name="1">
          <device name="node1"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="pve-02" nodeid="2" votes="1">
      <fence>
        <method name="1">
          <device name="node2"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="pve-03" nodeid="3" votes="1">
      <fence>
        <method name="1">
          <device name="node3"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <rm>
    <pvevm autostart="1" vmid="101"/>
    <pvevm autostart="1" vmid="102"/>
    <pvevm autostart="1" vmid="110"/>
    <pvevm autostart="1" vmid="113"/>
    <pvevm autostart="1" vmid="112"/>
  </rm>
</cluster>


rgmanager status:
Code: 
root@pve-01:/etc/pve# service rgmanager status
rgmanager (pid 3426 3425) is running...
root@pve-01:/etc/pve# ssh pve-02 "service rgmanager status"
rgmanager (pid 3449 3448) is running...
root@pve-01:/etc/pve# ssh pve-03 "service rgmanager status"
rgmanager (pid 3437 3436) is running...
root@pve-01:/etc/pve#


fencetool outputs:
Code: 
root@pve-01:/etc/pve# fence_tool ls
fence domain
member count  3
victim count  0
victim now    0
master nodeid 2
wait state    none
members       1 2 3 


root@pve-01:/etc/pve# ssh pve-02 fence_tool ls
fence domain
member count  3
victim count  0
victim now    0
master nodeid 2
wait state    none
members       1 2 3 


root@pve-01:/etc/pve# ssh pve-03 fence_tool ls
fence domain
member count  3
victim count  0
victim now    0
master nodeid 2
wait state    none
members       1 2 3




Cluster Status:
Code: 
root@pve-01:/etc/pve# clustat 
Cluster Status for CLUSTER @ Wed Feb 25 08:20:16 2015
Member Status: Quorate


 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 pve-01                                                              1 Online, Local
 pve-02                                                              2 Online
 pve-03                                                              3 Online


root@pve-01:/etc/pve# ssh pve-02 clustat 
Cluster Status for CLUSTER @ Wed Feb 25 08:20:24 2015
Member Status: Quorate


 Member Name                             ID   Status
 ------ ----                             ---- ------
 pve-01                                      1 Online
 pve-02                                      2 Online, Local
 pve-03                                      3 Online


root@pve-01:/etc/pve# ssh pve-03 clustat 
Cluster Status for CLUSTER @ Wed Feb 25 08:20:30 2015
Member Status: Quorate


 Member Name                             ID   Status
 ------ ----                             ---- ------
 pve-01                                      1 Online
 pve-02                                      2 Online
 pve-03                                      3 Online, Local
 
Okay, I think I figured it out. In their infinite wisdom, my orgs networking team must have disabled multicasting and not told me... After reading this thread I tested multicast and found that none of my nodes are replying. http://comments.gmane.org/gmane.linux.redhat.cluster/21911 On a side note, my cluster config is wrong as well. I needed to add cipher=1 for fence-node to work.After I sort out multicasting I'll post back.
 
The issue was with the Cluster Firewall. When I turned it off multicast started working. I thought that local_network group membership automatically configure all necessary services? Regardless, when I added an accept all rule to the cluster firewall for the local_network group then restarted the cluster services HA started working again.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back