Creating a cluster broke both hosts

M

Microkernel

New Member
Mar 31, 2019
14
0
1
24
I just created a cluster on host1 which already had a couple vm's including the local dns servers. After adding the second host which was a completely fresh install to the cluster, both hosts are listed under the datacenter tab, but host2 is offline on host1 and host2 is offline on host1. I've attempted to restart both of them, but now I cannot start any WM's on host1 as I get this error.
Code: 
cluster not ready - no quorum? (500)
Everything is down right now and I'm unsure why the cluster is failing.

As far as I understand my problem is that I only have two hosts and the vote is failing.
How can I fix this ? Can I somehow force host1 to be the master ? As far as I understood I though I only needed more than two hosts for HA.
 
I just reinstalled both nodes while giving one of the two node1 two votes. VM's run fine on node 1 now, but node1 still shows node2 as disconnected and node2 thinks node1 is disconnected.

pvecm status returns

node1
Code: 
Quorum information
------------------
Date:             Mon Apr  8 01:06:24 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/1232
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2  
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 10.0.0.50 (local)

node2
Code: 
Quorum information
------------------
Date:             Mon Apr  8 01:06:18 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000002
Ring ID:          2/1952
Quorate:          No

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      1
Quorum:           2 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 10.0.0.51 (local)
 
Please check the reference documentation on PVE-Clusters - I would guess that it's a problem with multicast - https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cluster_network

as for getting things running again temporarily - `pvecm expected 1` gives one node a quorum again - but keep in mind that this is only a workaround and that you can create inconsistencies when running this way! (also explained in the 'cluster manager' chapter of the reference documentation)

Hope this helps!
 
Stoiko Ivanov said:
Please check the reference documentation on PVE-Clusters - I would guess that it's a problem with multicast - https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cluster_network

as for getting things running again temporarily - `pvecm expected 1` gives one node a quorum again - but keep in mind that this is only a workaround and that you can create inconsistencies when running this way! (also explained in the 'cluster manager' chapter of the reference documentation)

Hope this helps!
Click to expand...

The servers are on the same subnet and multicast seems to be working perfectly fine.

Code: 
10.0.0.51 :   unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.083/0.123/0.260/0.009
10.0.0.51 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.088/0.142/0.276/0.020

Code: 
10.0.0.50 :   unicast, xmt/rcv/%loss = 9985/9985/0%, min/avg/max/std-dev = 0.093/0.135/0.279/0.021
10.0.0.50 : multicast, xmt/rcv/%loss = 9985/9918/0% (seq>=68 0%), min/avg/max/std-dev = 0.094/0.136/0.293/0.024

pvecm expected 1 didn't change anything either
 
I still haven't been able to fix this. Am I missing something ?

node1 corosync.config
Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: bank
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.51
  }
  node {
    name: blume
    nodeid: 1
    quorum_votes: 2
    ring0_addr: 10.0.0.50
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: CL-01
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.50
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

node2 corosync.conf
Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: bank
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.51
  }
  node {
    name: blume
    nodeid: 1
    quorum_votes: 2
    ring0_addr: 10.0.0.50
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: CL-01
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.50
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

mode1 pvecm status
Code: 
Quorum information
------------------
Date:             Fri Apr 19 20:00:04 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/1232
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2  
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 10.0.0.50 (local)

node2 pvecm status
Code: 
Quorum information
------------------
Date:             Fri Apr 19 20:00:10 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000002
Ring ID:          2/1952
Quorate:          No

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      1
Quorum:           2 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 10.0.0.51 (local)
 
please run both tests described in https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cluster_network and post the complete output
it's still possible that you don't have a multicast querier on your network, thus resulting in multicast packets being dropped eventually (that's what the second tests shows)

also please check the journal for messages from corosync and pmxcfs- `journalctl -u corosync.service`, `journalctl -u pve-cluster.service`

It seems as if node1 never saw node2 (Expected votes 2).

Hope this helps!
 
Stoiko Ivanov said:
please run both tests described in https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_cluster_network and post the complete output
it's still possible that you don't have a multicast querier on your network, thus resulting in multicast packets being dropped eventually (that's what the second tests shows)

also please check the journal for messages from corosync and pmxcfs- `journalctl -u corosync.service`, `journalctl -u pve-cluster.service`

It seems as if node1 never saw node2 (Expected votes 2).

Hope this helps!
Click to expand...
Thanks, seems like you're right.

Output on both nodes from the first test
Code: 
omping -c 10000 -i 0.001 -F -q 10.0.0.50 10.0.0.51
10.0.0.51 : waiting for response msg
10.0.0.51 : waiting for response msg
10.0.0.51 : waiting for response msg
10.0.0.51 : waiting for response msg
10.0.0.51 : joined (S,G) = (*, 232.43.211.234), pinging
10.0.0.51 : waiting for response msg
10.0.0.51 : server told us to stop

10.0.0.51 :   unicast, xmt/rcv/%loss = 9098/9098/0%, min/avg/max/std-dev = 0.105/0.126/0.485/0.012
10.0.0.51 : multicast, xmt/rcv/%loss = 9098/9098/0%, min/avg/max/std-dev = 0.114/0.144/0.221/0.019

omping -c 10000 -i 0.001 -F -q 10.0.0.50 10.0.0.51
10.0.0.50 : waiting for response msg
10.0.0.50 : joined (S,G) = (*, 232.43.211.234), pinging
10.0.0.50 : given amount of query messages was sent

10.0.0.50 :   unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.096/0.131/0.448/0.020
10.0.0.50 : multicast, xmt/rcv/%loss = 10000/9935/0% (seq>=66 0%), min/avg/max/std-dev = 0.099/0.135/0.460/0.022

Output of the second test on both nodes
Code: 
omping -c 600 -i 1 -q 10.0.0.50 10.0.0.51
10.0.0.51 : waiting for response msg
10.0.0.51 : waiting for response msg
10.0.0.51 : waiting for response msg
10.0.0.51 : joined (S,G) = (*, 232.43.211.234), pinging
10.0.0.51 : given amount of query messages was sent

10.0.0.51 :   unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.210/0.272/0.903/0.029
10.0.0.51 : multicast, xmt/rcv/%loss = 600/259/56%, min/avg/max/std-dev = 0.221/0.297/0.879/0.042

omping -c 600 -i 1 -q 10.0.0.50 10.0.0.51
10.0.0.50 : waiting for response msg
10.0.0.50 : joined (S,G) = (*, 232.43.211.234), pinging
10.0.0.50 : given amount of query messages was sent

10.0.0.50 :   unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.149/0.282/0.381/0.039
10.0.0.50 : multicast, xmt/rcv/%loss = 600/260/56% (seq>=2 56%), min/avg/max/std-dev = 0.234/0.295/0.381/0.034

node1
Code: 
journalctl -u corosync.service
Apr 08 00:57:28 blume corosync[1861]:  [TOTEM ] FAILED TO RECEIVE
Apr 08 00:57:29 blume corosync[1861]: notice  [TOTEM ] A new membership (10.0.0.50:1232) was formed. Members left: 2
Apr 08 00:57:29 blume corosync[1861]: notice  [TOTEM ] Failed to receive the leave message. failed: 2
Apr 08 00:57:29 blume corosync[1861]: warning [CPG   ] downlist left_list: 1 received
Apr 08 00:57:29 blume corosync[1861]:  [TOTEM ] A new membership (10.0.0.50:1232) was formed. Members left: 2
Apr 08 00:57:29 blume corosync[1861]: notice  [QUORUM] Members[1]: 1
Apr 08 00:57:29 blume corosync[1861]: notice  [MAIN  ] Completed service synchronization, ready to provide service.
Apr 08 00:57:29 blume corosync[1861]:  [TOTEM ] Failed to receive the leave message. failed: 2
Apr 08 00:57:29 blume corosync[1861]:  [CPG   ] downlist left_list: 1 received
Apr 08 00:57:29 blume corosync[1861]:  [QUORUM] Members[1]: 1
Apr 08 00:57:29 blume corosync[1861]:  [MAIN  ] Completed service synchronization, ready to provide service.

journalctl -u pve-cluster.service
Apr 08 00:52:52 blume systemd[1]: Starting The Proxmox VE cluster filesystem...
Apr 08 00:52:52 blume pmxcfs[1704]: [quorum] crit: quorum_initialize failed: 2
Apr 08 00:52:52 blume pmxcfs[1704]: [quorum] crit: can't initialize service
Apr 08 00:52:52 blume pmxcfs[1704]: [confdb] crit: cmap_initialize failed: 2
Apr 08 00:52:52 blume pmxcfs[1704]: [confdb] crit: can't initialize service
Apr 08 00:52:52 blume pmxcfs[1704]: [dcdb] crit: cpg_initialize failed: 2
Apr 08 00:52:52 blume pmxcfs[1704]: [dcdb] crit: can't initialize service
Apr 08 00:52:52 blume pmxcfs[1704]: [status] crit: cpg_initialize failed: 2
Apr 08 00:52:52 blume pmxcfs[1704]: [status] crit: can't initialize service
Apr 08 00:52:53 blume systemd[1]: Started The Proxmox VE cluster filesystem.
Apr 08 00:52:58 blume pmxcfs[1704]: [status] notice: update cluster info (cluster name  WAL-01, version = 2)
Apr 08 00:52:58 blume pmxcfs[1704]: [status] notice: node has quorum
Apr 08 00:52:58 blume pmxcfs[1704]: [dcdb] notice: members: 1/1704
Apr 08 00:52:58 blume pmxcfs[1704]: [dcdb] notice: all data is up to date
Apr 08 00:52:58 blume pmxcfs[1704]: [status] notice: members: 1/1704
Apr 08 00:52:58 blume pmxcfs[1704]: [status] notice: all data is up to date
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: members: 1/1704, 2/1570
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: starting data syncronisation
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: cpg_send_message retried 1 times
Apr 08 00:57:24 blume pmxcfs[1704]: [status] notice: members: 1/1704, 2/1570
Apr 08 00:57:24 blume pmxcfs[1704]: [status] notice: starting data syncronisation
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: received sync request (epoch 1/1704/00000002)
Apr 08 00:57:24 blume pmxcfs[1704]: [status] notice: received sync request (epoch 1/1704/00000002)
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: received all states
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: leader is 1/1704
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: synced members: 1/1704
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: start sending inode updates
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: sent all (14) updates
Apr 08 00:57:24 blume pmxcfs[1704]: [dcdb] notice: all data is up to date
Apr 08 00:57:24 blume pmxcfs[1704]: [status] notice: received all states
Apr 08 00:57:24 blume pmxcfs[1704]: [status] notice: all data is up to date
Apr 08 00:57:24 blume pmxcfs[1704]: [status] notice: dfsm_deliver_queue: queue length 16
Apr 08 00:57:25 blume pmxcfs[1704]: [status] notice: received log
Apr 08 00:57:29 blume pmxcfs[1704]: [dcdb] notice: members: 1/1704
Apr 08 00:57:29 blume pmxcfs[1704]: [status] notice: members: 1/1704
Apr 08 01:52:53 blume pmxcfs[1704]: [dcdb] notice: data verification successful
.....

node2
Code: 
journalctl -u corosync.service
-- No entries --

journalctl -u pve-cluster.service
-- Logs begin at Fri 2019-04-19 08:36:10 CEST, end at Tue 2019-04-23 16:11:10 CEST. --
Apr 19 08:57:17 bank pmxcfs[1570]: [dcdb] notice: data verification successful
...

Now what are the best networking practices when using proxmox in a cluster ? I currently have the proxmox hosts in the same subnet as my management interfaces for routers/swiches etc ... Is it good practice to have the proxmox management interfaces on their own subnet (ensuring that network allows multicast)? And then have another subnet for storage ? How would you deal with having a cluster spread among different geographical locations ?
 
Microkernel said:
Now what are the best networking practices when using proxmox in a cluster ? I currently have the proxmox hosts in the same subnet as my management interfaces for routers/swiches etc ... Is it good practice to have the proxmox management interfaces on their own subnet (ensuring that network allows multicast)? And then have another subnet for storage ? How would you deal with having a cluster spread among different geographical locations ?
Click to expand...
This is quite well explained in the reference documentation - https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvecm:
* Corosync relies on multicast and needs a very low latency network, which usually implies that it's best to keep it on a physical network of its own (using a separate VLAN is not enough in certain circumstances)
* It's usually best to keep the storage-traffic (lots of bandwidth) separate from corosync (many packets, latency sensitive, but not too much traffic)
* the latency requirements of 2ms are usually something which rules out running a PVE-cluster across multiple geographic locations (although it works in some usecases - see e.g. https://www.proxmox.com/en/proxmox-ve/testimonials/item/nic-at)

Microkernel said:
10.0.0.50 : multicast, xmt/rcv/%loss = 600/260/56% (seq>=2 56%), min/avg/max/std-dev = 0.234/0.295/0.381/0.034
Click to expand...
56% packetloss in the multicast case on one of your nodes, but no unicast packetloss would indicate a problem with your network-config (stormcontrol, anti-ddos, multicast-querier, igmp snooping... ) - see https://pve.proxmox.com/wiki/Multicast_notes (in addition to the reference documentation)

hope this helps!
 
Stoiko Ivanov said:
This is quite well explained in the reference documentation - https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvecm:
* Corosync relies on multicast and needs a very low latency network, which usually implies that it's best to keep it on a physical network of its own (using a separate VLAN is not enough in certain circumstances)
* It's usually best to keep the storage-traffic (lots of bandwidth) separate from corosync (many packets, latency sensitive, but not too much traffic)
* the latency requirements of 2ms are usually something which rules out running a PVE-cluster across multiple geographic locations (although it works in some usecases - see e.g. https://www.proxmox.com/en/proxmox-ve/testimonials/item/nic-at)


56% packetloss in the multicast case on one of your nodes, but no unicast packetloss would indicate a problem with your network-config (stormcontrol, anti-ddos, multicast-querier, igmp snooping... ) - see https://pve.proxmox.com/wiki/Multicast_notes (in addition to the reference documentation)

hope this helps!
Click to expand...
I've moved the proxmox hosts to a differnet vlan with igmp-snooping diabled and things seem to work now. Thanks!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back