Proxmox Join cluster frozen in "waiting for quorum"

[Brandon222]

I've created HOST1 a proxmox machine and created several VMs on it. I needed another hosts, so I've created HOST2. I created a cluster, lets call it cluster1. When I've attempted to join HOST2 to the cluster it freezes on waiting for quorum. I've read that proxmox really needs 3 votes to work, so I've added 2 votes to HOST1 and still nothing.

I've shutdown HOST2, and removed it from the cluster, and attempted to add it again, and still the same issue exists, "waiting on quorum."

Any ideas on what to do?

 

[dietmar]

You need working multicast. Check with omping ...

 

[Brandon222]

You need working multicast. Check with omping ...

mulitcast is now functioning, node still hangs on quorum when trying to add.

 

[Brandon222]

ping works
OMPING works
I can SSH into each box, from my desk and from EACH node
I tested joining in a lab enviroment with two new machines and it functioned fine with an unmanaged switch.
I'm using a netgear M4100-50G.

Any hints as to how to get these nodes talking within a cluster would be appreciated.

 

[Brandon222]

I reinstall Proxmox on NODE2, and made some switch changes, now I'm getting a different error message. Corosync failed to start.


root@HOST2:~# systemctl restart corosync.service nc.service
Failed to restart nc.service: Unit nc.service not found.
Job for corosync.service failed because the control process exited with error code.
See "systemctl status corosync.service" and "journalctl -xe" for details.
root@HOST2:~# nano /etc/hosts
root@HOST2:~# systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2018-08-03 15:55:02 CDT; 25s ago
Docs: man:corosync
man:corosync.conf
man:corosync_overview
Process: 9417 ExecStart=/usr/sbin/corosync -f $COROSYNC_OPTIONS (code=exited, status=20)
Main PID: 9417 (code=exited, status=20)
CPU: 121ms

Aug 03 15:55:02 HOST2 corosync[9417]: info [WD ] no resources configured.
Aug 03 15:55:02 HOST2 corosync[9417]: notice [SERV ] Service engine loaded: corosync watchdog service [7]
Aug 03 15:55:02 HOST2 corosync[9417]: notice [QUORUM] Using quorum provider corosync_votequorum
Aug 03 15:55:02 HOST2 corosync[9417]: crit [QUORUM] Quorum provider: corosync_votequorum failed to initialize.
Aug 03 15:55:02 HOST2 corosync[9417]: error [SERV ] Service engine 'corosync_quorum' failed to load for reason 'configuration error: nodelist or quorum.expected_votes must be configured!'
Aug 03 15:55:02 HOST2 corosync[9417]: error [MAIN ] Corosync Cluster Engine exiting with status 20 at service.c:356.
Aug 03 15:55:02 HOST2 systemd[1]: corosync.service: Main process exited, code=exited, status=20/n/a
Aug 03 15:55:02 HOST2 systemd[1]: Failed to start Corosync Cluster Engine.
Aug 03 15:55:02 HOST2 systemd[1]: corosync.service: Unit entered failed state.
Aug 03 15:55:02 HOST2 systemd[1]: corosync.service: Failed with result 'exit-code'.
root@pianola:~# journalctl -xe
Aug 03 15:55:02 HOST2 corosync[9417]: [QB ] server name: cfg
Aug 03 15:55:02 HOST2 corosync[9417]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Aug 03 15:55:02 HOST2 corosync[9417]: [QB ] server name: cpg
Aug 03 15:55:02 HOST2 corosync[9417]: [SERV ] Service engine loaded: corosync profile loading service [4]
Aug 03 15:55:02 HOST2 corosync[9417]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Aug 03 15:55:02 HOST2 corosync[9417]: [WD ] Watchdog /dev/watchdog exists but couldn't be opened.
Aug 03 15:55:02 HOST2 corosync[9417]: [WD ] resource load_15min missing a recovery key.
Aug 03 15:55:02 HOST2 corosync[9417]: [WD ] resource memory_used missing a recovery key.
Aug 03 15:55:02 HOST2 corosync[9417]: [WD ] no resources configured.
Aug 03 15:55:02 HOST2 corosync[9417]: [SERV ] Service engine loaded: corosync watchdog service [7]
Aug 03 15:55:02 HOST2 corosync[9417]: [QUORUM] Using quorum provider corosync_votequorum
Aug 03 15:55:02 HOST2 corosync[9417]: [QUORUM] Quorum provider: corosync_votequorum failed to initialize.
Aug 03 15:55:02 HOST2 corosync[9417]: [SERV ] Service engine 'corosync_quorum' failed to load for reason 'configuration error: nodelist or quorum.expected_votes must be configured!'
Aug 03 15:55:02 HOST2 corosync[9417]: [MAIN ] Corosync Cluster Engine exiting with status 20 at service.c:356.
Aug 03 15:55:02 HOST2 systemd[1]: corosync.service: Main process exited, code=exited, status=20/n/a
Aug 03 15:55:02 HOST2 systemd[1]: Failed to start Corosync Cluster Engine.
-- Subject: Unit corosync.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit corosync.service has failed.
--
-- The result is failed.
Aug 03 15:55:02 HOST2 systemd[1]: corosync.service: Unit entered failed state.
Aug 03 15:55:02 HOST2 systemd[1]: corosync.service: Failed with result 'exit-code'.
Aug 03 15:55:04 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:04 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:04 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:04 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:10 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:10 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:10 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:10 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:16 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:16 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:16 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:16 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:22 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:22 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:22 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:22 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:28 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:28 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:28 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:28 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:34 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:34 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:34 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:34 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:40 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:40 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:40 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:40 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:46 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:46 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:46 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:46 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2
Aug 03 15:55:52 HOST2 pmxcfs[4256]: [quorum] crit: quorum_initialize failed: 2
Aug 03 15:55:52 HOST2 pmxcfs[4256]: [confdb] crit: cmap_initialize failed: 2
Aug 03 15:55:52 HOST2 pmxcfs[4256]: [dcdb] crit: cpg_initialize failed: 2
Aug 03 15:55:52 HOST2 pmxcfs[4256]: [status] crit: cpg_initialize failed: 2

 

[dietmar]

Error in you corosync configuration file (I guess manually edited?)

 

[Brandon222]

Error in you corosync configuration file (I guess manually edited?)

I did everything from the GUI.....

What should be in the file?

 

[dietmar]

What should be in the file?

See error message:

Aug 03 15:55:02 HOST2 corosync[9417]: [SERV ] Service engine 'corosync_quorum' failed to load for reason 'configuration error: nodelist or quorum.expected_votes must be configured!'