How to re-include a node in cluster?

[chupacabra]

Ah, sorry my bad, pmxcfs is managed by the pve-cluster.service, so systemctl stop pve-cluster.service.

no wories, for a second there i thought my cluster was completely hosed... here are the journalctl outputs from the hosts:

PVE01

Apr 18 11:38:07 pve01 systemd[1]: Started Corosync Cluster Engine.
Apr 18 11:55:44 pve01 systemd[1]: Stopping Corosync Cluster Engine...
Apr 18 11:55:44 pve01 corosync-cfgtool[6811]: Shutting down corosync
Apr 18 11:55:44 pve01 corosync[3325]:   [MAIN  ] Node was shut down by a signal
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Unloading all Corosync service engines.
Apr 18 11:55:44 pve01 corosync[3325]:   [QB    ] withdrawing server sockets
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync vote quorum service v1.0
Apr 18 11:55:44 pve01 corosync[3325]:   [CFG   ] Node 1 was shut down by sysadmin
Apr 18 11:55:44 pve01 corosync[3325]:   [QB    ] withdrawing server sockets
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync configuration map access
Apr 18 11:55:44 pve01 corosync[3325]:   [QB    ] withdrawing server sockets
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync configuration service
Apr 18 11:55:44 pve01 corosync[3325]:   [QB    ] withdrawing server sockets
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync cluster closed process group service v1.01
Apr 18 11:55:44 pve01 corosync[3325]:   [QB    ] withdrawing server sockets
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync cluster quorum service v0.1
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync profile loading service
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync resource monitoring service
Apr 18 11:55:44 pve01 corosync[3325]:   [SERV  ] Service engine unloaded: corosync watchdog service
Apr 18 11:55:45 pve01 corosync[3325]:   [KNET  ] link: Resetting MTU for link 0 because host 1 joined
Apr 18 11:55:45 pve01 corosync[3325]:   [MAIN  ] Corosync Cluster Engine exiting normally
Apr 18 11:55:45 pve01 systemd[1]: corosync.service: Succeeded.
Apr 18 11:55:45 pve01 systemd[1]: Stopped Corosync Cluster Engine.
Apr 18 11:55:45 pve01 systemd[1]: corosync.service: Consumed 3.484s CPU time.
Apr 18 11:56:09 pve01 systemd[1]: Starting Corosync Cluster Engine...
Apr 18 11:56:09 pve01 corosync[6820]:   [MAIN  ] Corosync Cluster Engine 3.1.7 starting up
Apr 18 11:56:09 pve01 corosync[6820]:   [MAIN  ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Apr 18 11:56:09 pve01 corosync[6820]:   [TOTEM ] Initializing transport (Kronosnet).
Apr 18 11:56:09 pve01 corosync[6820]:   [TOTEM ] totemknet initialized
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] pmtud: MTU manually set to: 0
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync configuration map access [0]
Apr 18 11:56:09 pve01 corosync[6820]:   [QB    ] server name: cmap
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync configuration service [1]
Apr 18 11:56:09 pve01 corosync[6820]:   [QB    ] server name: cfg
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Apr 18 11:56:09 pve01 corosync[6820]:   [QB    ] server name: cpg
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync profile loading service [4]
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync resource monitoring service [6]
Apr 18 11:56:09 pve01 corosync[6820]:   [WD    ] Watchdog not enabled by configuration
Apr 18 11:56:09 pve01 corosync[6820]:   [WD    ] resource load_15min missing a recovery key.
Apr 18 11:56:09 pve01 corosync[6820]:   [WD    ] resource memory_used missing a recovery key.
Apr 18 11:56:09 pve01 corosync[6820]:   [WD    ] no resources configured.
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync watchdog service [7]
Apr 18 11:56:09 pve01 corosync[6820]:   [QUORUM] Using quorum provider corosync_votequorum
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
Apr 18 11:56:09 pve01 corosync[6820]:   [QB    ] server name: votequorum
Apr 18 11:56:09 pve01 corosync[6820]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Apr 18 11:56:09 pve01 corosync[6820]:   [QB    ] server name: quorum
Apr 18 11:56:09 pve01 corosync[6820]:   [TOTEM ] Configuring link 0
Apr 18 11:56:09 pve01 corosync[6820]:   [TOTEM ] Configured link number 0: local addr: 10.10.100.100, port=5405
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] link: Resetting MTU for link 0 because host 1 joined
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 0)
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 4 has no active links
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 4 has no active links
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Apr 18 11:56:09 pve01 corosync[6820]:   [KNET  ] host: host: 4 has no active links
Apr 18 11:56:09 pve01 corosync[6820]:   [QUORUM] Sync members[1]: 1
Apr 18 11:56:09 pve01 systemd[1]: Started Corosync Cluster Engine.
Apr 18 11:56:09 pve01 corosync[6820]:   [QUORUM] Sync joined[1]: 1
Apr 18 11:56:09 pve01 corosync[6820]:   [TOTEM ] A new membership (1.a22) was formed. Members joined: 1
Apr 18 11:56:09 pve01 corosync[6820]:   [QUORUM] Members[1]: 1
Apr 18 11:56:09 pve01 corosync[6820]:   [MAIN  ] Completed service synchronization, ready to provide service.

 

[Chris]

nodelist.node.0.ring0_addr (str) = 10.10.1.100

The nodelist of the quorate corosync nodes still contained the incorrect IP. So please doublecheck the /etc/corosync/corosync.conf and that this is fixed after restarting the corosync services.

 

[chupacabra]

PVE02

Apr 18 11:55:35 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:36 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:37 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:38 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:39 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:40 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:41 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:42 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:43 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:44 pve02 corosync[2138]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:47 pve02 systemd[1]: Stopping Corosync Cluster Engine...
Apr 18 11:55:47 pve02 corosync-cfgtool[84585]: Shutting down corosync
Apr 18 11:55:47 pve02 corosync[2138]:   [MAIN  ] Node was shut down by a signal
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Unloading all Corosync service engines.
Apr 18 11:55:47 pve02 corosync[2138]:   [QB    ] withdrawing server sockets
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync vote quorum service v1.0
Apr 18 11:55:47 pve02 corosync[2138]:   [CFG   ] Node 2 was shut down by sysadmin
Apr 18 11:55:47 pve02 corosync[2138]:   [QB    ] withdrawing server sockets
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync configuration map access
Apr 18 11:55:47 pve02 corosync[2138]:   [QB    ] withdrawing server sockets
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync configuration service
Apr 18 11:55:47 pve02 corosync[2138]:   [QB    ] withdrawing server sockets
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync cluster closed process group service v1.01
Apr 18 11:55:47 pve02 corosync[2138]:   [QB    ] withdrawing server sockets
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync cluster quorum service v0.1
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync profile loading service
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync resource monitoring service
Apr 18 11:55:47 pve02 corosync[2138]:   [SERV  ] Service engine unloaded: corosync watchdog service
Apr 18 11:55:48 pve02 corosync[2138]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Apr 18 11:55:48 pve02 corosync[2138]:   [KNET  ] link: Resetting MTU for link 0 because host 2 joined
Apr 18 11:55:48 pve02 corosync[2138]:   [MAIN  ] Corosync Cluster Engine exiting normally
Apr 18 11:55:48 pve02 systemd[1]: corosync.service: Succeeded.
Apr 18 11:55:48 pve02 systemd[1]: Stopped Corosync Cluster Engine.
Apr 18 11:55:48 pve02 systemd[1]: corosync.service: Consumed 12.618s CPU time.
Apr 18 11:56:11 pve02 systemd[1]: Starting Corosync Cluster Engine...
Apr 18 11:56:11 pve02 corosync[85021]:   [MAIN  ] Corosync Cluster Engine 3.1.7 starting up
Apr 18 11:56:11 pve02 corosync[85021]:   [MAIN  ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Apr 18 11:56:11 pve02 corosync[85021]:   [TOTEM ] Initializing transport (Kronosnet).
Apr 18 11:56:11 pve02 corosync[85021]:   [TOTEM ] totemknet initialized
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] pmtud: MTU manually set to: 0
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync configuration map access [0]
Apr 18 11:56:11 pve02 corosync[85021]:   [QB    ] server name: cmap
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync configuration service [1]
Apr 18 11:56:11 pve02 corosync[85021]:   [QB    ] server name: cfg
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Apr 18 11:56:11 pve02 corosync[85021]:   [QB    ] server name: cpg
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync profile loading service [4]
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync resource monitoring service [6]
Apr 18 11:56:11 pve02 corosync[85021]:   [WD    ] Watchdog not enabled by configuration
Apr 18 11:56:11 pve02 corosync[85021]:   [WD    ] resource load_15min missing a recovery key.
Apr 18 11:56:11 pve02 corosync[85021]:   [WD    ] resource memory_used missing a recovery key.
Apr 18 11:56:11 pve02 corosync[85021]:   [WD    ] no resources configured.
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync watchdog service [7]
Apr 18 11:56:11 pve02 corosync[85021]:   [QUORUM] Using quorum provider corosync_votequorum
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
Apr 18 11:56:11 pve02 corosync[85021]:   [QB    ] server name: votequorum
Apr 18 11:56:11 pve02 corosync[85021]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Apr 18 11:56:11 pve02 corosync[85021]:   [QB    ] server name: quorum
Apr 18 11:56:11 pve02 corosync[85021]:   [TOTEM ] Configuring link 0
Apr 18 11:56:11 pve02 corosync[85021]:   [TOTEM ] Configured link number 0: local addr: 10.10.100.105, port=5405
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 0)
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 1 has no active links
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 1 has no active links
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 1 has no active links
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 0)
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] link: Resetting MTU for link 0 because host 2 joined
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 4 has no active links
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 4 has no active links
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] host: host: 4 has no active links
Apr 18 11:56:11 pve02 corosync[85021]:   [QUORUM] Sync members[1]: 2
Apr 18 11:56:11 pve02 corosync[85021]:   [QUORUM] Sync joined[1]: 2
Apr 18 11:56:11 pve02 corosync[85021]:   [TOTEM ] A new membership (2.a17) was formed. Members joined: 2
Apr 18 11:56:11 pve02 corosync[85021]:   [QUORUM] Members[1]: 2
Apr 18 11:56:11 pve02 corosync[85021]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr 18 11:56:11 pve02 systemd[1]: Started Corosync Cluster Engine.
Apr 18 11:56:11 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:12 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:13 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:14 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:15 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:15 pve02 corosync[85021]:   [KNET  ] rx: host: 4 link: 0 is up
Apr 18 11:56:15 pve02 corosync[85021]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Apr 18 11:56:15 pve02 corosync[85021]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Apr 18 11:56:15 pve02 corosync[85021]:   [QUORUM] Sync members[2]: 2 4
Apr 18 11:56:15 pve02 corosync[85021]:   [QUORUM] Sync joined[1]: 4
Apr 18 11:56:15 pve02 corosync[85021]:   [TOTEM ] A new membership (2.a1f) was formed. Members joined: 4
Apr 18 11:56:15 pve02 corosync[85021]:   [QUORUM] This node is within the primary component and will provide service.
Apr 18 11:56:15 pve02 corosync[85021]:   [QUORUM] Members[2]: 2 4
Apr 18 11:56:15 pve02 corosync[85021]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr 18 11:56:15 pve02 corosync[85021]:   [KNET  ] pmtud: PMTUD link change for host: 4 link: 0 from 469 to 1397
Apr 18 11:56:15 pve02 corosync[85021]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Apr 18 11:56:16 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:17 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:18 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:19 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:20 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:21 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:22 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:23 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:24 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:25 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:26 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:27 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:28 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:29 pve02 corosync[85021]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405

 

[chupacabra]

PVE03

Apr 18 11:55:43 pve03 corosync[2038]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:44 pve03 corosync[2038]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:55:47 pve03 corosync[2038]:   [CFG   ] Node 2 was shut down by sysadmin
Apr 18 11:55:47 pve03 corosync[2038]:   [QUORUM] Sync members[1]: 4
Apr 18 11:55:47 pve03 corosync[2038]:   [QUORUM] Sync left[1]: 2
Apr 18 11:55:47 pve03 corosync[2038]:   [TOTEM ] A new membership (4.a16) was formed. Members left: 2
Apr 18 11:55:47 pve03 corosync[2038]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Apr 18 11:55:47 pve03 corosync[2038]:   [QUORUM] Members[1]: 4
Apr 18 11:55:47 pve03 corosync[2038]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr 18 11:55:48 pve03 corosync[2038]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:55:48 pve03 corosync[2038]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:55:48 pve03 corosync[2038]:   [KNET  ] link: host: 2 link: 0 is down
Apr 18 11:55:48 pve03 corosync[2038]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:55:48 pve03 corosync[2038]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:55:50 pve03 systemd[1]: Stopping Corosync Cluster Engine...
Apr 18 11:55:50 pve03 corosync-cfgtool[148727]: Shutting down corosync
Apr 18 11:55:50 pve03 corosync[2038]:   [MAIN  ] Node was shut down by a signal
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Unloading all Corosync service engines.
Apr 18 11:55:50 pve03 corosync[2038]:   [QB    ] withdrawing server sockets
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync vote quorum service v1.0
Apr 18 11:55:50 pve03 corosync[2038]:   [CFG   ] Node 4 was shut down by sysadmin
Apr 18 11:55:50 pve03 corosync[2038]:   [QB    ] withdrawing server sockets
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync configuration map access
Apr 18 11:55:50 pve03 corosync[2038]:   [QB    ] withdrawing server sockets
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync configuration service
Apr 18 11:55:50 pve03 corosync[2038]:   [QB    ] withdrawing server sockets
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync cluster closed process group service v1.01
Apr 18 11:55:50 pve03 corosync[2038]:   [QB    ] withdrawing server sockets
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync cluster quorum service v0.1
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync profile loading service
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync resource monitoring service
Apr 18 11:55:50 pve03 corosync[2038]:   [SERV  ] Service engine unloaded: corosync watchdog service
Apr 18 11:55:51 pve03 corosync[2038]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Apr 18 11:55:51 pve03 corosync[2038]:   [MAIN  ] Corosync Cluster Engine exiting normally
Apr 18 11:55:51 pve03 systemd[1]: corosync.service: Succeeded.
Apr 18 11:55:51 pve03 systemd[1]: Stopped Corosync Cluster Engine.
Apr 18 11:55:51 pve03 systemd[1]: corosync.service: Consumed 13.977s CPU time.
Apr 18 11:56:13 pve03 systemd[1]: Starting Corosync Cluster Engine...
Apr 18 11:56:13 pve03 corosync[149470]:   [MAIN  ] Corosync Cluster Engine 3.1.7 starting up
Apr 18 11:56:13 pve03 corosync[149470]:   [MAIN  ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Apr 18 11:56:13 pve03 corosync[149470]:   [TOTEM ] Initializing transport (Kronosnet).
Apr 18 11:56:13 pve03 corosync[149470]:   [TOTEM ] totemknet initialized
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] pmtud: MTU manually set to: 0
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync configuration map access [0]
Apr 18 11:56:13 pve03 corosync[149470]:   [QB    ] server name: cmap
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync configuration service [1]
Apr 18 11:56:13 pve03 corosync[149470]:   [QB    ] server name: cfg
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Apr 18 11:56:13 pve03 corosync[149470]:   [QB    ] server name: cpg
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync profile loading service [4]
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync resource monitoring service [6]
Apr 18 11:56:13 pve03 corosync[149470]:   [WD    ] Watchdog not enabled by configuration
Apr 18 11:56:13 pve03 corosync[149470]:   [WD    ] resource load_15min missing a recovery key.
Apr 18 11:56:13 pve03 corosync[149470]:   [WD    ] resource memory_used missing a recovery key.
Apr 18 11:56:13 pve03 corosync[149470]:   [WD    ] no resources configured.
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync watchdog service [7]
Apr 18 11:56:13 pve03 corosync[149470]:   [QUORUM] Using quorum provider corosync_votequorum
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
Apr 18 11:56:13 pve03 corosync[149470]:   [QB    ] server name: votequorum
Apr 18 11:56:13 pve03 corosync[149470]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Apr 18 11:56:13 pve03 corosync[149470]:   [QB    ] server name: quorum
Apr 18 11:56:13 pve03 corosync[149470]:   [TOTEM ] Configuring link 0
Apr 18 11:56:13 pve03 corosync[149470]:   [TOTEM ] Configured link number 0: local addr: 10.10.100.110, port=5405
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 1 has no active links
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 1 has no active links
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 1 has no active links
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] host: host: 2 has no active links
Apr 18 11:56:13 pve03 corosync[149470]:   [QUORUM] Sync members[1]: 4
Apr 18 11:56:13 pve03 corosync[149470]:   [QUORUM] Sync joined[1]: 4
Apr 18 11:56:13 pve03 corosync[149470]:   [TOTEM ] A new membership (4.a1b) was formed. Members joined: 4
Apr 18 11:56:13 pve03 corosync[149470]:   [QUORUM] Members[1]: 4
Apr 18 11:56:13 pve03 corosync[149470]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr 18 11:56:13 pve03 systemd[1]: Started Corosync Cluster Engine.
Apr 18 11:56:13 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:14 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:15 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:15 pve03 corosync[149470]:   [KNET  ] rx: host: 2 link: 0 is up
Apr 18 11:56:15 pve03 corosync[149470]:   [KNET  ] link: Resetting MTU for link 0 because host 2 joined
Apr 18 11:56:15 pve03 corosync[149470]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Apr 18 11:56:15 pve03 corosync[149470]:   [QUORUM] Sync members[2]: 2 4
Apr 18 11:56:15 pve03 corosync[149470]:   [QUORUM] Sync joined[1]: 2
Apr 18 11:56:15 pve03 corosync[149470]:   [TOTEM ] A new membership (2.a1f) was formed. Members joined: 2
Apr 18 11:56:15 pve03 corosync[149470]:   [QUORUM] This node is within the primary component and will provide service.
Apr 18 11:56:15 pve03 corosync[149470]:   [QUORUM] Members[2]: 2 4
Apr 18 11:56:15 pve03 corosync[149470]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr 18 11:56:15 pve03 corosync[149470]:   [KNET  ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397
Apr 18 11:56:15 pve03 corosync[149470]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Apr 18 11:56:16 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:17 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:18 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:19 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:20 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:21 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:22 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:23 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405
Apr 18 11:56:24 pve03 corosync[149470]:   [KNET  ] rx: Packet rejected from 10.10.100.100:5405

 

[Chris]

The nodelist of the quorate corosync nodes still contained the incorrect IP. So please doublecheck the /etc/corosync/corosync.conf and that this is fixed after restarting the corosync services.

Did you catch my reply from before? there still is an incorrect IP floating around in your config files on node PVE02 and PVE03.

 

[chupacabra]

hi, i just did.. and thank you!!!! that finally fixed it. i thought the changes on the /etc/pve/corosync.conf propagate to the
/etc/corosync/corosync.conf and not the other way around. changing that in the first one seems to have fixed it. i sincerely appreciate all the help and patience with me.