old timestamp - dead?

[stats]

I created a cluster that has 5 nodes. My local test was good. so I moved them to my production environment, but 2 nodes were removed from the cluster. How can I re-join them again?

The two nodes are not active.


pxdev01 and pcdev02 is running as standalone.

 

[dcsapak]

does multicast between all nodes work?

https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast

 

[stats]

root@pxmx01:~omping -c 10000 -i 0.001 -F -q pxdev01 pxdev02 pxmx01 pxmx02 pxmx03
pxdev01 : waiting for response msg
pxdev02 : waiting for response msg
pxmx02 : waiting for response msg
pxmx03 : waiting for response msg

It seems it doesn't work. Is multicast necessarily required?

 

[dcsapak]

did you really execute that command on all nodes simultaneously?
and yes multicast is required (there is a method with unicast, but for more than 3 nodes this gets slow and unreliable)

 

[stats]

Sorry, it was not simultaneously. This is the results.

* pxdev01
pxdev02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.088/0.154/0.347/0.039
pxdev02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.093/0.161/0.355/0.044
pxmx01 : unicast, xmt/rcv/%loss = 9894/9894/0%, min/avg/max/std-dev = 0.082/0.121/0.264/0.021
pxmx01 : multicast, xmt/rcv/%loss = 9894/9894/0%, min/avg/max/std-dev = 0.081/0.124/0.288/0.022
pxmx02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.078/0.122/0.271/0.024
pxmx02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.069/0.129/0.312/0.027
pxmx03 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.085/0.127/0.289/0.024
pxmx03 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.078/0.122/0.300/0.028

* pxdev02
pxdev01 : unicast, xmt/rcv/%loss = 8494/8494/0%, min/avg/max/std-dev = 0.092/0.140/0.323/0.018
pxdev01 : multicast, xmt/rcv/%loss = 8494/8494/0%, min/avg/max/std-dev = 0.093/0.147/0.356/0.022
pxmx01 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.065/0.122/0.260/0.021
pxmx01 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.073/0.125/0.272/0.019
pxmx02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.067/0.128/0.267/0.026
pxmx02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.067/0.130/0.310/0.031
pxmx03 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.082/0.126/0.245/0.022
pxmx03 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.069/0.123/0.271/0.027

* pxmx01
pxdev01 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.085/0.124/0.301/0.019
pxdev01 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.081/0.130/0.335/0.020
pxdev02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.073/0.146/0.352/0.044
pxdev02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.079/0.154/0.383/0.050
pxmx02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.078/0.119/0.268/0.019
pxmx02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.066/0.123/0.275/0.021
pxmx03 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.070/0.133/0.255/0.026
pxmx03 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.076/0.141/0.265/0.028

* pxmx02
pxdev01 : unicast, xmt/rcv/%loss = 9563/9563/0%, min/avg/max/std-dev = 0.088/0.131/0.325/0.018
pxdev01 : multicast, xmt/rcv/%loss = 9563/9563/0%, min/avg/max/std-dev = 0.085/0.139/0.343/0.021
pxdev02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.083/0.141/0.350/0.032
pxdev02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.085/0.151/0.374/0.037
pxmx01 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.067/0.120/0.278/0.017
pxmx01 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.070/0.119/0.282/0.018
pxmx03 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.067/0.122/0.255/0.021
pxmx03 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.071/0.125/0.260/0.023

* pxmx03
pxdev01 : unicast, xmt/rcv/%loss = 9397/9397/0%, min/avg/max/std-dev = 0.091/0.137/0.388/0.021
pxdev01 : multicast, xmt/rcv/%loss = 9397/9397/0%, min/avg/max/std-dev = 0.082/0.141/0.388/0.022
pxdev02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.084/0.145/0.399/0.038
pxdev02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.084/0.154/0.363/0.044
pxmx01 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.081/0.132/0.292/0.028
pxmx01 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.065/0.129/0.241/0.027
pxmx02 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.063/0.123/0.400/0.025
pxmx02 : multicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.063/0.133/0.395/0.027

 

[dcsapak]

so multicast in general works, can you also try a "long term" test?

omping -c 600 -i 1 -q NODE1-IP NODE2-IP ...

this takes around 10 minutes and often detects problems with igmp queriers

 

[stats]

It seems no problem. Here is the results.
When I moved the servers to My IDC. pxdev01 and pxdev02 was started at first. and pxmx01, pxmx02 and pxmx03 was started a few hours later. Was it a cause of the problem?

* pcdev01
pxdev02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.126/0.243/0.311/0.036
pxdev02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.133/0.252/0.336/0.038
pxmx01 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.105/0.187/0.298/0.041
pxmx01 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.112/0.197/0.310/0.043
pxmx02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.103/0.187/0.295/0.042
pxmx02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.199/0.303/0.042
pxmx03 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.099/0.190/0.276/0.037
pxmx03 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.109/0.202/0.291/0.040

*pxdev02
pxdev01 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.149/0.282/0.328/0.019
pxdev01 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.179/0.286/0.342/0.018
pxmx01 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.173/0.252/0.307/0.029
pxmx01 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.174/0.230/0.296/0.024
pxmx02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.142/0.230/0.293/0.025
pxmx02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.155/0.250/0.304/0.025
pxmx03 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.163/0.243/0.729/0.034
pxmx03 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.163/0.247/0.742/0.032

* pxmx01
pxdev01 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.135/0.222/0.328/0.047
pxdev01 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.139/0.226/0.319/0.044
pxdev02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.121/0.247/0.304/0.026
pxdev02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.126/0.256/0.313/0.026
pxmx02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.115/0.210/0.281/0.031
pxmx02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.118/0.222/0.307/0.033
pxmx03 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.119/0.206/0.265/0.027
pxmx03 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.112/0.218/0.286/0.033

*pxmx02
pxdev01 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.170/0.255/0.383/0.016
pxdev01 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.178/0.264/0.336/0.014
pxdev02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.141/0.246/0.294/0.013
pxdev02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.173/0.248/0.295/0.013
pxmx01 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.144/0.203/0.262/0.018
pxmx01 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.160/0.216/0.287/0.022
pxmx03 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.170/0.225/0.281/0.021
pxmx03 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.173/0.225/0.285/0.020

*pxmx03
pxdev02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.126/0.243/0.311/0.036
pxdev02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.133/0.252/0.336/0.038
pxmx01 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.105/0.187/0.298/0.041
pxmx01 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.112/0.197/0.310/0.043
pxmx02 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.103/0.187/0.295/0.042
pxmx02 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.199/0.303/0.042
pxmx03 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.099/0.190/0.276/0.037
pxmx03 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.109/0.202/0.291/0.040

 

[dcsapak]

hmm.. is corosync started on all nodes?

systemctl status pve-cluster corosync

 

[stats]

It seems pxdev01 and pxdev02 are failing to start corosync. How can I fix it?
Here is the results.

* pxdev01
root@pxdev01:~# systemctl status pve-cluster corosync
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: active (running) since Mon 2017-03-06 00:51:02 JST; 2 days ago
Process: 1982 ExecStartPost=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 1921 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 1980 (pmxcfs)
CGroup: /system.slice/pve-cluster.service
mq1980 /usr/bin/pmxcfs

Mar 06 21:43:26 pxdev01 pmxcfs[1980]: [dcdb] crit: cpg_initialize failed: 2
Mar 06 21:43:26 pxdev01 pmxcfs[1980]: [status] crit: cpg_initialize failed: 2
Mar 06 21:43:32 pxdev01 pmxcfs[1980]: [quorum] crit: quorum_initialize failed: 2
Mar 06 21:43:32 pxdev01 pmxcfs[1980]: [confdb] crit: cmap_initialize failed: 2
Mar 06 21:43:32 pxdev01 pmxcfs[1980]: [dcdb] crit: cpg_initialize failed: 2
Mar 06 21:43:32 pxdev01 pmxcfs[1980]: [status] crit: cpg_initialize failed: 2
Mar 06 21:43:38 pxdev01 pmxcfs[1980]: [quorum] crit: quorum_initialize failed: 2
Mar 06 21:43:38 pxdev01 pmxcfs[1980]: [confdb] crit: cmap_initialize failed: 2
Mar 06 21:43:38 pxdev01 pmxcfs[1980]: [dcdb] crit: cpg_initialize failed: 2
Mar 06 21:43:38 pxdev01 pmxcfs[1980]: [status] crit: cpg_initialize failed: 2

● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
Active: failed (Result: timeout) since Mon 2017-03-06 00:52:32 JST; 2 days ago
Process: 2031 ExecStart=/usr/share/corosync/corosync start (code=killed, signal=TERM)

Mar 06 00:51:02 pxdev01 systemd[1]: Starting Corosync Cluster Engine...
Mar 06 00:51:02 pxdev01 corosync[2037]: [MAIN ] Corosync Cluster Engine ('2....
Mar 06 00:51:02 pxdev01 corosync[2037]: [MAIN ] Corosync built-in features:...w
Mar 06 00:52:32 pxdev01 systemd[1]: corosync.service start operation timed ...g.
Mar 06 00:52:32 pxdev01 corosync[2031]: Starting Corosync Cluster Engine (co...:
Mar 06 00:52:32 pxdev01 systemd[1]: Failed to start Corosync Cluster Engine.
Mar 06 00:52:32 pxdev01 systemd[1]: Unit corosync.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.

* pxdev02
root@pxdev02:~# systemctl status pve-cluster corosync
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: active (running) since Mon 2017-03-06 00:06:58 JST; 2 days ago
Process: 2007 ExecStartPost=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 1953 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 2005 (pmxcfs)
CGroup: /system.slice/pve-cluster.service
mq2005 /usr/bin/pmxcfs

Mar 06 00:10:33 pxdev02 pmxcfs[2005]: [dcdb] crit: cpg_initialize failed: 2
Mar 06 00:10:33 pxdev02 pmxcfs[2005]: [status] crit: cpg_initialize failed: 2
Mar 06 00:10:39 pxdev02 pmxcfs[2005]: [quorum] crit: quorum_initialize failed: 2
Mar 06 00:10:39 pxdev02 pmxcfs[2005]: [confdb] crit: cmap_initialize failed: 2
Mar 06 00:10:39 pxdev02 pmxcfs[2005]: [dcdb] crit: cpg_initialize failed: 2
Mar 06 00:10:39 pxdev02 pmxcfs[2005]: [status] crit: cpg_initialize failed: 2
Mar 06 00:10:45 pxdev02 pmxcfs[2005]: [quorum] crit: quorum_initialize failed: 2
Mar 06 00:10:45 pxdev02 pmxcfs[2005]: [confdb] crit: cmap_initialize failed: 2
Mar 06 00:10:45 pxdev02 pmxcfs[2005]: [dcdb] crit: cpg_initialize failed: 2
Mar 06 00:10:45 pxdev02 pmxcfs[2005]: [status] crit: cpg_initialize failed: 2

● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
Active: failed (Result: timeout) since Mon 2017-03-06 00:08:28 JST; 2 days ago
Process: 2024 ExecStart=/usr/share/corosync/corosync start (code=killed, signal=TERM)

Mar 06 00:06:58 pxdev02 corosync[2030]: [MAIN ] Corosync Cluster Engine ('2....
Mar 06 00:06:58 pxdev02 corosync[2030]: [MAIN ] Corosync built-in features:...w
Mar 06 00:08:28 pxdev02 systemd[1]: corosync.service start operation timed ...g.
Mar 06 00:08:28 pxdev02 corosync[2024]: Starting Corosync Cluster Engine (co...:
Mar 06 00:08:28 pxdev02 systemd[1]: Failed to start Corosync Cluster Engine.
Mar 06 00:08:28 pxdev02 systemd[1]: Unit corosync.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.

* pxmx01
root@pxmx01:~# systemctl status pve-cluster corosync
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: active (running) since Sun 2017-03-05 23:46:41 JST; 2 days ago
Process: 2579 ExecStartPost=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 2540 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 2576 (pmxcfs)
CGroup: /system.slice/pve-cluster.service
mq2576 /usr/bin/pmxcfs

Mar 08 02:46:40 pxmx01 pmxcfs[2576]: [dcdb] notice: data verification successful
Mar 08 03:46:40 pxmx01 pmxcfs[2576]: [dcdb] notice: data verification successful
Mar 08 03:57:05 pxmx01 pmxcfs[2576]: [status] notice: received log
Mar 08 03:57:12 pxmx01 pmxcfs[2576]: [status] notice: received log
Mar 08 04:46:40 pxmx01 pmxcfs[2576]: [dcdb] notice: data verification successful
Mar 08 05:25:04 pxmx01 pmxcfs[2576]: [status] notice: received log
Mar 08 05:25:10 pxmx01 pmxcfs[2576]: [status] notice: received log
Mar 08 05:46:40 pxmx01 pmxcfs[2576]: [dcdb] notice: data verification successful
Mar 08 06:46:40 pxmx01 pmxcfs[2576]: [dcdb] notice: data verification successful
Mar 08 07:46:40 pxmx01 pmxcfs[2576]: [dcdb] notice: data verification successful

● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
Active: active (running) since Sun 2017-03-05 23:48:01 JST; 2 days ago
Process: 2671 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)
Main PID: 2742 (corosync)
CGroup: /system.slice/corosync.service
mq2742 corosync

Mar 05 23:51:12 pxmx01 corosync[2742]: [TOTEM ] Retransmit List: f2 f3 f4 f...f9
Mar 05 23:51:12 pxmx01 corosync[2742]: [TOTEM ] Retransmit List: f2 f3 f4 f...f9
Mar 05 23:51:12 pxmx01 corosync[2742]: [TOTEM ] Retransmit List: f2 f3 f4 f...f9
Mar 05 23:51:12 pxmx01 corosync[2742]: [TOTEM ] Retransmit List: f2 f3 f4 f...f9
Mar 05 23:51:12 pxmx01 corosync[2742]: [TOTEM ] Retransmit List: f2 f3 f4 f...f9
Mar 05 23:51:12 pxmx01 corosync[2742]: [TOTEM ] Retransmit List: f2 f3
Mar 05 23:53:00 pxmx01 corosync[2742]: [TOTEM ] A new membership (172.16.0.... 3
Mar 05 23:53:00 pxmx01 corosync[2742]: [QUORUM] This node is within the pri...e.
Mar 05 23:53:00 pxmx01 corosync[2742]: [QUORUM] Members[3]: 1 2 3
Mar 05 23:53:00 pxmx01 corosync[2742]: [MAIN ] Completed service synchroni...e.
Hint: Some lines were ellipsized, use -l to show in full.

* pxmx02
root@pxmx02:~# systemctl status pve-cluster corosync
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: active (running) since Sun 2017-03-05 23:48:42 JST; 2 days ago
Process: 2633 ExecStartPost=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 2557 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 2631 (pmxcfs)
CGroup: /system.slice/pve-cluster.service
mq2631 /usr/bin/pmxcfs

Mar 08 02:46:40 pxmx02 pmxcfs[2631]: [dcdb] notice: data verification successful
Mar 08 03:34:05 pxmx02 pmxcfs[2631]: [status] notice: received log
Mar 08 03:34:11 pxmx02 pmxcfs[2631]: [status] notice: received log
Mar 08 03:46:40 pxmx02 pmxcfs[2631]: [dcdb] notice: data verification successful
Mar 08 03:57:05 pxmx02 pmxcfs[2631]: [status] notice: received log
Mar 08 03:57:12 pxmx02 pmxcfs[2631]: [status] notice: received log
Mar 08 04:46:40 pxmx02 pmxcfs[2631]: [dcdb] notice: data verification successful
Mar 08 05:46:40 pxmx02 pmxcfs[2631]: [dcdb] notice: data verification successful
Mar 08 06:46:40 pxmx02 pmxcfs[2631]: [dcdb] notice: data verification successful
Mar 08 07:46:40 pxmx02 pmxcfs[2631]: [dcdb] notice: data verification successful

● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
Active: active (running) since Sun 2017-03-05 23:50:03 JST; 2 days ago
Process: 2646 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)
Main PID: 2721 (corosync)
CGroup: /system.slice/corosync.service
mq2721 corosync

Mar 05 23:50:03 pxmx02 corosync[2721]: [MAIN ] Completed service synchroni...e.
Mar 05 23:50:03 pxmx02 corosync[2721]: [TOTEM ] A new membership (172.16.0.... 1
Mar 05 23:50:03 pxmx02 corosync[2721]: [QUORUM] Members[2]: 1 2
Mar 05 23:50:03 pxmx02 corosync[2721]: [MAIN ] Completed service synchroni...e.
Mar 05 23:50:03 pxmx02 corosync[2646]: Starting Corosync Cluster Engine (co... ]
Mar 05 23:50:03 pxmx02 systemd[1]: Started Corosync Cluster Engine.
Mar 05 23:53:00 pxmx02 corosync[2721]: [TOTEM ] A new membership (172.16.0.... 3
Mar 05 23:53:00 pxmx02 corosync[2721]: [QUORUM] This node is within the pri...e.
Mar 05 23:53:00 pxmx02 corosync[2721]: [QUORUM] Members[3]: 1 2 3
Mar 05 23:53:00 pxmx02 corosync[2721]: [MAIN ] Completed service synchroni...e.
Hint: Some lines were ellipsized, use -l to show in full.

*pxmx03
root@pxmx03:~# systemctl status pve-cluster corosync
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: active (running) since Sun 2017-03-05 23:51:40 JST; 2 days ago
Process: 2744 ExecStartPost=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 2696 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 2742 (pmxcfs)
CGroup: /system.slice/pve-cluster.service
mq2742 /usr/bin/pmxcfs

Mar 08 02:46:40 pxmx03 pmxcfs[2742]: [dcdb] notice: data verification successful
Mar 08 03:34:05 pxmx03 pmxcfs[2742]: [status] notice: received log
Mar 08 03:34:11 pxmx03 pmxcfs[2742]: [status] notice: received log
Mar 08 03:46:40 pxmx03 pmxcfs[2742]: [dcdb] notice: data verification successful
Mar 08 04:46:40 pxmx03 pmxcfs[2742]: [dcdb] notice: data verification successful
Mar 08 05:25:04 pxmx03 pmxcfs[2742]: [status] notice: received log
Mar 08 05:25:10 pxmx03 pmxcfs[2742]: [status] notice: received log
Mar 08 05:46:40 pxmx03 pmxcfs[2742]: [dcdb] notice: data verification successful
Mar 08 06:46:40 pxmx03 pmxcfs[2742]: [dcdb] notice: data verification successful
Mar 08 07:46:40 pxmx03 pmxcfs[2742]: [dcdb] notice: data verification successful

● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
Active: active (running) since Sun 2017-03-05 23:53:01 JST; 2 days ago
Process: 2797 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)
Main PID: 2883 (corosync)
CGroup: /system.slice/corosync.service
mq2883 corosync

Mar 05 23:53:00 pxmx03 corosync[2883]: [TOTEM ] JOIN or LEAVE message was t...n.
Mar 05 23:53:00 pxmx03 corosync[2883]: [TOTEM ] JOIN or LEAVE message was t...n.
Mar 05 23:53:00 pxmx03 corosync[2883]: [QUORUM] Members[1]: 3
Mar 05 23:53:00 pxmx03 corosync[2883]: [MAIN ] Completed service synchroni...e.
Mar 05 23:53:01 pxmx03 corosync[2883]: [TOTEM ] A new membership (172.16.0.... 2
Mar 05 23:53:01 pxmx03 corosync[2883]: [QUORUM] This node is within the pri...e.
Mar 05 23:53:01 pxmx03 corosync[2883]: [QUORUM] Members[3]: 1 2 3
Mar 05 23:53:01 pxmx03 corosync[2883]: [MAIN ] Completed service synchroni...e.
Mar 05 23:53:01 pxmx03 corosync[2797]: Starting Corosync Cluster Engine (co... ]
Mar 05 23:53:01 pxmx03 systemd[1]: Started Corosync Cluster Engine.
Hint: Some lines were ellipsized, use -l to show in full.

 

[dcsapak]

try to start them

systemctl start corosync

or via the gui Node -> System

 

[stats]

I started it by GUI. It is running now. What should I do next?

 

[stats]

I did following command and it recovered now. Is it OK?

pvecm add pxmx01 --force