2 nodes showing red after upgrade to 2.3

[mess386]

I upgraded 3 nodes yesterday to 2.3 and restsrted all three nodes.
All was fine. When I returned to work this morning 2 nodes are showing red.

I restated cman and did not help.
I also restarted one node still not working.

can someone help me with is.

 

[mess386]

/etc/init.d/cman restart
Stopping cluster:
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown:[ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Tuning DLM kernel config... [ OK ]
Unfencing self... [ OK ]

 

[mess386]

Ok I have both nodes working again.
I dont know if this was right but I restarted /etc/init.d/pve-cluster stop
than /etc/init.d/pve-cluster start
and both nodes show online now.