ipcc_send_rec failed: Connection refused

[wahmed]

This morning one of the node running all VMs was kicked out of the cluster. A reboot seems to be fixed whatever the issue was. Going back in the log there are many dozen lines with this error msg:

pveproxy[299993]: WARNING: ipcc_send_rec failed: Connection refused

pvestatd[3148]: WARNING: ipcc_send_rec failed: Connection refused

Several lines of this msg:

/usr/sbin/cron[2764]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)

While trying to SSH before the reboot, i could access the server and goto /etc. But if i tried to goto /etc/pve it gave me Connection time out error. Any idea what might have caused this or how can i trace the root cause of the issue?

No backup was scheduled when the problem happened.

 

[dietmar]

Seems cman was not running? Any cman related errors in /var/log/syslog?

 

[wahmed]

Seems cman was not running? Any cman related errors in /var/log/syslog?

The host node stopped working at 7:09am. The furthest time stamp for the log is 6:28am. I checked the entire log, the only cman related info is on 13:10pm saying cman started successfully, which is when i rebooted the server.

 

[linuxuser]

I am getting this same issue. I ran cman and get the below
root@nice:~# /etc/init.d/cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Tuning DLM kernel config... [ OK ]
Unfencing self... [ OK ]

Not sure what else to check?