Syslog errors when creating a cluster. Totem is trying form a cluster. How can i stop

I

ictdude

Active Member
May 18, 2008
88
0
26
Syslog errors when creating a cluster. How can i stop from flooding my log file ?


Mar 1 20:20:57 host153 pmxcfs[1029929]: [status] crit: cpg_send_message failed: 9
Mar 1 20:20:57 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1740
Mar 1 20:20:57 host153 corosync[1001341]: [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Mar 1 20:20:58 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1750
Mar 1 20:20:59 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1760

Must be my firewall. But it seems Totem is trying form a cluster. How can i stop this process ?


Also get error No ticket (401) When start a VM !??

unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1034226' - Permission denied ???

It tells me my node is offline while all systems or running !? I can not create new VM or OpenVZ ?
 
Last edited:
Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

ictdude said:
Must be my firewall. But it seems Totem is trying form a cluster. How can i stop this process ?
Click to expand...

Make sure you do not block corosync related traffic.
 
Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

Temporarily switch off all IPTables just to see if it's indeed firewall related.

Verstuurd vanaf mijn Nexus 5 met Tapatalk
 
Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

masterdam79 said:
Temporarily switch off all IPTables just to see if it's indeed firewall related. Verstuurd vanaf mijn Nexus 5 met Tapatalk
Click to expand...
That i did not figure it out to test that haha .. Goin to test it tonight. But i have also a special network setup. Maybe thats the problem ? Will post it.
 
Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

dietmar said:
Make sure you do not block corosync related traffic.
Click to expand...

What or the ports (Firewall) of corosync ? Can i stop service corosync ? Here my network.
I also use Shorewall version 4.4.11.6 on top of Proxmox works perfect !!!

My hostname of Proxmox show up red. (Offline) Needs to be green. I also cant make a new KVM Openvz any more
Like the system believe my local hostname node is offline !? All this occure while try to make a cluster. Pffff ..
First i could not login any more !? Then i put a backup back: /etc/cluster and /var/lib/pve-cluster
On those files must be a mistake i believe. Need to restore back befor i did try to create the cluster.

When i try to make a backup i get error:



INFO: starting new backup job: vzdump 100 --remove 0 --mode snapshot --compress lzo --storage store_1 --node host153
INFO: Starting Backup of VM 100 (qemu)
INFO: status = stopped
INFO: unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1038526' - Permission denied
ERROR: Backup of VM 100 failed - command 'qm set 100 --lock backup' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors

My Syslog:

Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
Mar 3 11:29:17 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364930
Mar 3 11:29:18 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364940
Mar 3 11:29:19 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364950
Mar 3 11:29:20 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364960
Mar 3 11:29:21 host153 corosync[1001341]: [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Mar 3 11:29:21 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364970
Mar 3 11:29:22 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364980
Mar 3 11:29:23 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364990
Mar 3 11:29:24 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365000
Mar 3 11:29:25 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365010
Mar 3 11:29:26 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365020
Mar 3 11:29:26 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9

Newtork:

auto lo
iface lo inet loopback

# device: eth0
auto eth0
iface eth0 inet static
address 78.xx.xx.153
broadcast 78.46.xx.159
netmask 255.255.255.255
gateway 78.46.xx.129
pointopoint 78.46.xx.129

# Routed network

auto vmbr1
iface vmbr1 inet static
address 78.xx.xx.153
netmask 255.255.255.255
bridge_ports none
bridge_stp off
bridge_fd 0


# Dnat Routed

auto vmbr0
iface vmbr0 inet static
address 10.254.254.254
netmask 255.0.0.0
broadcast 10.255.255.255
bridge_ports none
bridge_stp off
bridge_fd 0
 
Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

Problem have been solved. By settings the system "cluster" in Local mode :-)

So my problem was. System went to read only mode. Then you can not make backups or create KVM or Openvz
any more. All this because something went wrong when creating a cluster.

I think my machine did think it was a slave, So he could not do much like backups or create KVM or Openvz.

Errors like:

Also get error No ticket (401) When start a VM

unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1034226' - Permission denied

You can solve this by settings system in Local mode. :D

Source: http://forum.proxmox.com/threads/9567-help-emergency


First stop it..
# /etc/init.d/pve-cluster stop

then start in Local mode
# /usr/bin/pmxcfs -l

I don't use the cluster so i keep my system running in Local mode. So i have again total control :-) ppffff

I hope this will help out others ... :-)
 
Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

I run a three node cluster without fencing and once in a while I get the same problem with one of the nodes turning red and PVE not accepting my PAM root credentials.

What I do in that case is use the command line to migrate the virtual machines to a correctly working node and completely take the machine out of the cluster, reinstall, join the cluster again and migrate the virtual machines back to the fresh node.

A bit of a hassle but works for me especially because of the live migration feature I have no downtime.

Verstuurd vanaf mijn Nexus 5 met Tapatalk
 
Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

masterdam79 said:
I run a three node cluster without fencing and once in a while I get the same problem with one of the nodes turning red and PVE not accepting my PAM root credentials. What I do in that case is use the command line to migrate the virtual machines to a correctly working node and completely take the machine out of the cluster, reinstall, join the cluster again and migrate the virtual machines back to the fresh node. A bit of a hassle but works for me especially because of the live migration feature I have no downtime. Verstuurd vanaf mijn Nexus 5 met Tapatalk
Click to expand...
Thnx for your info. Haha yeah that's some hassle. But for me this solved my problem. I only needed to move all vm's to a new server. I just copy my backups and restore them. Thats ok for me. Because i use a firewall, can give problems like i had. Maybe off topic. But do you use a firewall ? Some protection ?
 
You must log in or register to reply here.
Top Bottom