Syslog errors when creating a cluster. Totem is trying form a cluster. How can i stop

[ictdude]

Syslog errors when creating a cluster. How can i stop from flooding my log file ?


Mar 1 20:20:57 host153 pmxcfs[1029929]: [status] crit: cpg_send_message failed: 9
Mar 1 20:20:57 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1740
Mar 1 20:20:57 host153 corosync[1001341]: [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Mar 1 20:20:58 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1750
Mar 1 20:20:59 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1760

Must be my firewall. But it seems Totem is trying form a cluster. How can i stop this process ?


Also get error No ticket (401) When start a VM !??

unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1034226' - Permission denied ???

It tells me my node is offline while all systems or running !? I can not create new VM or OpenVZ ?

 

[dietmar]

Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

Must be my firewall. But it seems Totem is trying form a cluster. How can i stop this process ?

Make sure you do not block corosync related traffic.

 

[masterdam79]

Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

Temporarily switch off all IPTables just to see if it's indeed firewall related.

Verstuurd vanaf mijn Nexus 5 met Tapatalk

 

[ictdude]

Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

Temporarily switch off all IPTables just to see if it's indeed firewall related. Verstuurd vanaf mijn Nexus 5 met Tapatalk

That i did not figure it out to test that haha .. Goin to test it tonight. But i have also a special network setup. Maybe thats the problem ? Will post it.

 

[ictdude]

Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

Make sure you do not block corosync related traffic.

What or the ports (Firewall) of corosync ? Can i stop service corosync ? Here my network.
I also use Shorewall version 4.4.11.6 on top of Proxmox works perfect !!!

My hostname of Proxmox show up red. (Offline) Needs to be green. I also cant make a new KVM Openvz any more
Like the system believe my local hostname node is offline !? All this occure while try to make a cluster. Pffff ..
First i could not login any more !? Then i put a backup back: /etc/cluster and /var/lib/pve-cluster
On those files must be a mistake i believe. Need to restore back befor i did try to create the cluster.

When i try to make a backup i get error:



INFO: starting new backup job: vzdump 100 --remove 0 --mode snapshot --compress lzo --storage store_1 --node host153
INFO: Starting Backup of VM 100 (qemu)
INFO: status = stopped
INFO: unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1038526' - Permission denied
ERROR: Backup of VM 100 failed - command 'qm set 100 --lock backup' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors

My Syslog:

Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
Mar 3 11:29:17 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364930
Mar 3 11:29:18 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364940
Mar 3 11:29:19 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364950
Mar 3 11:29:20 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364960
Mar 3 11:29:21 host153 corosync[1001341]: [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Mar 3 11:29:21 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364970
Mar 3 11:29:22 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364980
Mar 3 11:29:23 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364990
Mar 3 11:29:24 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365000
Mar 3 11:29:25 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365010
Mar 3 11:29:26 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365020
Mar 3 11:29:26 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9

Newtork:

auto lo
iface lo inet loopback

# device: eth0
auto eth0
iface eth0 inet static
address 78.xx.xx.153
broadcast 78.46.xx.159
netmask 255.255.255.255
gateway 78.46.xx.129
pointopoint 78.46.xx.129

# Routed network

auto vmbr1
iface vmbr1 inet static
address 78.xx.xx.153
netmask 255.255.255.255
bridge_ports none
bridge_stp off
bridge_fd 0


# Dnat Routed

auto vmbr0
iface vmbr0 inet static
address 10.254.254.254
netmask 255.0.0.0
broadcast 10.255.255.255
bridge_ports none
bridge_stp off
bridge_fd 0

 

[ictdude]

Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

Problem have been solved. By settings the system "cluster" in Local mode

So my problem was. System went to read only mode. Then you can not make backups or create KVM or Openvz
any more. All this because something went wrong when creating a cluster.

I think my machine did think it was a slave, So he could not do much like backups or create KVM or Openvz.

Errors like:

Also get error No ticket (401) When start a VM

unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1034226' - Permission denied

You can solve this by settings system in Local mode.

Source: http://forum.proxmox.com/threads/9567-help-emergency


First stop it..
# /etc/init.d/pve-cluster stop

then start in Local mode
# /usr/bin/pmxcfs -l

I don't use the cluster so i keep my system running in Local mode. So i have again total control ppffff

I hope this will help out others ...

 

[masterdam79]

Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

I run a three node cluster without fencing and once in a while I get the same problem with one of the nodes turning red and PVE not accepting my PAM root credentials.

What I do in that case is use the command line to migrate the virtual machines to a correctly working node and completely take the machine out of the cluster, reinstall, join the cluster again and migrate the virtual machines back to the fresh node.

A bit of a hassle but works for me especially because of the live migration feature I have no downtime.

Verstuurd vanaf mijn Nexus 5 met Tapatalk

 

[ictdude]

Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

I run a three node cluster without fencing and once in a while I get the same problem with one of the nodes turning red and PVE not accepting my PAM root credentials. What I do in that case is use the command line to migrate the virtual machines to a correctly working node and completely take the machine out of the cluster, reinstall, join the cluster again and migrate the virtual machines back to the fresh node. A bit of a hassle but works for me especially because of the live migration feature I have no downtime. Verstuurd vanaf mijn Nexus 5 met Tapatalk

Thnx for your info. Haha yeah that's some hassle. But for me this solved my problem. I only needed to move all vm's to a new server. I just copy my backups and restore them. Thats ok for me. Because i use a firewall, can give problems like i had. Maybe off topic. But do you use a firewall ? Some protection ?