Syslog errors when creating a cluster. Totem is trying form a cluster. How can i stop

Discussion in 'Proxmox VE: Installation and configuration' started by ictdude, Mar 1, 2014.

  1. ictdude

    ictdude Member

    Joined:
    May 18, 2008
    Messages:
    88
    Likes Received:
    0
    Syslog errors when creating a cluster. How can i stop from flooding my log file ?


    Mar 1 20:20:57 host153 pmxcfs[1029929]: [status] crit: cpg_send_message failed: 9
    Mar 1 20:20:57 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1740
    Mar 1 20:20:57 host153 corosync[1001341]: [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
    Mar 1 20:20:58 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1750
    Mar 1 20:20:59 host153 pmxcfs[1029929]: [dcdb] notice: cpg_join retry 1760

    Must be my firewall. But it seems Totem is trying form a cluster. How can i stop this process ?


    Also get error No ticket (401) When start a VM !??

    unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1034226' - Permission denied ???

    It tells me my node is offline while all systems or running !? I can not create new VM or OpenVZ ?
     
    #1 ictdude, Mar 1, 2014
    Last edited: Mar 1, 2014
  2. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,055
    Likes Received:
    257
    Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

    Make sure you do not block corosync related traffic.
     
  3. masterdam79

    masterdam79 New Member

    Joined:
    Feb 15, 2014
    Messages:
    18
    Likes Received:
    0
    Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

    Temporarily switch off all IPTables just to see if it's indeed firewall related.

    Verstuurd vanaf mijn Nexus 5 met Tapatalk
     
  4. ictdude

    ictdude Member

    Joined:
    May 18, 2008
    Messages:
    88
    Likes Received:
    0
    Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

    That i did not figure it out to test that haha .. Goin to test it tonight. But i have also a special network setup. Maybe thats the problem ? Will post it.
     
  5. ictdude

    ictdude Member

    Joined:
    May 18, 2008
    Messages:
    88
    Likes Received:
    0
    Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

    What or the ports (Firewall) of corosync ? Can i stop service corosync ? Here my network.
    I also use Shorewall version 4.4.11.6 on top of Proxmox works perfect !!!

    My hostname of Proxmox show up red. (Offline) Needs to be green. I also cant make a new KVM Openvz any more
    Like the system believe my local hostname node is offline !? All this occure while try to make a cluster. Pffff ..
    First i could not login any more !? Then i put a backup back: /etc/cluster and /var/lib/pve-cluster
    On those files must be a mistake i believe. Need to restore back befor i did try to create the cluster.

    When i try to make a backup i get error:



    INFO: starting new backup job: vzdump 100 --remove 0 --mode snapshot --compress lzo --storage store_1 --node host153
    INFO: Starting Backup of VM 100 (qemu)
    INFO: status = stopped
    INFO: unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1038526' - Permission denied
    ERROR: Backup of VM 100 failed - command 'qm set 100 --lock backup' failed: exit code 2
    INFO: Backup job finished with errors
    TASK ERROR: job errors

    My Syslog:

    Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
    Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
    Mar 3 11:29:16 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9
    Mar 3 11:29:17 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364930
    Mar 3 11:29:18 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364940
    Mar 3 11:29:19 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364950
    Mar 3 11:29:20 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364960
    Mar 3 11:29:21 host153 corosync[1001341]: [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
    Mar 3 11:29:21 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364970
    Mar 3 11:29:22 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364980
    Mar 3 11:29:23 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1364990
    Mar 3 11:29:24 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365000
    Mar 3 11:29:25 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365010
    Mar 3 11:29:26 host153 pmxcfs[9348]: [dcdb] notice: cpg_join retry 1365020
    Mar 3 11:29:26 host153 pmxcfs[9348]: [status] crit: cpg_send_message failed: 9

    Newtork:

    auto lo
    iface lo inet loopback

    # device: eth0
    auto eth0
    iface eth0 inet static
    address 78.xx.xx.153
    broadcast 78.46.xx.159
    netmask 255.255.255.255
    gateway 78.46.xx.129
    pointopoint 78.46.xx.129

    # Routed network

    auto vmbr1
    iface vmbr1 inet static
    address 78.xx.xx.153
    netmask 255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0


    # Dnat Routed

    auto vmbr0
    iface vmbr0 inet static
    address 10.254.254.254
    netmask 255.0.0.0
    broadcast 10.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0
     
  6. ictdude

    ictdude Member

    Joined:
    May 18, 2008
    Messages:
    88
    Likes Received:
    0
    Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

    Problem have been solved. By settings the system "cluster" in Local mode :)

    So my problem was. System went to read only mode. Then you can not make backups or create KVM or Openvz
    any more. All this because something went wrong when creating a cluster.

    I think my machine did think it was a slave, So he could not do much like backups or create KVM or Openvz.

    Errors like:

    Also get error No ticket (401) When start a VM

    unable to open file '/etc/pve/nodes/host153/qemu-server/100.conf.tmp.1034226' - Permission denied

    You can solve this by settings system in Local mode. :D

    Source: http://forum.proxmox.com/threads/9567-help-emergency


    First stop it..
    # /etc/init.d/pve-cluster stop

    then start in Local mode
    # /usr/bin/pmxcfs -l

    I don't use the cluster so i keep my system running in Local mode. So i have again total control :) ppffff

    I hope this will help out others ... :)
     
  7. masterdam79

    masterdam79 New Member

    Joined:
    Feb 15, 2014
    Messages:
    18
    Likes Received:
    0
    Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

    I run a three node cluster without fencing and once in a while I get the same problem with one of the nodes turning red and PVE not accepting my PAM root credentials.

    What I do in that case is use the command line to migrate the virtual machines to a correctly working node and completely take the machine out of the cluster, reinstall, join the cluster again and migrate the virtual machines back to the fresh node.

    A bit of a hassle but works for me especially because of the live migration feature I have no downtime.

    Verstuurd vanaf mijn Nexus 5 met Tapatalk
     
  8. ictdude

    ictdude Member

    Joined:
    May 18, 2008
    Messages:
    88
    Likes Received:
    0
    Re: Syslog errors when creating a cluster. Totem is trying form a cluster. How can i

    Thnx for your info. Haha yeah that's some hassle. But for me this solved my problem. I only needed to move all vm's to a new server. I just copy my backups and restore them. Thats ok for me. Because i use a firewall, can give problems like i had. Maybe off topic. But do you use a firewall ? Some protection ?
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice