Deleted Cluster and now I can't access web UI

Discussion in 'Proxmox VE: Installation and configuration' started by tuckerritti, Jan 11, 2019.

  1. tuckerritti

    tuckerritti New Member

    Joined:
    Aug 17, 2018
    Messages:
    7
    Likes Received:
    0
    Hello,

    I'm kind of a noob in the proxmox shell. I just got my R710 that was going to upgrade to from my 2950 that I have had for a super long time. I installed Proxmox on my new server and created a cluster on my 2950. I was having issues connecting the two so I decided to scrap the idea and thought it would just be best to create VZdump backup files on an external hard drive. I followed this tutorial (1st reply) to delete the cluster on the 2950 (I never connected the R710 to the 2950) but when I restarted the 2950, I could no longer access the web UI. I went to the shell and typed in pvecm status but I get the error:

    Cannot initialize CMAP service

    When I try and start any vm using qm start, I get this error:

    cluster not ready - no quorum?

    When I try pvecm e 1 and then pvecm status again, I get the same cannot initialize CMAP service error.

    What do you think I should do? I don't need to restore the server, I just need all of my VMs so I can put them on the new server.
     
  2. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,161
    Likes Received:
    153
    The next time please follow our official documentation, the mentioned thread is a bit older (possible outdated, e.g. it mentions cman and sysvinit which are not valid in current Proxmox VE 5.X) and sometimes a bit weird things are suggested. https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_separate_node_without_reinstall

    those two command are expected to behave like this if there is no cluster configured, as the cannot connect to the cluster communication stack (corosync).

    First ensure all required services are up and running:
    Code:
    systemctl restart pve-cluster pveproxy pvedaemon
    
    If that does not work, or throws error I need a bit more info from you. E.g., as root run:
    Code:
    systemctl list-units --failed --plain -l
    ls -l /etc/corosync/ /etc/pve/corosync.conf
    systemctl status pveproxy pve-cluster
    
    and post the output here, preferably in [code] output [/code] tags.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. tuckerritti

    tuckerritti New Member

    Joined:
    Aug 17, 2018
    Messages:
    7
    Likes Received:
    0
    When I run systemctl list-units --failed --plain -l, I see that corosync, influxdb, and pvesr have all failed
    Code:
    UNIT             LOAD   ACTIVE SUB    DESCRIPTION                          
    
      corosync.service loaded failed failed Corosync Cluster Engine              
    
      influxdb.service loaded failed failed InfluxDB is an open-source, distributed,
    
      pvesr.service    loaded failed failed Proxmox VE replication runner
    
     
  4. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,161
    Likes Received:
    153
    OK, that should not matter much for your specific case. Corosync has no config anymore thus it failed, but it's only needed in a cluster which you got rid off.
    Did you try the service restarts too and could you please post the rest of the requested outputs.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. tuckerritti

    tuckerritti New Member

    Joined:
    Aug 17, 2018
    Messages:
    7
    Likes Received:
    0
    There is no output from the restart command.
    Code:
    ls -l /etc/corosync/ /etc/pve/corosync.conf
    ls: cannot access '/etc/corosync/': No such file or directory
    
    -r--r----- 1 root www-data 362 Jan  9 16:47 /etc/pve/corosync.conf
    
    Code:
    systemctl status pveproxy pve-cluster
    ● pveproxy.service - PVE API Proxy Server
    
       Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset:
    
       Active: active (running) since Fri 2019-01-11 08:20:34 PST; 47min ago
    
      Process: 23323 ExecStop=/usr/bin/pveproxy stop (code=exited, status=0/SUCCESS)
    
      Process: 23165 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUC
    
      Process: 23417 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCES
    
     Main PID: 23433 (pveproxy)
    
        Tasks: 4 (limit: 4915)
    
       Memory: 113.6M
    
          CPU: 1min 41.919s
    
       CGroup: /system.slice/pveproxy.service
    
              ├─23433 pveproxy
    
              ├─28455 pveproxy worker
    
              ├─28456 pveproxy worker
    
              └─28457 pveproxy worker
    
    
    Jan 11 09:07:57 proxmox pveproxy[23433]: worker 28448 finished
    
    Jan 11 09:07:57 proxmox pveproxy[23433]: starting 1 worker(s)
    
    Jan 11 09:07:57 proxmox pveproxy[23433]: worker 28456 started
    
    Jan 11 09:07:57 proxmox pveproxy[28455]: /etc/pve/local/pve-ssl.key: failed to l
    
    Jan 11 09:07:57 proxmox pveproxy[28456]: /etc/pve/local/pve-ssl.key: failed to l
    
    Jan 11 09:07:57 proxmox pveproxy[28449]: worker exit
    
    Jan 11 09:07:57 proxmox pveproxy[23433]: worker 28449 finished
    
     
  6. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,161
    Likes Received:
    153
    and there's your issue, the proxy cannot load it's ssl key.

    can you try:
    Code:
    pvecm updatecerts
    systemctl restart pveproxy
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. tuckerritti

    tuckerritti New Member

    Joined:
    Aug 17, 2018
    Messages:
    7
    Likes Received:
    0
    Now I get
    Code:
    no quorum - unable to update files
    
     
  8. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,161
    Likes Received:
    153
    Oh well, the pmxcfs is in limbo from the cluster separation try. /etc/pve/corosync.conf is still present thus the configuration file systems thinks it's still clustered.
    To fix that do:
    Code:
    systemctl stop pve-cluster
    # start in local mode
    pmxcfs -l 
    rm /etc/pve/corosync.conf 
    killall pmxcfs
    systemctl stop pve-cluster
    
    then repeat the commands from my previous post.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. tuckerritti

    tuckerritti New Member

    Joined:
    Aug 17, 2018
    Messages:
    7
    Likes Received:
    0
    Now only corosync and influxdb are failing
    Code:
    UNIT             LOAD   ACTIVE SUB    DESCRIPTION                                                  
    
      corosync.service loaded failed failed Corosync Cluster Engine                                      
    
      influxdb.service loaded failed failed InfluxDB is an open-source, distributed, time series database
    
     
  10. tuckerritti

    tuckerritti New Member

    Joined:
    Aug 17, 2018
    Messages:
    7
    Likes Received:
    0
    Corosync fails because /etc/corosync/corosync.conf does not exist
    Code:
    ● corosync.service - Corosync Cluster Engine
    
       Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset:
    
       Active: failed (Result: exit-code) since Wed 2019-01-09 16:51:35 PST; 4 days
    
    Condition: start condition failed at Fri 2019-01-11 12:13:34 PST; 2 days ago
    
              └─ ConditionPathExists=/etc/corosync/corosync.conf was not met
    
    Influxdb gives no information
    Code:
    ● influxdb.service - InfluxDB is an open-source, distributed, time series database
    
       Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
    
       Active: failed (Result: exit-code) since Wed 2019-01-09 16:51:33 PST; 4 days ago
    
        Docs: man:influxd(1)
    
      Process: 1866 ExecStart=/usr/bin/influxd -config /etc/influxdb/influxdb.conf $INFLUXD_OPTS (code=exited, status=1/FAILURE)
    
     Main PID: 1866 (code=exited, status=1/FAILURE)
    
          CPU: 14ms
    
    Also, I don't need to restore this server. I am just wondering if there is any way to get the VMs off of it and on to the new server.
     
  11. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,161
    Likes Received:
    153
    As I already said above, corosync is expected to fail, influxdb isn't important for us right now (not direct part of PVE)...
    As you do not say what exactly you did and do not post all requested output this is a bit hard to address, I'm afraid...

    I assume you fixed pve-cluster as I said above and it is writeable again, so did you rerun the updatecerts and pveproxy commads:

    ?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. tuckerritti

    tuckerritti New Member

    Joined:
    Aug 17, 2018
    Messages:
    7
    Likes Received:
    0
    Code:
    root@proxmox:~# pvecm updatecerts
    
    (re)generate node files
    
    merge authorized SSH keys and known hosts
    
    and systemctl does not give any output

    I also noticed that local-lvm does not exist. When I try to start a VM now, I get:
    Code:
    root@proxmox:~# qm start 100
    
    storage 'local-lvm' does not exists
    
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice