quorum_initialize failed

J

Jiang.Lee

New Member
Nov 25, 2021
2
0
1
25
note : i am noob of proxmox. we didn;t have any maintenaner in here. something
last night , we found some server cannot connect VM ( HotKey valid .....). so , we decide reinstall the server. but we forgot to removed VMs first and removed server from cluster.
when we success startup new promox server , start to join cluster. find these issue
  1. the vm cannot removed
  2. the web page was slow, only trun off the new promox server, the page loading back to normal.
so, we find some way to remove that VMs ( delete `rm -f /etc/pve/qemu-server/<vm id>.conf` ). and it works, but web page loading is very slow. we thing something is doing. we wait for few hours, now the entire cluster down (from dashboard). but login other cluster node , the node shows normal online.

here is some info
  • master
Bash: 
root@pveg6:~# systemctl status pve-cluster.service corosync.service
● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-11-25 12:35:36 CST; 32min ago
  Process: 942 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
 Main PID: 958 (pmxcfs)
    Tasks: 6 (limit: 4915)
   Memory: 57.8M
   CGroup: /system.slice/pve-cluster.service
           └─958 /usr/bin/pmxcfs

Nov 25 12:35:35 pveg6 pmxcfs[958]: [quorum] crit: quorum_initialize failed: 2
Nov 25 12:35:35 pveg6 pmxcfs[958]: [quorum] crit: can't initialize service
Nov 25 12:35:35 pveg6 pmxcfs[958]: [confdb] crit: cmap_initialize failed: 2
Nov 25 12:35:35 pveg6 pmxcfs[958]: [confdb] crit: can't initialize service
Nov 25 12:35:35 pveg6 pmxcfs[958]: [dcdb] crit: cpg_initialize failed: 2
Nov 25 12:35:35 pveg6 pmxcfs[958]: [dcdb] crit: can't initialize service
Nov 25 12:35:35 pveg6 pmxcfs[958]: [status] crit: cpg_initialize failed: 2
Nov 25 12:35:35 pveg6 pmxcfs[958]: [status] crit: can't initialize service
Nov 25 12:35:36 pveg6 systemd[1]: Started The Proxmox VE cluster filesystem.
Nov 25 12:35:41 pveg6 pmxcfs[958]: [status] notice: update cluster info (cluster name  LAN-GROUP, version = 36)

● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-11-25 12:35:36 CST; 32min ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
 Main PID: 1071 (corosync)
    Tasks: 9 (limit: 4915)
   Memory: 161.2M
   CGroup: /system.slice/corosync.service
           └─1071 /usr/sbin/corosync -f

Nov 25 13:07:19 pveg6 corosync[1071]:   [KNET  ] rx: host: 3 link: 0 is up
Nov 25 13:07:26 pveg6 corosync[1071]:   [KNET  ] link: host: 10 link: 0 is down
Nov 25 13:07:37 pveg6 corosync[1071]:   [KNET  ] rx: host: 10 link: 0 is up
Nov 25 13:07:38 pveg6 corosync[1071]:   [KNET  ] link: host: 3 link: 0 is down
Nov 25 13:07:49 pveg6 corosync[1071]:   [KNET  ] link: host: 10 link: 0 is down
Nov 25 13:07:58 pveg6 corosync[1071]:   [KNET  ] rx: host: 10 link: 0 is up
Nov 25 13:08:03 pveg6 corosync[1071]:   [KNET  ] link: host: 5 link: 0 is down
Nov 25 13:08:07 pveg6 corosync[1071]:   [KNET  ] link: host: 10 link: 0 is down
Nov 25 13:08:08 pveg6 corosync[1071]:   [KNET  ] rx: host: 5 link: 0 is up
Nov 25 13:08:08 pveg6 corosync[1071]:   [KNET  ] rx: host: 10 link: 0 is up

Bash: 
Nov 25 13:32:09 pveg6 systemd[1]: pvesr.service: Main process exited, code=exited, status=13/n/a
Nov 25 13:32:09 pveg6 systemd[1]: pvesr.service: Failed with result 'exit-code'.
Nov 25 13:32:09 pveg6 systemd[1]: Failed to start Proxmox VE replication runner.
Nov 25 13:32:09 pveg6 pveproxy[8776]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:09 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:09 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:10 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:10 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:10 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:11 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:11 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:11 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:12 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:12 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:13 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:13 pveg6 pveproxy[8776]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:13 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:13 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:14 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:14 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:14 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:14 pveg6 corosync[1071]:   [KNET  ] rx: host: 7 link: 0 is up
Nov 25 13:32:14 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:15 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:15 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:15 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:16 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:16 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:17 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:17 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:17 pveg6 pveproxy[8776]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:17 pveg6 pvedaemon[1117]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:18 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:18 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:18 pveg6 corosync[1071]:   [KNET  ] link: host: 7 link: 0 is down
Nov 25 13:32:18 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:19 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:19 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:19 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:20 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:20 pveg6 pveproxy[5545]: Cluster not quorate - extending auth key lifetime!
Nov 25 13:32:20 pveg6 pvedaemon[1115]: Cluster not quorate - extending auth key lifetime!

Bash: 
 pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-1
pve-kernel-helper: 6.2-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.22-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-2~bpo10+1
libproxmox-acme-perl: 1.0.3
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-5
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.2-1~bpo10+1
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1


  • pv1g6 ( reinstall one )
Bash: 
root@pv1g6:~# systemctl status pve-cluster.service corosync.service
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-11-25 09:22:37 CST; 4h 12min ago
    Process: 871 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
   Main PID: 872 (pmxcfs)
      Tasks: 10 (limit: 9257)
     Memory: 57.1M
        CPU: 8.502s
     CGroup: /system.slice/pve-cluster.service
             └─872 /usr/bin/pmxcfs

Nov 25 13:34:40 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 70
Nov 25 13:34:41 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 80
Nov 25 13:34:42 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 90
Nov 25 13:34:43 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 100
Nov 25 13:34:43 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retried 100 times
Nov 25 13:34:43 pv1g6 pmxcfs[872]: [status] crit: cpg_send_message failed: 6
Nov 25 13:34:44 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 10
Nov 25 13:34:45 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 20
Nov 25 13:34:46 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 30
Nov 25 13:34:47 pv1g6 pmxcfs[872]: [status] notice: cpg_send_message retry 40

● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-11-25 10:43:55 CST; 2h 50min ago
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview
   Main PID: 11203 (corosync)
      Tasks: 9 (limit: 9257)
     Memory: 166.2M
        CPU: 7h 25min 4.694s
     CGroup: /system.slice/corosync.service
             └─11203 /usr/sbin/corosync -f

Nov 25 13:33:33 pv1g6 corosync[11203]:   [KNET  ] rx: host: 3 link: 0 is up
Nov 25 13:33:45 pv1g6 corosync[11203]:   [KNET  ] link: host: 3 link: 0 is down
Nov 25 13:33:48 pv1g6 corosync[11203]:   [KNET  ] rx: host: 3 link: 0 is up
Nov 25 13:34:00 pv1g6 corosync[11203]:   [KNET  ] link: host: 3 link: 0 is down
Nov 25 13:34:01 pv1g6 corosync[11203]:   [KNET  ] link: host: 1 link: 0 is down
Nov 25 13:34:01 pv1g6 corosync[11203]:   [KNET  ] rx: host: 3 link: 0 is up
Nov 25 13:34:17 pv1g6 corosync[11203]:   [KNET  ] rx: host: 1 link: 0 is up
Nov 25 13:34:29 pv1g6 corosync[11203]:   [KNET  ] link: host: 3 link: 0 is down
Nov 25 13:34:32 pv1g6 corosync[11203]:   [KNET  ] rx: host: 3 link: 0 is up
Nov 25 13:34:47 pv1g6 corosync[11203]:   [KNET  ] link: host: 7 link: 0 is down

Bash: 
pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.11.22-4-pve: 5.11.22-8
ceph-fuse: 15.2.14-pve1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
 
hi,brother,have you solved the problem yet？

If I can get it out of the way, please help me, thanks.
 
You must log in or register to reply here.
Top Bottom