Cluster information greyed out

H

Harish Kamath

New Member
Sep 20, 2022
12
0
1
Cluster-Information-Greyed-Out.jpgHi,

I have a Proxmox cluster and find cluster information being greyed out. The current configuration includes two nodes with status as available. I attempted to add third node but failed.

I have tried rebooting nodes - pve and pve-2.
I recreated pve-70 (now has a new IP address). The failed join action of pve-70 is with old IP address.

I am unable to fetch "Join Information".
I am unable to remove pve-70 because it is not reachable

Output of version details of the nodes are provided below.

Request your inputs.

Thanks,

root@pve:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
root@pve:~#

root@pve-2:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
root@pve-2:~#
 
Hi,

I get "cannot initialize CMAP service"

I suspect the issue is observed only on adding this node. Here are the steps that I followed

  1. Removed the failing node (pve-70) from /etc/pve/corosync.conf
  2. Cluster info section activated
  3. Added a new node using Join Cluster GUI approach
  4. Cluster info section is active
  5. Updated /etc/pve/corosync.conf with failing node (pve-70)
  6. Cluster info section is deactivated again
Screenshot 2022-09-24 at 12.57.58 PM.pngScreenshot 2022-09-24 at 12.59.45 PM.pngScreenshot 2022-09-24 at 1.00.18 PM.pngScreenshot 2022-09-24 at 1.01.37 PM.png
 

Attachments

  • Screenshot 2022-09-24 at 1.01.00 PM.png
    Screenshot 2022-09-24 at 1.01.00 PM.png
    190.9 KB · Views: 1
What does your journalctl -b 0 -u corosync say? Can you call that on the new node and one of the old ones?
 
Hi,

I have attached the screen shots of failing node followed by one of the cluster member.

I now observe that the web connection to the faulty node is also failing with site not reachable. However, the ping succeeds from the cluster member succeeds.


1664194813037.pngScreenshot 2022-09-26 at 5.54.58 PM.pngScreenshot 2022-09-26 at 5.54.33 PM.png
 
Try to copy /etc/corosync/authkey from one of the old nodes onto the new node. Then restart the node, an please post the output of the corosync log again from that node.
 
Hi,

As suggested, I copied the authkey file and observe green icon with the node. However, I still find two issues in my Datacenter cluster environment -

  1. Cluster GUI portion is greyed out
  2. The node site is still not accessible on the Google Chrome Browser
  3. I don't understand why one of the node (pve-2) is not indicated as enabled but can access its shell
The output of the log is provided below -

journalctl -b 0 -u corosync
-- Journal begins at Tue 2022-09-20 18:40:58 IST, ends at Mon 2022-09-26 22:25:28 IST. --
Sep 26 22:18:22 pve-70 systemd[1]: Starting Corosync Cluster Engine...
Sep 26 22:18:22 pve-70 corosync[858]: [MAIN ] Corosync Cluster Engine 3.1.5 starting up
Sep 26 22:18:22 pve-70 corosync[858]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] Initializing transport (Kronosnet).
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] totemknet initialized
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync configuration map access [0]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: cmap
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync configuration service [1]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: cfg
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Sep 26 22:18:22 pve-70 systemd[1]: Started Corosync Cluster Engine.
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: cpg
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync profile loading service [4]
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] Watchdog not enabled by configuration
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] resource load_15min missing a recovery key.
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] resource memory_used missing a recovery key.
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] no resources configured.
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync watchdog service [7]
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Using quorum provider corosync_votequorum
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: votequorum
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: quorum
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] Configuring link 0
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] Configured link number 0: local addr: 192.168.122.237, port=5405
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 0)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Sync members[1]: 4
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Sync joined[1]: 4
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] A new membership (4.eca) was formed. Members joined: 4
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Members[1]: 4
Sep 26 22:18:22 pve-70 corosync[858]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] rx: host: 3 link: 0 is up
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] rx: host: 2 link: 0 is up
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] rx: host: 1 link: 0 is up
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] pmtud: Global data MTU changed to: 469
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] Sync joined[3]: 1 2 3
Sep 26 22:18:26 pve-70 corosync[858]: [TOTEM ] A new membership (1.f28) was formed. Members joined: 1 2 3
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] This node is within the primary component and will provide service.
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] Members[4]: 1 2 3 4
Sep 26 22:18:26 pve-70 corosync[858]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: PMTUD link change for host: 1 link: 0 from 469 to 1397
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: Global data MTU changed to: 1397
root@pve-70:~#
Screenshot 2022-09-26 at 10.36.15 PM.png
 
Hi
now there are two new nodes ( pve-70 and proxmox-v71-4 ). What does the log of pve-2 say journalctl -b 0 -u corosync? Can you post your /etc/pve/corosync.conf?
 
The output of "journalctl -b 0 -u corosync" on pve-2 is provided below -


Last login: Tue Sep 27 13:03:07 IST 2022 from 192.168.122.29 on pts/0
root@pve-2:~# journalctl -b 0 -u corosync
Journal file /var/log/journal/a52871dde0ce40bb809e41092678d0c5/system.journal is truncated, ignoring file.
-- Journal begins at Tue 2022-09-20 12:40:53 IST, ends at Tue 2022-09-27 13:03:07 IST. --
Sep 27 12:05:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:07 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:14 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:19 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:25 pve-2 corosync[738]: [TOTEM ] Retransmit List: 1f19
Sep 27 12:10:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:31 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:31 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:42 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:43 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:43 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:46 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:10:50 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:10:50 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:10:52 pve-2 corosync[738]: [TOTEM ] Token has not been received in 9078 ms
Sep 27 12:10:52 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:10:52 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:10:56 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:10:56 pve-2 corosync[738]: [TOTEM ] A new membership (1.f39) was formed. Members
Sep 27 12:10:56 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:10:56 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:04 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:07 pve-2 corosync[738]: [TOTEM ] Retransmit List: 2a 2b
Sep 27 12:11:10 pve-2 corosync[738]: [TOTEM ] Retransmit List: 31
Sep 27 12:11:12 pve-2 corosync[738]: [TOTEM ] Retransmit List: 31
Sep 27 12:11:12 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:11:12 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:12 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:11:14 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:11:14 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:24 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:25 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:26 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:26 pve-2 corosync[738]: [TOTEM ] A new membership (1.f3d) was formed. Members
Sep 27 12:11:26 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:26 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:31 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:11:31 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:31 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:11:31 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:32 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:38 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:11:38 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] Sync members[2]: 2 3
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] Sync left[2]: 1 4
Sep 27 12:11:38 pve-2 corosync[738]: [TOTEM ] A new membership (2.f45) was formed. Members left: 1 4
Sep 27 12:11:38 pve-2 corosync[738]: [TOTEM ] Failed to receive the leave message. failed: 1 4
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] Members[2]: 2 3
Sep 27 12:11:38 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] Sync joined[2]: 1 4
Sep 27 12:11:39 pve-2 corosync[738]: [TOTEM ] A new membership (1.f49) was formed. Members joined: 1 4
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] This node is within the primary component and will provide service.
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:39 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:39 pve-2 corosync[738]: [TOTEM ] Retransmit List: 1b
Sep 27 12:11:39 pve-2 corosync[738]: [TOTEM ] Retransmit List: 2a
Sep 27 12:11:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:12:06 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:12:06 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:12:06 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:12:09 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3226 ms
Sep 27 12:12:10 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:12:10 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:12:10 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:12:14 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:12:14 pve-2 corosync[738]: [TOTEM ] A new membership (1.f4d) was formed. Members
Sep 27 12:12:14 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:12:14 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:12:16 pve-2 corosync[738]: [TOTEM ] Retransmit List: 1e
Sep 27 12:13:54 pve-2 corosync[738]: [TOTEM ] Retransmit List: 131
Sep 27 12:13:59 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:14:11 pve-2 corosync[738]: [TOTEM ] Retransmit List: 15d
Sep 27 12:14:15 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:14:15 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:14:15 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:14:18 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:14:18 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:14:18 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:14:19 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:14:25 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:14:25 pve-2 corosync[738]: [TOTEM ] A new membership (1.f51) was formed. Members
Sep 27 12:14:29 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:14:29 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:14:39 pve-2 corosync[738]: [TOTEM ] Retransmit List: 38
Sep 27 12:15:25 pve-2 corosync[738]: [TOTEM ] Retransmit List: cb
Sep 27 12:22:49 pve-2 corosync[738]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:22:49 pve-2 corosync[738]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:22:49 pve-2 corosync[738]: [KNET ] host: host: 3 has no active links
Sep 27 12:22:52 pve-2 corosync[738]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:22:52 pve-2 corosync[738]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:23:04 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:23:05 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:23:05 pve-2 corosync[738]: [TOTEM ] A new membership (1.f55) was formed. Members
Sep 27 12:23:05 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:23:05 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:24:58 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
root@pve-2:~#

root@pve-2:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: proxmox-v71-4
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.122.85
}
node {
name: pve
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.122.29
}
node {
name: pve-2
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.122.99
}
node {
name: pve-70
nodeid: 4
quorum_votes: 1
ring0_addr: 192.168.122.237
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: demo-cluster-1
config_version: 7
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

root@pve-2:~#
 
Last edited:
There is some issue with the corosync network. It can't reach host 1 (it should be the "pve" machine) [1]. Can you check the logs there as well?


[1]
Code: 
Sep 27 12:11:31 pve-2 corosync[738]:   [KNET  ] link: host: 1 link: 0 is down
Sep 27 12:11:31 pve-2 corosync[738]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:31 pve-2 corosync[738]:   [KNET  ] host: host: 1 has no active links
Sep 27 12:11:31 pve-2 corosync[738]:   [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:32 pve-2 corosync[738]:   [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:38 pve-2 corosync[738]:   [KNET  ] rx: host: 1 link: 0 is up
[code]
 
I have pasted the log output of the pve node.

root@pve:~# journalctl -b 0 -u corosync
-- Journal begins at Sat 2022-09-17 20:03:55 IST, ends at Sat 2022-10-01 11:04:24 IST. --
Sep 26 17:44:27 pve systemd[1]: Starting Corosync Cluster Engine...
Sep 26 17:44:27 pve corosync[879]: [MAIN ] Corosync Cluster Engine 3.1.5 starting up
Sep 26 17:44:27 pve corosync[879]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Sep 26 17:44:27 pve corosync[879]: [TOTEM ] Initializing transport (Kronosnet).
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] totemknet initialized
Sep 26 17:44:28 pve corosync[879]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync configuration map access [0]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: cmap
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync configuration service [1]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: cfg
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: cpg
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync profile loading service [4]
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Sep 26 17:44:28 pve corosync[879]: [WD ] Watchdog not enabled by configuration
Sep 26 17:44:28 pve corosync[879]: [WD ] resource load_15min missing a recovery key.
Sep 26 17:44:28 pve corosync[879]: [WD ] resource memory_used missing a recovery key.
Sep 26 17:44:28 pve corosync[879]: [WD ] no resources configured.
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync watchdog service [7]
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Using quorum provider corosync_votequorum
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: votequorum
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: quorum
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] Configuring link 0
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] Configured link number 0: local addr: 192.168.122.29, port=5405
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 0)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Sync members[1]: 1
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Sync joined[1]: 1
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] A new membership (1.f1c) was formed. Members joined: 1
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Members[1]: 1
Sep 26 17:44:28 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 17:44:28 pve systemd[1]: Started Corosync Cluster Engine.
Sep 26 17:44:30 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 26 17:44:30 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:30 pve corosync[879]: [KNET ] pmtud: Global data MTU changed to: 469
Sep 26 17:44:31 pve corosync[879]: [QUORUM] Sync members[2]: 1 3
Sep 26 17:44:31 pve corosync[879]: [QUORUM] Sync joined[1]: 3
Sep 26 17:44:31 pve corosync[879]: [TOTEM ] A new membership (1.f20) was formed. Members joined: 3
Sep 26 17:44:31 pve corosync[879]: [QUORUM] Members[2]: 1 3
Sep 26 17:44:31 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 17:44:34 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 26 17:44:34 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 17:44:34 pve corosync[879]: [QUORUM] Sync members[3]: 1 2 3
Sep 26 17:44:34 pve corosync[879]: [QUORUM] Sync joined[1]: 2
Sep 26 17:44:34 pve corosync[879]: [TOTEM ] A new membership (1.f24) was formed. Members joined: 2
Sep 26 17:44:34 pve corosync[879]: [QUORUM] This node is within the primary component and will provide service.
Sep 26 17:44:34 pve corosync[879]: [QUORUM] Members[3]: 1 2 3
Sep 26 17:44:34 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 17:44:52 pve corosync[879]: [KNET ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397
Sep 26 17:44:52 pve corosync[879]: [KNET ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397
Sep 26 17:44:52 pve corosync[879]: [KNET ] pmtud: Global data MTU changed to: 1397
Sep 26 22:18:26 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 26 22:18:26 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 22:18:26 pve corosync[879]: [KNET ] pmtud: PMTUD link change for host: 4 link: 0 from 469 to 1397
Sep 26 22:18:26 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 26 22:18:26 pve corosync[879]: [QUORUM] Sync joined[1]: 4
Sep 26 22:18:26 pve corosync[879]: [TOTEM ] A new membership (1.f28) was formed. Members joined: 4
Sep 26 22:18:26 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 26 22:18:26 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
 
Sep 27 11:12:34 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 11:12:34 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:12:34 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 11:12:37 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 11:12:37 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:38:58 pve corosync[879]: [QUORUM] Sync members[3]: 1 3 4
Sep 27 11:38:58 pve corosync[879]: [QUORUM] Sync left[1]: 2
Sep 27 11:38:58 pve corosync[879]: [TOTEM ] A new membership (1.f2c) was formed. Members left: 2
Sep 27 11:38:58 pve corosync[879]: [QUORUM] Members[3]: 1 3 4
Sep 27 11:38:58 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 11:39:00 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 11:39:00 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:39:00 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 11:40:47 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 11:40:47 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:40:47 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 11:40:47 pve corosync[879]: [QUORUM] Sync joined[1]: 2
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] A new membership (1.f31) was formed. Members joined: 2
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 1
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 8
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: b
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 10
Sep 27 11:40:47 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 11:40:47 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 13
Sep 27 12:10:43 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:10:43 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:10:43 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:10:46 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:10:47 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:10:47 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:10:47 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:10:50 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:10:52 pve corosync[879]: [TOTEM ] Token has not been received in 9590 ms
Sep 27 12:10:53 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:10:53 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:10:53 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:10:53 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:10:56 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:10:56 pve corosync[879]: [TOTEM ] A new membership (1.f39) was formed. Members
Sep 27 12:10:56 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:10:56 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:00 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:04 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:21 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:11:21 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:21 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:11:24 pve corosync[879]: [TOTEM ] Token has not been received in 3226 ms
Sep 27 12:11:25 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:26 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:11:26 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:26 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:26 pve corosync[879]: [TOTEM ] A new membership (1.f3d) was formed. Members
Sep 27 12:11:26 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:26 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:31 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:32 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:32 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:11:32 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:11:32 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:11:36 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:11:36 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:37 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:11:37 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:11:37 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:11:37 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:11:37 pve corosync[879]: [QUORUM] Sync members[2]: 1 4
Sep 27 12:11:37 pve corosync[879]: [QUORUM] Sync left[2]: 2 3
Sep 27 12:11:37 pve corosync[879]: [TOTEM ] A new membership (1.f41) was formed. Members left: 2 3
Sep 27 12:11:37 pve corosync[879]: [TOTEM ] Failed to receive the leave message. failed: 2 3
Sep 27 12:11:37 pve corosync[879]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 27 12:11:37 pve corosync[879]: [QUORUM] Members[2]: 1 4
Sep 27 12:11:37 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:38 pve corosync[879]: [QUORUM] Sync members[2]: 1 4
Sep 27 12:11:38 pve corosync[879]: [TOTEM ] A new membership (1.f45) was formed. Members
Sep 27 12:11:39 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:39 pve corosync[879]: [QUORUM] Sync joined[2]: 2 3
Sep 27 12:11:39 pve corosync[879]: [TOTEM ] A new membership (1.f49) was formed. Members joined: 2 3
Sep 27 12:11:39 pve corosync[879]: [QUORUM] This node is within the primary component and will provide service.
Sep 27 12:11:39 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:39 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:12:10 pve corosync[879]: [TOTEM ] Token has not been received in 3579 ms
Sep 27 12:12:14 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:12:14 pve corosync[879]: [TOTEM ] A new membership (1.f4d) was formed. Members
Sep 27 12:12:14 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:12:14 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:13:57 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:13:57 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:13:57 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:13:59 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:13:59 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:13:59 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:13:59 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:13:59 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:13:59 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:13:59 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:14:16 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:14:16 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:14:17 pve corosync[879]: [TOTEM ] Token has not been received in 3436 ms
Sep 27 12:14:18 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:14:19 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:14:19 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:14:19 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:14:19 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:14:19 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:14:19 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:14:26 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:14:26 pve corosync[879]: [TOTEM ] A new membership (1.f51) was formed. Members
Sep 27 12:14:29 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:14:29 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:15:49 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:22:52 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:23:03 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:23:04 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:23:05 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:23:05 pve corosync[879]: [TOTEM ] A new membership (1.f55) was formed. Members
Sep 27 12:23:05 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:23:05 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:23:51 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:24:58 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
root@pve:~#

-----------------------------
 
oot@pve-2:~# journalctl -b 0 -u corosync
Journal file /var/log/journal/a52871dde0ce40bb809e41092678d0c5/system.journal is truncated, ignoring file.
-- Journal begins at Tue 2022-09-20 12:40:53 IST, ends at Sat 2022-10-01 11:47:16 IST. --
-- No entries --
root@pve-2:~#
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom