Cluster information greyed out

[Harish Kamath]

Hi,

I have a Proxmox cluster and find cluster information being greyed out. The current configuration includes two nodes with status as available. I attempted to add third node but failed.

I have tried rebooting nodes - pve and pve-2.
I recreated pve-70 (now has a new IP address). The failed join action of pve-70 is with old IP address.

I am unable to fetch "Join Information".
I am unable to remove pve-70 because it is not reachable

Output of version details of the nodes are provided below.

Request your inputs.

Thanks,

root@pve:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
root@pve:~#

root@pve-2:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
root@pve-2:~#

 

[shrdlicka]

Hi,

what does your pvecm status say?

Regards
Stefan Hrdlicka

 

[Harish Kamath]

Hi,

I get "cannot initialize CMAP service"

I suspect the issue is observed only on adding this node. Here are the steps that I followed

1. Removed the failing node (pve-70) from /etc/pve/corosync.conf
2. Cluster info section activated
3. Added a new node using Join Cluster GUI approach
4. Cluster info section is active
5. Updated /etc/pve/corosync.conf with failing node (pve-70)
6. Cluster info section is deactivated again

 

[shrdlicka]

What does your journalctl -b 0 -u corosync say? Can you call that on the new node and one of the old ones?

 

[Harish Kamath]

Hi,

I have attached the screen shots of failing node followed by one of the cluster member.

I now observe that the web connection to the faulty node is also failing with site not reachable. However, the ping succeeds from the cluster member succeeds.

 

[shrdlicka]

Try to copy /etc/corosync/authkey from one of the old nodes onto the new node. Then restart the node, an please post the output of the corosync log again from that node.

 

[Harish Kamath]

Hi,

As suggested, I copied the authkey file and observe green icon with the node. However, I still find two issues in my Datacenter cluster environment -

1. Cluster GUI portion is greyed out
2. The node site is still not accessible on the Google Chrome Browser
3. I don't understand why one of the node (pve-2) is not indicated as enabled but can access its shell

The output of the log is provided below -

journalctl -b 0 -u corosync
-- Journal begins at Tue 2022-09-20 18:40:58 IST, ends at Mon 2022-09-26 22:25:28 IST. --
Sep 26 22:18:22 pve-70 systemd[1]: Starting Corosync Cluster Engine...
Sep 26 22:18:22 pve-70 corosync[858]: [MAIN ] Corosync Cluster Engine 3.1.5 starting up
Sep 26 22:18:22 pve-70 corosync[858]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] Initializing transport (Kronosnet).
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] totemknet initialized
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync configuration map access [0]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: cmap
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync configuration service [1]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: cfg
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Sep 26 22:18:22 pve-70 systemd[1]: Started Corosync Cluster Engine.
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: cpg
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync profile loading service [4]
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] Watchdog not enabled by configuration
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] resource load_15min missing a recovery key.
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] resource memory_used missing a recovery key.
Sep 26 22:18:22 pve-70 corosync[858]: [WD ] no resources configured.
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync watchdog service [7]
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Using quorum provider corosync_votequorum
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: votequorum
Sep 26 22:18:22 pve-70 corosync[858]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Sep 26 22:18:22 pve-70 corosync[858]: [QB ] server name: quorum
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] Configuring link 0
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] Configured link number 0: local addr: 192.168.122.237, port=5405
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 1 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 0)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 2 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:22 pve-70 corosync[858]: [KNET ] host: host: 3 has no active links
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Sync members[1]: 4
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Sync joined[1]: 4
Sep 26 22:18:22 pve-70 corosync[858]: [TOTEM ] A new membership (4.eca) was formed. Members joined: 4
Sep 26 22:18:22 pve-70 corosync[858]: [QUORUM] Members[1]: 4
Sep 26 22:18:22 pve-70 corosync[858]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] rx: host: 3 link: 0 is up
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] rx: host: 2 link: 0 is up
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] rx: host: 1 link: 0 is up
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 26 22:18:25 pve-70 corosync[858]: [KNET ] pmtud: Global data MTU changed to: 469
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] Sync joined[3]: 1 2 3
Sep 26 22:18:26 pve-70 corosync[858]: [TOTEM ] A new membership (1.f28) was formed. Members joined: 1 2 3
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] This node is within the primary component and will provide service.
Sep 26 22:18:26 pve-70 corosync[858]: [QUORUM] Members[4]: 1 2 3 4
Sep 26 22:18:26 pve-70 corosync[858]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: PMTUD link change for host: 1 link: 0 from 469 to 1397
Sep 26 22:18:47 pve-70 corosync[858]: [KNET ] pmtud: Global data MTU changed to: 1397
root@pve-70:~#

 

[shrdlicka]

Hi
now there are two new nodes ( pve-70 and proxmox-v71-4 ). What does the log of pve-2 say journalctl -b 0 -u corosync? Can you post your /etc/pve/corosync.conf?

 

[Harish Kamath]

The output of "journalctl -b 0 -u corosync" on pve-2 is provided below -


Last login: Tue Sep 27 13:03:07 IST 2022 from 192.168.122.29 on pts/0
root@pve-2:~# journalctl -b 0 -u corosync
Journal file /var/log/journal/a52871dde0ce40bb809e41092678d0c5/system.journal is truncated, ignoring file.
-- Journal begins at Tue 2022-09-20 12:40:53 IST, ends at Tue 2022-09-27 13:03:07 IST. --
Sep 27 12:05:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:05:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:07 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:14 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:19 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:06:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:07:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:08:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:37 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:41 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:09:53 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:01 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:05 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:13 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:17 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:25 pve-2 corosync[738]: [TOTEM ] Retransmit List: 1f19
Sep 27 12:10:25 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:29 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:31 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:31 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:42 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:43 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:43 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:10:46 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:10:50 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:10:50 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:10:52 pve-2 corosync[738]: [TOTEM ] Token has not been received in 9078 ms
Sep 27 12:10:52 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:10:52 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:10:56 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:10:56 pve-2 corosync[738]: [TOTEM ] A new membership (1.f39) was formed. Members
Sep 27 12:10:56 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:10:56 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:04 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:07 pve-2 corosync[738]: [TOTEM ] Retransmit List: 2a 2b
Sep 27 12:11:10 pve-2 corosync[738]: [TOTEM ] Retransmit List: 31
Sep 27 12:11:12 pve-2 corosync[738]: [TOTEM ] Retransmit List: 31
Sep 27 12:11:12 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:11:12 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:12 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:11:14 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:11:14 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:24 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:25 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:26 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:26 pve-2 corosync[738]: [TOTEM ] A new membership (1.f3d) was formed. Members
Sep 27 12:11:26 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:26 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:31 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:11:31 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:31 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:11:31 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:32 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:38 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:11:38 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] Sync members[2]: 2 3
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] Sync left[2]: 1 4
Sep 27 12:11:38 pve-2 corosync[738]: [TOTEM ] A new membership (2.f45) was formed. Members left: 1 4
Sep 27 12:11:38 pve-2 corosync[738]: [TOTEM ] Failed to receive the leave message. failed: 1 4
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 27 12:11:38 pve-2 corosync[738]: [QUORUM] Members[2]: 2 3
Sep 27 12:11:38 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] Sync joined[2]: 1 4
Sep 27 12:11:39 pve-2 corosync[738]: [TOTEM ] A new membership (1.f49) was formed. Members joined: 1 4
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] This node is within the primary component and will provide service.
Sep 27 12:11:39 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:39 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:39 pve-2 corosync[738]: [TOTEM ] Retransmit List: 1b
Sep 27 12:11:39 pve-2 corosync[738]: [TOTEM ] Retransmit List: 2a
Sep 27 12:11:49 pve-2 corosync[738]: [CPG ] *** 0x5643d64bbe10 can't mcast to group pve_dcdb_v1 state:1, error:12
Sep 27 12:12:06 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:12:06 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:12:06 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:12:09 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3226 ms
Sep 27 12:12:10 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:12:10 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:12:10 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:12:14 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:12:14 pve-2 corosync[738]: [TOTEM ] A new membership (1.f4d) was formed. Members
Sep 27 12:12:14 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:12:14 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:12:16 pve-2 corosync[738]: [TOTEM ] Retransmit List: 1e
Sep 27 12:13:54 pve-2 corosync[738]: [TOTEM ] Retransmit List: 131
Sep 27 12:13:59 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:14:11 pve-2 corosync[738]: [TOTEM ] Retransmit List: 15d
Sep 27 12:14:15 pve-2 corosync[738]: [KNET ] link: host: 1 link: 0 is down
Sep 27 12:14:15 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:14:15 pve-2 corosync[738]: [KNET ] host: host: 1 has no active links
Sep 27 12:14:18 pve-2 corosync[738]: [KNET ] rx: host: 1 link: 0 is up
Sep 27 12:14:18 pve-2 corosync[738]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:14:18 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:14:19 pve-2 corosync[738]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:14:25 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:14:25 pve-2 corosync[738]: [TOTEM ] A new membership (1.f51) was formed. Members
Sep 27 12:14:29 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:14:29 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:14:39 pve-2 corosync[738]: [TOTEM ] Retransmit List: 38
Sep 27 12:15:25 pve-2 corosync[738]: [TOTEM ] Retransmit List: cb
Sep 27 12:22:49 pve-2 corosync[738]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:22:49 pve-2 corosync[738]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:22:49 pve-2 corosync[738]: [KNET ] host: host: 3 has no active links
Sep 27 12:22:52 pve-2 corosync[738]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:22:52 pve-2 corosync[738]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:23:04 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:23:05 pve-2 corosync[738]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:23:05 pve-2 corosync[738]: [TOTEM ] A new membership (1.f55) was formed. Members
Sep 27 12:23:05 pve-2 corosync[738]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:23:05 pve-2 corosync[738]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:24:58 pve-2 corosync[738]: [TOTEM ] Token has not been received in 3225 ms
root@pve-2:~#

root@pve-2:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: proxmox-v71-4
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.122.85
}
node {
name: pve
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.122.29
}
node {
name: pve-2
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.122.99
}
node {
name: pve-70
nodeid: 4
quorum_votes: 1
ring0_addr: 192.168.122.237
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: demo-cluster-1
config_version: 7
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

root@pve-2:~#

 

[shrdlicka]

There is some issue with the corosync network. It can't reach host 1 (it should be the "pve" machine) [1]. Can you check the logs there as well?


[1]

Sep 27 12:11:31 pve-2 corosync[738]:   [KNET  ] link: host: 1 link: 0 is down
Sep 27 12:11:31 pve-2 corosync[738]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 27 12:11:31 pve-2 corosync[738]:   [KNET  ] host: host: 1 has no active links
Sep 27 12:11:31 pve-2 corosync[738]:   [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:32 pve-2 corosync[738]:   [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:38 pve-2 corosync[738]:   [KNET  ] rx: host: 1 link: 0 is up
[code]

 

[Harish Kamath]

I have pasted the log output of the pve node.

root@pve:~# journalctl -b 0 -u corosync
-- Journal begins at Sat 2022-09-17 20:03:55 IST, ends at Sat 2022-10-01 11:04:24 IST. --
Sep 26 17:44:27 pve systemd[1]: Starting Corosync Cluster Engine...
Sep 26 17:44:27 pve corosync[879]: [MAIN ] Corosync Cluster Engine 3.1.5 starting up
Sep 26 17:44:27 pve corosync[879]: [MAIN ] Corosync built-in features: dbus monitoring watchdog systemd xmlconf vqsim nozzle snmp pie relro bindnow
Sep 26 17:44:27 pve corosync[879]: [TOTEM ] Initializing transport (Kronosnet).
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] totemknet initialized
Sep 26 17:44:28 pve corosync[879]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib/x86_64-linux-gnu/kronosnet/crypto_nss.so
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync configuration map access [0]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: cmap
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync configuration service [1]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: cfg
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: cpg
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync profile loading service [4]
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync resource monitoring service [6]
Sep 26 17:44:28 pve corosync[879]: [WD ] Watchdog not enabled by configuration
Sep 26 17:44:28 pve corosync[879]: [WD ] resource load_15min missing a recovery key.
Sep 26 17:44:28 pve corosync[879]: [WD ] resource memory_used missing a recovery key.
Sep 26 17:44:28 pve corosync[879]: [WD ] no resources configured.
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync watchdog service [7]
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Using quorum provider corosync_votequorum
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: votequorum
Sep 26 17:44:28 pve corosync[879]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Sep 26 17:44:28 pve corosync[879]: [QB ] server name: quorum
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] Configuring link 0
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] Configured link number 0: local addr: 192.168.122.29, port=5405
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 0)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 17:44:28 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Sync members[1]: 1
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Sync joined[1]: 1
Sep 26 17:44:28 pve corosync[879]: [TOTEM ] A new membership (1.f1c) was formed. Members joined: 1
Sep 26 17:44:28 pve corosync[879]: [QUORUM] Members[1]: 1
Sep 26 17:44:28 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 17:44:28 pve systemd[1]: Started Corosync Cluster Engine.
Sep 26 17:44:30 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 26 17:44:30 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 26 17:44:30 pve corosync[879]: [KNET ] pmtud: Global data MTU changed to: 469
Sep 26 17:44:31 pve corosync[879]: [QUORUM] Sync members[2]: 1 3
Sep 26 17:44:31 pve corosync[879]: [QUORUM] Sync joined[1]: 3
Sep 26 17:44:31 pve corosync[879]: [TOTEM ] A new membership (1.f20) was formed. Members joined: 3
Sep 26 17:44:31 pve corosync[879]: [QUORUM] Members[2]: 1 3
Sep 26 17:44:31 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 17:44:34 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 26 17:44:34 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 26 17:44:34 pve corosync[879]: [QUORUM] Sync members[3]: 1 2 3
Sep 26 17:44:34 pve corosync[879]: [QUORUM] Sync joined[1]: 2
Sep 26 17:44:34 pve corosync[879]: [TOTEM ] A new membership (1.f24) was formed. Members joined: 2
Sep 26 17:44:34 pve corosync[879]: [QUORUM] This node is within the primary component and will provide service.
Sep 26 17:44:34 pve corosync[879]: [QUORUM] Members[3]: 1 2 3
Sep 26 17:44:34 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 26 17:44:52 pve corosync[879]: [KNET ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397
Sep 26 17:44:52 pve corosync[879]: [KNET ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397
Sep 26 17:44:52 pve corosync[879]: [KNET ] pmtud: Global data MTU changed to: 1397
Sep 26 22:18:26 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 26 22:18:26 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 26 22:18:26 pve corosync[879]: [KNET ] pmtud: PMTUD link change for host: 4 link: 0 from 469 to 1397
Sep 26 22:18:26 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 26 22:18:26 pve corosync[879]: [QUORUM] Sync joined[1]: 4
Sep 26 22:18:26 pve corosync[879]: [TOTEM ] A new membership (1.f28) was formed. Members joined: 4
Sep 26 22:18:26 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 26 22:18:26 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.

 

[Harish Kamath]

Sep 27 11:12:34 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 11:12:34 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:12:34 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 11:12:37 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 11:12:37 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:38:58 pve corosync[879]: [QUORUM] Sync members[3]: 1 3 4
Sep 27 11:38:58 pve corosync[879]: [QUORUM] Sync left[1]: 2
Sep 27 11:38:58 pve corosync[879]: [TOTEM ] A new membership (1.f2c) was formed. Members left: 2
Sep 27 11:38:58 pve corosync[879]: [QUORUM] Members[3]: 1 3 4
Sep 27 11:38:58 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 11:39:00 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 11:39:00 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:39:00 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 11:40:47 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 11:40:47 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 11:40:47 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 11:40:47 pve corosync[879]: [QUORUM] Sync joined[1]: 2
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] A new membership (1.f31) was formed. Members joined: 2
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 1
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 8
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: b
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 10
Sep 27 11:40:47 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 11:40:47 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 11:40:47 pve corosync[879]: [TOTEM ] Retransmit List: 13
Sep 27 12:10:43 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:10:43 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:10:43 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:10:46 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:10:47 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:10:47 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:10:47 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:10:50 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:10:50 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:10:52 pve corosync[879]: [TOTEM ] Token has not been received in 9590 ms
Sep 27 12:10:53 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:10:53 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:10:53 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:10:53 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:10:56 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:10:56 pve corosync[879]: [TOTEM ] A new membership (1.f39) was formed. Members
Sep 27 12:10:56 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:10:56 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:00 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:04 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:21 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:11:21 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:21 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:11:24 pve corosync[879]: [TOTEM ] Token has not been received in 3226 ms
Sep 27 12:11:25 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:26 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:11:26 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:26 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:26 pve corosync[879]: [TOTEM ] A new membership (1.f3d) was formed. Members
Sep 27 12:11:26 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:26 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:31 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:11:32 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:11:32 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:11:32 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:11:32 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:11:32 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:11:36 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:11:36 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:11:37 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:11:37 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:11:37 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:11:37 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:11:37 pve corosync[879]: [QUORUM] Sync members[2]: 1 4
Sep 27 12:11:37 pve corosync[879]: [QUORUM] Sync left[2]: 2 3
Sep 27 12:11:37 pve corosync[879]: [TOTEM ] A new membership (1.f41) was formed. Members left: 2 3
Sep 27 12:11:37 pve corosync[879]: [TOTEM ] Failed to receive the leave message. failed: 2 3
Sep 27 12:11:37 pve corosync[879]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 27 12:11:37 pve corosync[879]: [QUORUM] Members[2]: 1 4
Sep 27 12:11:37 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:11:38 pve corosync[879]: [QUORUM] Sync members[2]: 1 4
Sep 27 12:11:38 pve corosync[879]: [TOTEM ] A new membership (1.f45) was formed. Members
Sep 27 12:11:39 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:11:39 pve corosync[879]: [QUORUM] Sync joined[2]: 2 3
Sep 27 12:11:39 pve corosync[879]: [TOTEM ] A new membership (1.f49) was formed. Members joined: 2 3
Sep 27 12:11:39 pve corosync[879]: [QUORUM] This node is within the primary component and will provide service.
Sep 27 12:11:39 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:11:39 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:12:10 pve corosync[879]: [TOTEM ] Token has not been received in 3579 ms
Sep 27 12:12:14 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:12:14 pve corosync[879]: [TOTEM ] A new membership (1.f4d) was formed. Members
Sep 27 12:12:14 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:12:14 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:13:57 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:13:57 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:13:57 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:13:57 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:13:59 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:13:59 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:13:59 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:13:59 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:13:59 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:13:59 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:13:59 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] link: host: 4 link: 0 is down
Sep 27 12:14:16 pve corosync[879]: [KNET ] link: host: 2 link: 0 is down
Sep 27 12:14:16 pve corosync[879]: [KNET ] link: host: 3 link: 0 is down
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 4 has no active links
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 2 has no active links
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:14:16 pve corosync[879]: [KNET ] host: host: 3 has no active links
Sep 27 12:14:17 pve corosync[879]: [TOTEM ] Token has not been received in 3436 ms
Sep 27 12:14:18 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:14:19 pve corosync[879]: [KNET ] rx: host: 4 link: 0 is up
Sep 27 12:14:19 pve corosync[879]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 27 12:14:19 pve corosync[879]: [KNET ] rx: host: 2 link: 0 is up
Sep 27 12:14:19 pve corosync[879]: [KNET ] rx: host: 3 link: 0 is up
Sep 27 12:14:19 pve corosync[879]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 27 12:14:19 pve corosync[879]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 27 12:14:26 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:14:26 pve corosync[879]: [TOTEM ] A new membership (1.f51) was formed. Members
Sep 27 12:14:29 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:14:29 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:15:49 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:22:52 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:23:03 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:23:04 pve corosync[879]: [TOTEM ] A processor failed, forming new configuration: token timed out (4300ms), waiting 5160ms for consensus.
Sep 27 12:23:05 pve corosync[879]: [QUORUM] Sync members[4]: 1 2 3 4
Sep 27 12:23:05 pve corosync[879]: [TOTEM ] A new membership (1.f55) was formed. Members
Sep 27 12:23:05 pve corosync[879]: [QUORUM] Members[4]: 1 2 3 4
Sep 27 12:23:05 pve corosync[879]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 27 12:23:51 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
Sep 27 12:24:58 pve corosync[879]: [TOTEM ] Token has not been received in 3225 ms
root@pve:~#

-----------------------------

 

[Harish Kamath]

oot@pve-2:~# journalctl -b 0 -u corosync
Journal file /var/log/journal/a52871dde0ce40bb809e41092678d0c5/system.journal is truncated, ignoring file.
-- Journal begins at Tue 2022-09-20 12:40:53 IST, ends at Sat 2022-10-01 11:47:16 IST. --
-- No entries --
root@pve-2:~#