I have three node cluster and one of the node just restarted. I tried to look something interesting in logs but couldn't find anything beside this:
what does it mean? why one of the node restarted?
what does it mean? why one of the node restarted?
Bash:
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: Started User Manager for UID 0.
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: Started Session 931 of user root.
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: session-931.scope: Succeeded.
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: Started Session 933 of user root.
Aug 26 21:40:17 proxmox-prod-2 qm[47240]: <root@pam> starting task UPID:proxmox-prod-2:0000B889:123CB797:6127EE21:qmstart:121:root@pam:
Aug 26 21:40:17 proxmox-prod-2 qm[47241]: start VM 121: UPID:proxmox-prod-2:0000B889:123CB797:6127EE21:qmstart:121:root@pam:
Aug 26 21:40:18 proxmox-prod-2 systemd[1]: Started 121.scope.
Aug 26 21:40:18 proxmox-prod-2 systemd-udevd[47210]: Using default interface naming scheme 'v240'.
Aug 26 21:40:18 proxmox-prod-2 systemd-udevd[47210]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:18 proxmox-prod-2 systemd-udevd[47210]: Could not generate persistent MAC address for tap121i0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: Could not generate persistent MAC address for vmbr0v242: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: Using default interface naming scheme 'v240'.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: Could not generate persistent MAC address for fwbr121i0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: Could not generate persistent MAC address for fwpr121p0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47211]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47211]: Using default interface naming scheme 'v240'.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47211]: Could not generate persistent MAC address for fwln121i0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 qm[47240]: <root@pam> end task UPID:proxmox-prod-2:0000B889:123CB797:6127EE21:qmstart:121:root@pam: OK
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]: [KNET ] link: host: 3 link: 0 is down
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]: [KNET ] link: host: 1 link: 0 is down
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]: [KNET ] host: host: 3 has no active links
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]: [KNET ] host: host: 1 has no active links
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]: [TOTEM ] Token has not been received in 2737 ms
Aug 26 21:40:22 proxmox-prod-2 corosync[3088]: [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms), waiting 4380ms for consensus.
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]: [QUORUM] Sync members[1]: 2
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]: [QUORUM] Sync left[2]: 1 3
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]: [TOTEM ] A new membership (2.f5) was formed. Members left: 1 3
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]: [TOTEM ] Failed to receive the leave message. failed: 1 3
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: members: 2/2932
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [status] notice: members: 2/2932
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]: [QUORUM] Members[1]: 2
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [status] notice: node lost quorum
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: received write while not quorate - trigger resync
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: leaving CPG group
Aug 26 21:40:27 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: start cluster connection
Aug 26 21:40:27 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: cpg_join failed: 14
Aug 26 21:40:27 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: can't initialize service
Aug 26 21:40:27 proxmox-prod-2 pve-ha-lrm[3355]: lost lock 'ha_agent_proxmox-prod-2_lock - cfs lock update failed - Device or resource busy
Aug 26 21:40:27 proxmox-prod-2 pve-ha-crm[3150]: status change slave => wait_for_quorum
Aug 26 21:40:28 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online
Aug 26 21:40:29 proxmox-prod-2 pvestatd[3110]: status update time (5.373 seconds)
Aug 26 21:40:31 proxmox-prod-2 pve-ha-lrm[3355]: status change active => lost_agent_lock
Aug 26 21:40:33 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: members: 2/2932
Aug 26 21:40:33 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: all data is up to date
Aug 26 21:40:38 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online
Aug 26 21:40:38 proxmox-prod-2 pvestatd[3110]: status update time (5.392 seconds)
Aug 26 21:40:48 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online
Aug 26 21:40:48 proxmox-prod-2 pvestatd[3110]: status update time (5.375 seconds)
Aug 26 21:40:58 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online