unexpected restart of pve node

popey

New Member
Jul 24, 2020
22
0
1
39
I have three node cluster and one of the node just restarted. I tried to look something interesting in logs but couldn't find anything beside this:

what does it mean? why one of the node restarted?

Bash:
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: Started User Manager for UID 0.
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: Started Session 931 of user root.
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: session-931.scope: Succeeded.
Aug 26 21:40:16 proxmox-prod-2 systemd[1]: Started Session 933 of user root.
Aug 26 21:40:17 proxmox-prod-2 qm[47240]: <root@pam> starting task UPID:proxmox-prod-2:0000B889:123CB797:6127EE21:qmstart:121:root@pam:
Aug 26 21:40:17 proxmox-prod-2 qm[47241]: start VM 121: UPID:proxmox-prod-2:0000B889:123CB797:6127EE21:qmstart:121:root@pam:
Aug 26 21:40:18 proxmox-prod-2 systemd[1]: Started 121.scope.
Aug 26 21:40:18 proxmox-prod-2 systemd-udevd[47210]: Using default interface naming scheme 'v240'.
Aug 26 21:40:18 proxmox-prod-2 systemd-udevd[47210]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:18 proxmox-prod-2 systemd-udevd[47210]: Could not generate persistent MAC address for tap121i0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: Could not generate persistent MAC address for vmbr0v242: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: Using default interface naming scheme 'v240'.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47210]: Could not generate persistent MAC address for fwbr121i0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47236]: Could not generate persistent MAC address for fwpr121p0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47211]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47211]: Using default interface naming scheme 'v240'.
Aug 26 21:40:19 proxmox-prod-2 systemd-udevd[47211]: Could not generate persistent MAC address for fwln121i0: No such file or directory
Aug 26 21:40:19 proxmox-prod-2 qm[47240]: <root@pam> end task UPID:proxmox-prod-2:0000B889:123CB797:6127EE21:qmstart:121:root@pam: OK
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]:   [KNET  ] link: host: 3 link: 0 is down
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]:   [KNET  ] link: host: 1 link: 0 is down
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]:   [KNET  ] host: host: 3 has no active links
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]:   [KNET  ] host: host: 1 has no active links
Aug 26 21:40:21 proxmox-prod-2 corosync[3088]:   [TOTEM ] Token has not been received in 2737 ms
Aug 26 21:40:22 proxmox-prod-2 corosync[3088]:   [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms), waiting 4380ms for consensus.
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]:   [QUORUM] Sync members[1]: 2
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]:   [QUORUM] Sync left[2]: 1 3
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]:   [TOTEM ] A new membership (2.f5) was formed. Members left: 1 3
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]:   [TOTEM ] Failed to receive the leave message. failed: 1 3
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: members: 2/2932
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [status] notice: members: 2/2932
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]:   [QUORUM] Members[1]: 2
Aug 26 21:40:26 proxmox-prod-2 corosync[3088]:   [MAIN  ] Completed service synchronization, ready to provide service.
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [status] notice: node lost quorum
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: received write while not quorate - trigger resync
Aug 26 21:40:26 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: leaving CPG group
Aug 26 21:40:27 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: start cluster connection
Aug 26 21:40:27 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: cpg_join failed: 14
Aug 26 21:40:27 proxmox-prod-2 pmxcfs[2932]: [dcdb] crit: can't initialize service
Aug 26 21:40:27 proxmox-prod-2 pve-ha-lrm[3355]: lost lock 'ha_agent_proxmox-prod-2_lock - cfs lock update failed - Device or resource busy
Aug 26 21:40:27 proxmox-prod-2 pve-ha-crm[3150]: status change slave => wait_for_quorum
Aug 26 21:40:28 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online
Aug 26 21:40:29 proxmox-prod-2 pvestatd[3110]: status update time (5.373 seconds)
Aug 26 21:40:31 proxmox-prod-2 pve-ha-lrm[3355]: status change active => lost_agent_lock
Aug 26 21:40:33 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: members: 2/2932
Aug 26 21:40:33 proxmox-prod-2 pmxcfs[2932]: [dcdb] notice: all data is up to date
Aug 26 21:40:38 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online
Aug 26 21:40:38 proxmox-prod-2 pvestatd[3110]: status update time (5.392 seconds)
Aug 26 21:40:48 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online
Aug 26 21:40:48 proxmox-prod-2 pvestatd[3110]: status update time (5.375 seconds)
Aug 26 21:40:58 proxmox-prod-2 pvestatd[3110]: storage 'nas-backup' is not online
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!