C
Chris Rivera
Guest
This has took me some time to track down and find out what has happened and I still don't know exactly what happened.
It seems that proxmox (on my production cloud) detected 1 cpu failed and literally shutdown all vms and rebooted its self.
Sep 4 16:44:21 proxmox1 corosync[648411]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 16:47:04 proxmox1 shutdown[649841]: shutting down for system reboot
When it came back online i checked the logs again and see:
Sep 4 17:10:44 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 31 32
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 4 17:11:53 proxmox1 corosync[1482]: [CPG ] chosen downlist: sender r(0) ip(63.217.249.158) ; members(old:8 left:0)
Sep 4 17:11:53 proxmox1 corosync[1482]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 4 17:12:19 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 17:13:30 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 31 32
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 70 71 72 73 75 76 77 78 79 7a 7b
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: c7 c8 c9 ca cb cc cd ce cf d0
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 4 17:13:44 proxmox1 corosync[1482]: [CPG ] chosen downlist: sender r(0) ip(63.217.249.158) ; members(old:8 left:0)
Sep 4 17:13:44 proxmox1 corosync[1482]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 4 17:26:01 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:29:02 proxmox1 corosync[1482]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 4 17:29:12 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
WHAT IS GOING ON HERE????
It seems that proxmox (on my production cloud) detected 1 cpu failed and literally shutdown all vms and rebooted its self.
Sep 4 16:44:21 proxmox1 corosync[648411]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 16:47:04 proxmox1 shutdown[649841]: shutting down for system reboot
When it came back online i checked the logs again and see:
Sep 4 17:10:44 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 31 32
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:11:53 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:11:53 proxmox1 corosync[1482]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 4 17:11:53 proxmox1 corosync[1482]: [CPG ] chosen downlist: sender r(0) ip(63.217.249.158) ; members(old:8 left:0)
Sep 4 17:11:53 proxmox1 corosync[1482]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 4 17:12:19 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 17:13:30 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 31 32
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 70 71 72 73 75 76 77 78 79 7a 7b
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] Retransmit List: c7 c8 c9 ca cb cc cd ce cf d0
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:13:44 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:13:44 proxmox1 corosync[1482]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 4 17:13:44 proxmox1 corosync[1482]: [CPG ] chosen downlist: sender r(0) ip(63.217.249.158) ; members(old:8 left:0)
Sep 4 17:13:44 proxmox1 corosync[1482]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 4 17:26:01 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] CLM CONFIGURATION CHANGE
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] New Configuration:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.154)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.155)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.156)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.157)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.158)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.159)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.160)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] #011r(0) ip(63.217.249.161)
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Left:
Sep 4 17:29:02 proxmox1 corosync[1482]: [CLM ] Members Joined:
Sep 4 17:29:02 proxmox1 corosync[1482]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Sep 4 17:29:12 proxmox1 corosync[1482]: [TOTEM ] A processor failed, forming new configuration.
WHAT IS GOING ON HERE????