Proxmox clusters node reboot at same time

Joran_C

New Member
Sep 27, 2020
7
0
1
23
We are experienced something very weird last night, we manage a few proxmox clusters for our customers and they had a node reboot at roughly the same time.
The syslog also throws a very weird format.

We are experiencing this issue with a couple of customers around the same time +-10 mins

All clusters run PVE7.0-13/PVE7.1-12

Code:
May 29 19:25:01 proxmox06 CRON[255581]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
May 29 19:27:34 proxmox06 pmxcfs[2196]: [dcdb] notice: data verification successful
May 29 19:30:33 proxmox06 smartd[1134]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 71 to 70
May 29 19:30:33 proxmox06 smartd[1134]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 71 to 70
May 29 19:35:01 proxmox06 CRON[259587]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
May 29 19:45:01 proxmox06 CRON[263582]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May 29 20:20:01 proxmox06-zav systemd-modules-load[717]: Inserted module 'iscsi_tcp'
May 29 20:20:01 proxmox06 kernel: [    0.000000] Linux version 5.13.19-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100) ()
May 29 20:20:01 proxmox06 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.19-2-pve root=/dev/mapper/pve-root ro
May 29 20:20:01 proxmox06 kernel: [    0.000000] KERNEL supported cpus:
May 29 20:20:01 proxmox06 kernel: [    0.000000]   Intel GenuineIntel
May 29 20:20:01 proxmox06 systemd-modules-load[717]: Inserted module 'ib_iser'
May 29 20:20:01 proxmox06 kernel: [    0.000000]   AMD AuthenticAMD
 

Attachments

  • logs.txt
    31.9 KB · Views: 1

Joran_C

New Member
Sep 27, 2020
7
0
1
23
Null bytes in the logfile give a hint that the cause of the issue was a power loss. The Proxmox hosts were not shutdown correctly.
We did think the same, the actual weird part is they are located in different datacenters, We looked at graphs from our power distribution panels and saw no indication of a power loss.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!