Proxmox clusters node reboot at same time

[Joran_C]

We are experienced something very weird last night, we manage a few proxmox clusters for our customers and they had a node reboot at roughly the same time.
The syslog also throws a very weird format.

We are experiencing this issue with a couple of customers around the same time +-10 mins

All clusters run PVE7.0-13/PVE7.1-12

May 29 19:25:01 proxmox06 CRON[255581]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
May 29 19:27:34 proxmox06 pmxcfs[2196]: [dcdb] notice: data verification successful
May 29 19:30:33 proxmox06 smartd[1134]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 71 to 70
May 29 19:30:33 proxmox06 smartd[1134]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 71 to 70
May 29 19:35:01 proxmox06 CRON[259587]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
May 29 19:45:01 proxmox06 CRON[263582]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May 29 20:20:01 proxmox06-zav systemd-modules-load[717]: Inserted module 'iscsi_tcp'
May 29 20:20:01 proxmox06 kernel: [    0.000000] Linux version 5.13.19-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100) ()
May 29 20:20:01 proxmox06 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.19-2-pve root=/dev/mapper/pve-root ro
May 29 20:20:01 proxmox06 kernel: [    0.000000] KERNEL supported cpus:
May 29 20:20:01 proxmox06 kernel: [    0.000000]   Intel GenuineIntel
May 29 20:20:01 proxmox06 systemd-modules-load[717]: Inserted module 'ib_iser'
May 29 20:20:01 proxmox06 kernel: [    0.000000]   AMD AuthenticAMD

 

[gurubert]

Null bytes in the logfile give a hint that the cause of the issue was a power loss. The Proxmox hosts were not shutdown correctly.

 

[Joran_C]

Null bytes in the logfile give a hint that the cause of the issue was a power loss. The Proxmox hosts were not shutdown correctly.

We did think the same, the actual weird part is they are located in different datacenters, We looked at graphs from our power distribution panels and saw no indication of a power loss.