Random node reboots

tema228

New Member
Oct 27, 2026
4
0
1
Good afternoon, Proxmox 8.4 is installed; the node is part of a cluster.
There are no logs from when the node went down; just "-- Reboot --"
What was done:
1) BIOS updated
2) amd64-microcode package installed
3) Power supply unit replaced
4) CPU stress test
5) RAM test

Could you tell me what might be causing this?

1793091974574.png
 

Attachments

  • 1793092102419.png
    1793092102419.png
    119.3 KB · Views: 0
Last edited:
It might be due to node fencing, when a node hosts HA resources, and it loses corosync quorum for over 50-60 seconds it will fence itself (reboot). If the network becomes unusable from a majority of nodes in the cluster, that would result in the entire cluster rebooting. See [0] for more info about fencing.

As you know corosync is sensitive to latency changes and network stability and therefore needs a dedicated NIC. The dedicated NIC does not need much bandwidth, 1G is usually recommended.

If that's the case, then ensure to follow the below recommendation:
From Proxmox VE 9 documentation [1]:
  • Recommended to use a dedicated NIC exclusively for Corosync communication.
  • Configure the migration network on a different interface from the Corosync interface:
    Datacenter → Options → Migration Settings. Please make sure to read our wiki 5.14.2. Migration Network for more details [2].
  • Recommended to give Corosync at least a secondary link for redundancy.
  • Recommended to not use a network bond for Corosync's primary link, see [3] for more details.
[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#ha_manager_fencing
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_cluster_requirements
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_migration_network
[3] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_corosync_over_bonds
 
Last edited:
It might be due to node fencing, when a node hosts HA resources, and it loses corosync quorum for over 50-60 seconds it will fence itself (reboot). If the network becomes unusable from a majority of nodes in the cluster, that would result in the entire cluster rebooting. See [0] for more info about fencing.

As you know corosync is sensitive to latency changes and network stability and therefore needs a dedicated NIC. The dedicated NIC does not need much bandwidth, 1G is usually recommended.

If that's the case, then ensure to follow the below recommendation:
From Proxmox VE 9 documentation [1]:
  • Recommended to use a dedicated NIC exclusively for Corosync communication.
  • Configure the migration network on a different interface from the Corosync interface:
    Datacenter → Options → Migration Settings. Please make sure to read our wiki 5.14.2. Migration Network for more details [2].
  • Recommended to give Corosync at least a secondary link for redundancy.
  • Recommended to not use a network bond for Corosync's primary link, see [3] for more details.
[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#ha_manager_fencing
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_cluster_requirements
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_migration_network
[3] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_corosync_over_bonds
There is a dedicated NIC. The problem is not with the network. I was running Proxmox version 7.4 and had no issues. I’m thinking of rolling back.
 
I was running Proxmox version 7.4 and had no issues. I’m thinking of rolling back.
From my point of view this is only an option if you are living under a rock - and the PVE has no network connectivity. None.

Unfortunately hard reboots are not easy to debug, sometimes.
There are no logs from when the node went down; just "-- Reboot --"
This should not happen, of course. Sometimes a remote log helps to catch messages that could not be stored locally for any reason. I utilize a "remote journal" for this.

I can only imagine a hardware induced trigger event. Did you check the BIOS for error logging? (Simpler BIOSes don't tell you much; Dell iDRAC and others have a luxury amount of hardware related events being logged - the availability depends heavily on the manufacturer...)

The other nodes have the same hardware? Probably you hesitate to create the same trouble there..., but that would be the one possible way to go.
 
Last edited:
I am attaching a file with tests.
Well...

That file gets saved as "px3_diagnostics_summary.zip" - which is a lie not the case. Inside of it there is a *.rar archive. I stopped there.

But then my curiosity won. I installed an appropriate software package - exclusively to be able to read that file and (possibly) to help you.

After extraction I found... a file named "px3_diagnostics_summary.docx". And I have no tooling for that kind of stuff readily available on this machine - and I won't install LibreOffice only to read your report.

Sorry, I'm out...
 
Last edited:
Well...

That file gets saved as "px3_diagnostics_summary.zip" - which is a lie not the case. Inside of it there is a *.rar archive. I stopped there.

But then my curiosity won. I installed an appropriate software package - exclusively to be able to read that file and (possibly) to help you.

After extraction I found... a file named "px3_diagnostics_summary.docx". And I have no tooling for that kind of stuff readily available on this machine - and I won't install LibreOffice only to read your report.

Sorry, I'm out...
px3-office — Summary of Reboot Diagnostics

Investigation materials as of 20.10.2026. Main series of load tests: 15–19.10.2026. Log timestamps are UTC+3. The configuration and PCIe topology below refer to the system prior to the motherboard replacement on 20.10.2026, unless otherwise stated.

1. Manifestation of the Fault

The node periodically reboots without a logged orderly OS shutdown. Initially, a correlation was found between reboots and incoming replication of CT 104. Later, the fault was reproduced without pvesr: with a local zfs send | zfs receive from the NVMe pool to the SATA pool concurrently with incoming iperf3 TCP traffic.

In the available logs, there is no panic, Oops, MCE, or SATA timeout immediately before the investigated failures. After boot, the pools returned to ONLINE, and the READ/WRITE/CKSUM counters in the checks shown were zero.

The fault does not occur on every run. Operations of the same type completed successfully in some cases. State was not recorded separately.

2. Configuration (prior to motherboard replacement)

ComponentParameters
ProxmoxBranch PVE 8.4; in the original configuration pve-manager 8.4.19
Kernel px36.8.12-9-pve, after update — 6.8.12-43-pve
OpenZFS px32.2.7-pve2, after update — 2.2.10-pve1
Kernel px5During comparative testing — 6.8.12-9-pve
BootLegacy BIOS, GRUB, root file system on ZFS
Pool ssdMirror of 2× Samsung 870 QVO 4 TB, firmware SVQ02B6Q, AMD AHCI
Management px3192.168.77.40/24, enp4s0f0 → vmbr0
Replication and Corosync px3172.20.20.3/24, enp4s0f1 → vmbr1 → vmbr1.200, VLAN 200
Addresses px5Management 192.168.77.9; data 172.20.20.5
Both network ports used on px3 belong to a single BCM5720. Checking via management and data changed the port and network route, but not the adapter model or driver.

3. Changes and Preliminary Checks

ActionResult / data limitation
BIOS update and installation of amd64-microcodeReboots continued
Power-supply replacementIssue persisted
CPU stress testConducted as noted in the report; program, parameters, and detailed results not saved
MemTest of mixed ECC/non-ECC kitAccording to the report, the test did not pass
Separate verification of memory kitsNo errors found; this does not confirm the correctness of all subsequent memory configurations
Kernel and OpenZFS updateFailure persisted
4. Online Testing

The following snapshot was used:

text

ssd/subvol-104-disk-0@diag-px3-20261016-base

Full stream — about 65.3 billion bytes, or 60.8 GiB. The resulting copy, when compressed, occupied about 28.2 GiB. Separate diagnostic datasets were created for manual receive tests; the working CT 104 dataset was not used as their recipient.

TestResult
Only iperf3: 400, 560, 720, 880, 930 Mb/sShort tests completed without resets or TCP retransmits. Separate longer runs at 720 Mb/s also worked
fio-3.33, sequential 128 KiB write to SATAShort tests without reset. Used ioengine=sync, direct=0, end_fsync=1. Multi-second delays were observed
Short fio run together with iperf3Successful
Full zfs send → SSH → /dev/null on px3Successful
Local full receive in NVMe rpoolSuccessful
Full remote receive in NVMe on the new kernelSuccessful via management and data. In alternating… The statement "NVMe never failed" does not match the results
Checking Replication Launch Methods

Used:

bash

pvesr run --id 104-1 --verbose

pvesr schedule-now 104-1

On the new kernel, the transfer of about 4.62 GiB with rate=200 and the transfer of about 1.62 GiB without the rate parameter completed successfully. After testing, rate=50 was restored.

The difference between direct launch and launch via the scheduler was not identified as the cause of the failure. The failure was reproduced even without pvesr.

5. Concise Commands of Performed Tests

The commands are provided to document the methodology. In the tests, tmux, separate dataset names, set -o pipefail, logging of output, and exit-code checking were used. zfs receive -F was not applied. SSH parameters for host-key verification are omitted below.

Remote Receive on SATA

On px5:

bash

zfs send -Pv ssd/subvol-104-disk-0@diag-px3-20261016-base |

ssh -T root@172.20.20.3 \

'zfs receive -u -o mountpoint=none -o canmount=off'

netconsole via the Management Network

text

/var/lib/px3-crash/netconsole/kernel.log

On the px3 system, /root/px3-io-watch.py was created. Every second it collected CPU usage, I/O wait, MemAvailable, RX on two interfaces, completed writes and their average latency, outstanding I/O, dmu_tx counters, and the last completed TXG. Lines prefixed with PX3_IO_DIAG were sent through /dev/kmsg to netconsole.

Failure 19 Oct 2026, 14:24

TimeObserved
14:24:09Data-interface RX 774 Mbit/s, write to Samsung 131/132 MB/s
14:24:10RX drops to 59 Mbit/s; at 14:24:10.949 the neighboring node records a Corosync link down
14:24:11–19Data-interface RX 0; monitor keeps running, messages arrive via management, both Samsungs continue completing writes
14:24:19Last measurement: write 39/42 MB/s, awaitW 0.7/28.1 ms, inflight 0/10, MemAvailable 118.6 GiB
14:24:40Start of a new boot according to uptime -s
14:25:05–06Node returns to Corosync
The metrics refer to completed operations, not the age of pending requests. The TXG sync value refers to the last completed transaction group, not the current wait time. These metrics do not establish the cause of the reset.

6. Kdump, Watchdog and Crash Logs

Configured:

text

crashkernel=1G

kernel.nmi_watchdog=1

kernel.softlockup_panic=1

kernel.hardlockup_panic=1

kernel.panic=60

softdog: soft_panic=1

Before the load tests, the following were checked:

bash

cat /sys/kernel/kexec_crash_loaded

kdump-config status

systemctl is-active px3-netconsole.service

Returned 1, ready to kdump, and active. Kdump is configured to send via SSH to px6. The watchdog-mux was running; in one of the checks the local HA LRM was idle.

A full vmcore from real failures was not obtained. In the shown file lists, only dmesg.202610141437 of size 97 682 bytes was present. Successful saving of a full crash dump has not been confirmed.

The absence of a dump does not establish hardware cause. The loads were zero. Network error/drop counters after reboot were also zero. The moment of setting PCIe flags and their relation to failures was not recorded.

On several boots there was TSC unstable relative to HPET; the clocksource hpet was used. The warning persisted after the BIOS update. Its connection to the reboots has not been established.

7. Latest Confirmed State and Diagnostic Data

During the completion of the test series:

  • 104-0: replication px5 → px2 was temporarily disabled, then re-enabled and completed successfully.
  • 104-1: replication px5 → px3 enabled, returned rate=50 MB/s.
  • The quota rpool/diag-recv was increased from 128 to 200 GiB for additional copies.
  • On px6 the incident directory /var/lib/px3-crash/incident-20261019-142524/ was saved with a copy of kernel.log.
Created rpool/diag-recv/*, ssd/diag-write-test, ssd/diag-local-recv, as well as datasets named ssd/diag-remote-recv*, ssd/diag-mgmt-recv and others.

8. Conclusions

Based on the available data, it was not possible to identify the specific faulty component and the reboot mechanism. Replacing the motherboard and power supply, updating BIOS, microcode, kernel, and OpenZFS did not resolve the problem.

There are no results of testing via Intel I211, moving the Samsung to another SATA controller, CPU replacement, tests in SystemRescue or other independent boot environments in the materials. The model of the new motherboard and the final memory configuration need clarification.