A sudden error in the cluster, 2 reasons and 2nd one me being an idiot :-)

[Arctica67]

I am maybe not a beginner with Proxmox, been running a cluster of 3 Lenovo ThinkCentre M70q at home for soon a year now, but the thing is, it's been runnig flawlessly since I did set it up, so I have no experience in troubleshooting when the shit hits the fan. Which it did today...
Around noon I received an alert email:

The node 'proxmox2' in cluster 'cluster' failed and needs manual intervention.

The PVE HA manager will now fence 'proxmox2'.

Status: Try to fence node 'proxmox2'
Timestamp: 2026-07-31 12:11:41

Cluster Node Status:​

• proxmox1: online
• proxmox2: unknown
• proxmox3: online [master]

That was worrying. But a much bigger problem, my Hermes VM was migrated to node 'proxmox3' and it was stopped with error:

Jul 31 12:23:05 proxmox3 pve-ha-lrm[3121544]: <root@pam> end task UPIDroxmox3:002FA18A:1EC3E949:6A6C765E:qmstart:210:root@pam: timeout: no zvol device link for 'vm-210-disk-0' found after 300 sec.
Jul 31 12:23:05 proxmox3 pve-ha-lrm[3121544]: unable to start service vm:210

No matter how I tried to start or stop the node on either node 2 or 3, I couldn't start up vm 210 (Hermes). Could not migrate back, could not issue qm unlock 210 or anything else.
I have both VM and file-based backup to my NAS, but still I was panicking a bit. What the hell has happened? I googled and learned that the first thing to do so at least I get my Hermes VM up and running on node 2 was to go to node 3 and directory /etc/pve/qemu-server/ and there move the 210.conf file to same location on node 2, and then I could start up my VM again. That was a relief, but what the hell had happened in the first place?

I took the logfile of node 2 and asked Claude to analyze it: The answer was clear:

Root cause (12:10:45–12:11:40)​

12:10:45 corosync: links to node 1 AND node 3 both go down
12:10:46 kernel: e1000e 0000:00:1f.6 nic0: Detected Hardware Unit Hang
(repeats every ~2 seconds for about 60 seconds)
12:10:51 proxmox2 loses quorum ("non-primary component")
12:10:51 pve-ha-lrm: loses its HA lock ("lost lock ... Device or resource busy")
12:11:29 watchdog-mux: "watchdog is about to expire"
12:11:39 watchdog-mux: "watchdog expired - disable watchdog updates"
12:11:40 kernel: "watchdog0: watchdog did not stop!" → hard reset
12:12:08 node comes back up

The physical NIC on node 2 (Intel e1000e, PCI 0000:00:1f.6, shows up as eno2/nic0, the only port in vmbr0) hung at the hardware level — the transmit descriptor ring (TDH) stopped advancing and the interface stopped responding. This is a well-known e1000e failure mode ("Hardware Unit Hang"), often tied to PCIe power-saving (ASPM) or TSO/GSO offload interacting badly with the driver.


Since all traffic on this node — cluster communication (corosync), storage, and VM networking — runs over that single NIC, the whole node got cut off from the cluster the instant the card hung.

So the NIC was the root cause! I find it odd that this happens after almost a year of usage, but I guess that I have been pushing a lot of tasks on Hermes in the last day, so more network traffic maybe provokes the error? The fix according to Claude:

1. Separate the cluster network from everything else (most important) — right now corosync, storage, and VM traffic all share the single NIC. If you have or can add a second network card, add it as a second corosync link (corosync.conf → second ring/link) so the node doesn't lose quorum just because one card hangs. Even a cheap USB NIC or a second onboard interface would give you redundancy.
2. Mitigate the e1000e bug for now:

bash
ethtool -K eno2 tso off gso off gro off

Consider making this persistent (a systemd service, or a post-up line in /etc/network/interfaces), since this is a known workaround for this type of hang on some e1000e hardware.

1. Try disabling PCIe ASPM if it recurs — add pcie_aspm=off to the kernel command line (GRUB / /etc/kernel/cmdline), since power-saving features are a common culprit for exactly this NIC failure on this class of onboard chip.
2. Update pve-kernel — check pveversion -v against the latest pve-kernel package; e1000e-related fixes show up periodically in newer kernel builds.
3. If you'd like an early warning next time instead of finding out after the fact: a simple monitoring script that alerts on Hardware Unit Hang in dmesg/the journal would flag it before the watchdog has a chance to fire.

The Thinkcentre chassis are small, so maybe I am not too keen on installing a separate NIC, what would you do? The fix "ethtool -K eno2 tso off gso off gro off", do you agree that it is a good thing to do?

The next problem I had, why the live migration to node 3 didn't work was the next mystery. Asking Claude again and the answer was really embarrasing (to me) ;-)

Root cause​

• VM 210 has a configured replication job (210-0), but it's been failing on every single run with dataset does not exist — the target ZFS dataset tank/vm-210-disk-0 was never actually created on node3. Either the very first full sync never completed, or the dataset got removed/renamed on node3 at some point, and every incremental replication since has been silently failing.
  
  Because the zvol wasn't there, qm start didn't fail instantly — Proxmox waits up to 300 seconds hoping the device link appears (in case it's just slow), then times out. That's why it looked "stuck" rather than erroring immediately. HA then retried the whole start cycle repeatedly (12:12:57 → 12:18:06 → 12:23:26 → 12:26:30 → 12:31:32...) and you also triggered manual attempts via the GUI at 12:44:49 and 12:48:26 — all failing identically, for about 40 minutes straight, which is exactly why you ended up needing to intervene by hand.

I admit: I thiought I had the HA and replication correctly set up and never looked back at it. Stupid, and I have learned my lesson now. I remade the configuration, and this time replicated VM 210 to node 3, logged in and tested functionality and then moved back to node 2 again and confirmed the whole thing to work.

So I am basically telling this so nobody else is stupid like I am and set up replication without testing it that it really works. And to ask if Claude's suggestion about the fix for the NIC should be done on all 3 nodes and sounds like the correct solution?

 

[wendellrodriguez]

Do you have a external qdevice for vote ? I´m using my raspberry pi to do this job, in case I lost one node.
https://www.xda-developers.com/i-use-my-raspberry-pi-with-proxmox-but-not-in-the-way-you-think/

 

[Arctica67]

Do you have a external qdevice for vote ? I´m using my raspberry pi to do this job, in case I lost one node.
https://www.xda-developers.com/i-use-my-raspberry-pi-with-proxmox-but-not-in-the-way-you-think/

No, since I have 3 nodes, I think I should be covered if one of them goes down. But now that you mentioned it I have a couple of Rasberries lying around doing nothing so maybe I shouid add one to act as voting qdevice, thanks for the feedback!

 

[alexskysilk]

You dont need a qdevice for a 3 node cluster; it will just serve as a 4th vote that doesnt have any consequence since its not a tiebreaker anyway.

The lessons here are well learned, namely:
1. your cluster is only as stable as its network design
2. without share storage you're dependent on flakey asynchronous "replication" protocols.
3. Without defining what your RTO/RPO expectations are its impossible to require the solution to perform them.

 

[Johannes S]

The Thinkcentre chassis are small, so maybe I am not too keen on installing a separate NIC, what would you do?

Don't cluster them but use the DatacenterManager for migrating vms between the nodes and pve-zsnc for replicating via zfs. You will need to start them manually instead of HA though. Alternatively buy two USB-NICs to have a dedicated cluster network and your existing network as fallback. . And you will really need a third node or a qdevice. The qdevice is not needed if you have at least three nodes and not recommended if you have an odd number of nodes.
My guess is that your traffic was low enough that you didn't run into problems until your network resources were used more. Corosync really needs low latency and this

See the manual: https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network for network requirements and
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support for qdevice

Such a setup ( usb-nics ) won't be ideal but better than nothing. If you can't satisfy either of them: Don't cluster.

The fix "ethtool -K eno2 tso off gso off gro off", do you agree that it is a good thing to do?

This workaround seems to be the only optltion to get some kind of intel nics with the e1000 errors to work. But even with different nics you need to satisfy the cluster requirements, they have nothing to do with the bug. The bullshit-as-a-service AI just put them together because many people have issues related to both ( the e1000 bugs and insufficient cluster setups).

 

[Johannes S]

You dont need a qdevice for a 3 node cluster; it will just serve as a 4th vote that doesnt have any consequence since its not a tiebreaker anyway.

In fact the pve manual recommends not to have a qdevice if you have at least three nodes and an odd number of nodes due to the way corosync handles them:

Supported Setups
We support QDevices for clusters with an even number of nodes and recommend it for 2 node clusters, if they should provide higher availability. For clusters with an odd node count, we currently discourage the use of QDevices. The reason for this is the difference in the votes which the QDevice provides for each cluster type. Even numbered clusters get a single additional vote, which only increases availability, because if the QDevice itself fails, you are in the same position as with no QDevice at all.

On the other hand, with an odd numbered cluster size, the QDevice provides (N-1) votes — where N corresponds to the cluster node count. This alternative behavior makes sense; if it had only one additional vote, the cluster could get into a split-brain situation. This algorithm allows for all nodes but one (and naturally the QDevice itself) to fail. However, there are two drawbacks to this:

If the QNet daemon itself fails, no other node may fail or the cluster immediately loses quorum. For example, in a cluster with 15 nodes, 7 could fail before the cluster becomes inquorate. But, if a QDevice is configured here and it itself fails, no single node of the 15 may fail. The QDevice acts almost as a single point of failure in this case.

The fact that all but one node plus QDevice may fail sounds promising at first, but this may result in a mass recovery of HA services, which could overload the single remaining node. Furthermore, a Ceph server will stop providing services if only ((N-1)/2) nodes or less remain online.

If you understand the drawbacks and implications, you can decide yourself if you want to use this technology in an odd numbered cluster setup. https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

 

[J-Rod]

My opinion is that if you really want a properly running cluster with HA enabled, then SFF/USFF machines are mostly a dead end. Either find mini-towers so you can have dual 10GbE, a boot mirror and a some drives for Ceph, or forgo HA and just rely on ZFS replication.

 

[Johannes S]

HA with ZFS (let's call it Pseudo-HA) definitely works if you can live with a minimal dataloss (the default is 15 minutes, can be reduced one minute). This doesn't change the need for a dedicated corosync network though which is deficult to achieve with SFF/USFF. pve-zsync wouldn't need a cluster though and won't have HA, do you mean this?

 

[UdoB]

the default is 15 minutes, can be reduced one minute

Actually I usually set the replication interval to 24h, once per night. For both my $dayjob's cluster and my homelab, and for most - but not all - VMs.

Remember: data loss from "too-long-an-interval" will only happen when a node crashes. It is not relevant for normal work with migration, be it live or not or if a node is going to shutdown for maintenance.

 

[J-Rod]

For HA, you want redundant and robust connectivity, to avoid undesirable fencing/watchdog/reboot incidents, and then there is the ZFS replication "data loss", as mentioned, which is a bit of a can of worms in of itself:

You either have to give up the data that was written since the last sync, or skip reversing the replication and fallback until you can mount the virtual disks of the failed node and copy the data out and possibly merge it with the running VM. Which is going to be some work.

Or, you could skip automatically starting the VM on another node until you fix the failed node or make the decision that it's a goner and lose the data and forgo the recovery. But not automatically starting the VM on another node means those VM's services are down until you manually intervene.

There could be other permutations, but those are typically the choices you'll have.

Ceph has its own challenges, but it avoids those HA ZFS replication conundrums.


Every environment is different, but a 24 hour replication schedule seems like an odd choice to me. My schedule is between 1-9 minutes depending on criticality, and I use rate limits and disable guest freeze/thaw where necessary to avoid impact (actually not needed in my 10G, all-flash home lab cluster). PBS backups are less than 30 minutes for running VMs and every few hours for powered off VMs. Anything much less than that seems like giving up a lot of advantages provided by these features. Also, the data needing to be synchronized is likely still in disk cache.


Side note: I consider Ceph to be a single point of failure logically, even with RBD mirroring. So have aggressive backups at the VM level. And test your backups regularly.