Intel E810 / ICE / IRDMA HMC Error causing Corosync link loss and node reboots – Proxmox VE 7.3

[pvpaulo]

Hello,

I have a 6-node Proxmox VE cluster running HA and Ceph, using Intel E810-XXV network adapters.

I am experiencing recurring node reboots on some of the nodes, and based on the logs, I suspect the issue may be related to the Intel E810 / ICE / IRDMA stack.

The environment is currently running Proxmox VE 7.3, with:

Kernel: 5.15.74-1-pve
NIC: Intel E810-XXV
Driver: ice
Firmware: 2.50 0x800077a8 1.2960.0
On the affected nodes, I repeatedly see:

ice ... irdmaX: ICE OICR event notification
ice ... irdmaX: HMC Error
ice ... irdmaX: Requesting a reset
After that, Corosync loses communication, followed by Token timeout, loss of quorum, and eventually a node reboot.

Interestingly, I found the same error on PVE04, which has been stable for months:

ice 0000:0f:00.0 irdma0: ICE OICR event notification: oicr = 0x04000003
ice 0000:0f:00.0 irdma0: HMC Error
ice 0000:0f:00.0 irdma0: Requesting a reset
Immediately after the reset, Corosync also lost its links temporarily:

09:00:54 - KNET link is down
09:01:01 - KNET link is up
09:01:02 - Completed service synchronization
So it appears that the E810 reset is actually causing a temporary interruption in cluster communication.

My question​

I am considering upgrading the environment to Proxmox VE 8.4, but I would like to get some guidance before proceeding.

I am currently investigating PVE04, and I have already migrated all VMs from this node to the other nodes.

However, if I upgrade PVE04 to Proxmox VE 8.4, I would likely need to proceed with upgrading the rest of the cluster as well.

My concern is performing a full cluster upgrade and then discovering that the root cause is actually related to the E810 firmware, ICE/IRDMA, or another hardware/driver issue, meaning the upgrade would not solve the problem.

Is there any known workaround or mitigation that I can test while still running Proxmox VE 7.3?

For example:

• Disabling irdma, if RDMA is not required;
• Updating only the kernel/driver;
• Updating the E810 firmware;
• Changing any ice/irdma configuration;
• Using a specific kernel/module parameter;
• Any other way to prevent an E810 reset from causing Corosync/quorum loss.

Has anyone experienced the following error with Intel E810 on Proxmox?

ICE OICR event notification: oicr = 0x04000003
HMC Error
Requesting a reset
If so, is there a known workaround or recommended mitigation that can be applied before upgrading the Proxmox cluster?

Any advice would be greatly appreciated.

Thank you.

 

[leesteken]

In order to update to PVE 8.4 (which is, or is about to be, out of support) you need to update to the latest 7.4.x (see the guide), which you have not done for years. Maybe start a new (supported) 9.2 cluster and see if you still have this issue? Even if Proxmox staff wants to investigate and fix this issue, they probably won't do it on 7.3 (or 7.4) except maybe the latest 9.

If this is a new problem then the questions is: what changed? Debian did not release updates, Proxmox did not release updates. Maybe your hardware is breaking down? Or it is a local issue with your network switches or something. That cannot be fixed in software and needs to be investigated on your end. Maybe there is a local Proxmox partner?

 

[Tmanok]

Hi PvPaulo,

Leesteken is right about the hypervisor being too far out of date. There have been some issues online about this same driver issue back when I was more active, some people claimed that the solution was upgrading to the 6 kernel from 5.15, other people claimed it didn't help. But anyway, I think where Leesteken might be incorrect is the assumption that static software indicates a hardware fault.

It might have been possible that this was hardware until this line came up:

irdma: HMC Error
irdma: Requesting a reset

A switch problem could trigger an edge case in the driver, certainly. But when the Linux driver announces that its RDMA component has encountered an internal error and is explicitly requesting a NIC reset before Corosync dies, I would investigate that first. And the easiest way to resolve it if it's a recurring issue and you don't use RDMA, is simply to turn it off:

echo "blacklist irdma" > /etc/modprobe.d/blacklist-irdma.conf
update-initramfs -u
reboot

Obviously, please take great care in moving your guests to other nodes and or schedule downtime maintenance.
Thanks, and good luck with this issue! Please consider upgrading to the latest PVE sooner rather than later!


Tmanok
P.S.

You can check that irdma is disabled with: lsmod | grep irdma

 

[pvpaulo]

Olá PvPaulo,

Leesteken está certo sobre o hipervisor estar muito desatualizado. Houve alguns problemas online sobre esse mesmo problema de driver quando eu era mais ativo; algumas pessoas afirmavam que a solução era atualizar do kernel 5.15 para o 6, outras afirmavam que não ajudava. De qualquer forma, acho que onde Leesteken pode estar errado é na suposição de que software estático indica uma falha de hardware.

Até então, era possível que se tratasse de um problema de hardware, até que essa linha de código apareceu:
[CÓDIGO]irdma: Erro HMC
irdma: Solicitando uma reinicialização[/CODE]

Um problema com o switch certamente poderia desencadear uma situação atípica no driver. Mas quando o driver do Linux anuncia que seu componente RDMA encontrou um erro interno e solicita explicitamente uma reinicialização da placa de rede antes que o Corosync seja encerrado, eu investigaria isso primeiro. E a maneira mais fácil de resolver o problema, caso seja recorrente e você não utilize RDMA, é simplesmente desativá-lo.

[CÓDIGO]echo "blacklist irdma" > /etc/modprobe.d/blacklist-irdma.conf
atualizar-initramfs -u
reiniciar[/CODE]

Obviamente, tenha muito cuidado ao transferir seus convidados para outros nós e/ou ao agendar manutenções programadas.
Obrigado e boa sorte com esse problema! Por favor, considere atualizar para a versão mais recente do PvE o quanto antes!


Tmanok
PS

Você pode verificar se o irdma está desativado com:lsmod | grep irdma

Obrigado pela
resposta. vou testar

 

[longer]

I think the upgrade is necessary in any case since PVE7/Debian 11 is EOL.

I found one thread related to Intel E810 NICs, where it was observed it can be related to causing memory leaks:

Post in thread 'Slow memory leak in 6.8.12-13-pve'

Sep 9, 2025

EDIT: This post has been edited with the newest information we currently have.

From multiple reports over the last weeks, we can confirm that the issue is caused by:

1. Intel ICE drivers used by multiple Intel NICs, at least in combination with the Intel E810 NICs, possibly also with other NICs that use the ICE driver.
2. Jumbo Frames / MTU 9000 with a lot of network traffic, which is often the case when using Ceph (but Ceph is unrelated, except that it accelerates the memory leaks due to the amount of network traffic).
3. Either of the following affectedkernel versions...

• l.leahu-vladucu

• 

While its not the same issue you are experiencing, by the upgrade you make sure you avoid such issue. I would storngly recommend to upgrade to the latest PVE9.