[SOLVED] Is a vzdump backupjob on a slow nfs-server able to trigger watchdog

[jsterr]

Hello Proxmox Forum,

Is a slow vzdump backupjob able to trigger watchdog? I had some unknown reboot today at 07:38 on a customers system:

- 07:35:34 vzdump-Start (Ziel: langsames NFS)
- 07:38:24 pve-ha-crm: loop took too long (33s)
- 07:38:36 pve-ha-lrm: loop took too long (79s) + cfs-lock 'file-jobs_cfg' / 'file-replication_cfg' got lock request timeout
- 07:38:36 watchdog did not stop -> Reset

Jul 20 07:38:24 pve2 pve-ha-crm[1875]: loop took too long (33 seconds)
Jul 20 07:38:31 pve2 pveproxy[2675959]: proxy detected vanished client connection
Jul 20 07:38:32 pve2 pveproxy[2657524]: proxy detected vanished client connection
Jul 20 07:38:36 pve2 pve-ha-lrm[1909]: loop took too long (79 seconds)
Jul 20 07:38:36 pve2 pvescheduler[3143029]: jobs: cfs-lock 'file-jobs_cfg' error: got lock request timeout
Jul 20 07:38:36 pve2 pvescheduler[3143028]: replication: cfs-lock 'file-replication_cfg' error: got lock request timeout
Jul 20 07:38:36 pve2 kernel: watchdog: watchdog0: watchdog did not stop!

I cant find any other issues, corosync latency is fine, links are fine, hardware is fine. But might the backupjob be the issue? Is a not-responding or slow nfs server a problem regarding host-stability?

proxmox-ve: 9.2.0 (running kernel: 7.0.6-2-pve)
pve-manager: 9.2.3 (running version: 9.2.3/d0fde103346cf89a)

 

[Francois Baillargeon]

It may, but also, is corosync colocated on the same link? that's also something I have often seen as issues.

 

[jsterr]

It may, but also, is corosync colocated on the same link? that's also something I have often seen as issues.

Nope corosync has its own two links with their own two physical ports not used for backup.

 

[Francois Baillargeon]

I think a slow nfs can absolutely be a problem in that case, how fast is it?

 

[Sim0n]

Maybe the the IO-Wait gets so high that corosync drops out or the Watchdog gets triggered?

You could try to set a bandwidth limit in the backup job or a fleecing storage?

 

[jsterr]

The customer also asked the proxmox enterprise support and proxmox also thinks that stalled i/o caused the issue. The solution could be using a fleecing storage. Thanks for your (also correct ) replies.