My k3s cluster dies when PBS runs backups

[strugglingbackupuser]

I am a newbie.

I have a toy k3s cluster running on 3 proxmox hosts (Minisforum MS-A2 - so fairly beefy for home lab). Each node has 2 Samsung 990 Pro SSDs. The issue I am having is that whenever PBS runs it's backups k3s experiences all sorts of issues.

The issues I get are:

• k3s logs show context deadline exceeded / api timeout around backup windows
• etcd reports “apply entries took too long” and occasional leader re-elections
• k3s API server briefly becomes unresponsive
• Longhorn volumes enter Degraded state and sometimes rebuild replicas

I just feel that given the spec of my hardware and the very low load of what is running on it, that PBS backups shouldn't cause these pod restarts and k3s node restarts. I think there must be something fundamentally wrong somewhere.

The backups are configures as snapshots and the backup destination is a NAS drive.

 

[bl1mp]

Hi,

the issues indicate, that your controlplane is stalling during the backup.
Etcd is very sensitive to disk write latency, which might be an issue in your setup.
But maybe there are further configuration incompatibilities.

Do you use Backup Fleecing? (https://pve.proxmox.com/wiki/Backup_and_Restore#_vm_backup_fleecing)
This could minimize the impact.

BR, Lucas

 

[strugglingbackupuser]

Hi,

the issues indicate, that your controlplane is stalling during the backup.
Etcd is very sensitive to disk write latency, which might be an issue in your setup.
But maybe there are further configuration incompatibilities.

Do you use Backup Fleecing? (https://pve.proxmox.com/wiki/Backup_and_Restore#_vm_backup_fleecing)
This could minimize the impact.

BR, Lucas

Thanks for replying. I’ll look into the fleecing option.

I guess I just feel like everyone who runs k3s and uses PBS would run into these issues if I am. I am on decent hardware on an empty cluster. I just feel like it would be more common if it was a case of tweaking knobs to get it to work.