Backup job finished with errors

[alex_31_ntl]

Hello everyone,

For several days now, I've been experiencing problems backing up my virtual machines (VMs) on my Proxmox Backup Server (PBS).

To put this in context, I've deployed a dozen Proxmox Virtual Environment (PVE) nodes in my company, on which around 3 VMs are installed. I've scheduled daily backups and made sure that the backups of the 3 VMs don't run simultaneously (with an hour's interval between each PVE VM).

However, I encounter a random error affecting a few VMs every day. It's never the same VMs that cause the problem, and the next day, the backup runs correctly while another VM fails. Here are the logs I've observed on the pve :

and the pbs :



For your information, the QEMU guest agent is activated on VMs, I've read on some forums that this can cause problems during backups...
Here is l/O during the backup phase on VMs, I have this peak for each server. :



Could the problem come from the fact that the backups of each PVE are done at the same time?

exemple :







Do you have any idea what might be causing the problem?

I'm getting desperate...
Alexis

 

[Chris]

Hi,
what Proxmox VE version are you using, please post the output of pveversion -v. The error indicates that the client disconnected from the backup server without finishing the backup. Do you see any errors around that time in the systemd journal? journalctl --since <DATETIME> --until <DATETIME> will give you a paginated view of the journal for the selected timespan.

 

[alex_31_ntl]

Hi Christ,
Here is the proxmox version for each pve :

For the first two pve
pveversion -v
proxmox-ve: 8.0.1 (running kernel: 6.2.16-5-pve)

For the third pve
proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)

And the result of the systemd journal :

journalctl --since "2024-02-28 00:00:00" --until "2024-02-28 00:15:00"



same kind of result for the second and third pve :

 

[Chris]

Hi Christ,
Here is the proxmox version for each pve :

For the first two pve
pveversion -v
proxmox-ve: 8.0.1 (running kernel: 6.2.16-5-pve)

For the third pve
proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)

And the result of the systemd journal :

journalctl --since "2024-02-28 00:00:00" --until "2024-02-28 00:15:00"

View attachment 63897

same kind of result for the second and third pve :

View attachment 63898

Seems like the VMs are not responsive anymore. Are they still operating normally after the backup? Please check if the process is stuck in an uninterruptible sleep state via ps auwxf or htop . Could be that your backup server is not able to handle the load? Please check also the systemd journal on the PBS side.

On a side note: You do not need to create an individual backup schedule for each VM, if you create one, the backups for that node are performed sequentially already, not concurrently.

Edit: Please also have a look at this thread and perform the suggestions/ouputs given by @fiona https://forum.proxmox.com/threads/e...-action-on-proxmox-qmp-command-failed.139892/