backup failed randomly

[fabienfs]

Hello,

I have a Proxmox Server and a Proxmox Backup Server.
Every night I make a backup of all my VMs.
Unfortunately, almost every night, I have a VM that can't make its backup.
Every night is a different VM backup failed (please see screenshot), but always with the same error message : "VM xxx qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed out".
If I then try to manually backup that VM, it works. It's just that when he backs up all the VMs at night, there's almost automatically one that he can't do. However the internet connection is good.
And sometimes, he still manages to make nights without problem.

What could this be due to?

thanks

 

[Moayad]

Hello,

Always the same VM that failed with backup?

If I then try to manually backup that VM, it works.

I would check if there is a high I/O during the backup process, and try to set bandwidth limits for the backup to narrow down the issue. If the issue occurs again, please provide us with the task log of the failed backup and the Syslog during the backup time.

 

[fabienfs]

Always the same VM that failed with backup?

No it's every time a different

and try to set bandwidth limits for the backup to narrow down the issue.

Can we define in Proxmox the bandwidth dedicated to backups?

Here is the log :

INFO: 100% (39.4 GiB of 39.4 GiB) in 2m 56s, read: 2.0 GiB/s, write: 0 B/s
INFO: backup is sparse: 23.02 GiB (58%) total zero data
INFO: backup was done incrementally, reused 73.65 GiB (81%)
INFO: transferred 39.43 GiB in 178 seconds (226.8 MiB/s)
INFO: Finished Backup of VM 150 (00:03:02)
INFO: Backup finished at 2023-02-03 04:42:43
storing login ticket failed: $XDG_RUNTIME_DIR must be set
INFO: Starting Backup of VM 153 (qemu)
INFO: Backup started at 2023-02-03 04:42:43
INFO: status = running
INFO: VM Name: test
INFO: include disk 'scsi0' 'dataz:vm-153-disk-0' 37G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/153/2023-02-03T03:42:43Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 153 qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed out
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 153 failed - VM 153 qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed out
INFO: Failed at 2023-02-03 04:44:47
INFO: Starting Backup of VM 159 (qemu)
INFO: Backup started at 2023-02-03 04:44:47
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: xxxxxx
INFO: include disk 'virtio0' 'dataz:vm-159-disk-0' 32G
INFO: creating Proxmox Backup Server archive 'vm/159/2023-02-03T03:44:47Z'
INFO: starting kvm to execute backup task
INFO: enabling encryption
INFO: started backup task 'f0f9d1dc-900f-439c-982d-00b5c332ef61'
INFO: virtio0: dirty-bitmap status: created new
INFO:   8% (2.6 GiB of 32.0 GiB) in 3s, read: 890.7 MiB/s, write: 0 B/s
INFO:  15% (5.0 GiB of 32.0 GiB) in 6s, read: 805.3 MiB/s, write: 0 B/s

Where can I find the syslog?

thanks

 

[Moayad]

Can we define in Proxmox the bandwidth dedicated to backups?

You can set the bandwidth limits of the backup from the `vzdump.conf` file you will find the file in Proxmox VE `/etc/vzdump.conf` [0]

[0] https://pve.proxmox.com/pve-docs/chapter-vzdump.html#vzdump_configuration