All backups work except one VM always with error "http request timed out"

[timdonovan]

Hi,

I have a remote PBS and have it configured so PVE runs the backups. All other VM's, split across two PVE nodes, get backuped fine to this target. The one that won't back up reports:

INFO: starting new backup job: vzdump 201 --mode snapshot --node proxmox1 --remove 0 --storage pbs-remote
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2021-05-26 09:54:03
INFO: status = running
INFO: VM Name: pihole-1
INFO: include disk 'scsi0' 'local-zfs:vm-201-disk-0' 8G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/201/2021-05-26T08:54:03Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 201 qmp command 'backup' failed - backup connect failed: command error: http request timed out
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 201 failed - VM 201 qmp command 'backup' failed - backup connect failed: command error: http request timed out
INFO: Failed at 2021-05-26 09:54:24
INFO: Backup job finished with errors
TASK ERROR: job errors

I cannot see anything wrong with this VM that would stop it being backed up, and the error reported almost looks like vzump suddenly cannot connect to the PBS...but then works fine 30 seconds later on another VM.

root@proxmox1:~# qm status 201 --verbose|grep -A5 proxmox-support
proxmox-support:
        pbs-dirty-bitmap: 1
        pbs-dirty-bitmap-migration: 1
        pbs-dirty-bitmap-savevm: 1
        pbs-library-version: 1.0.3 (8de935110ed4cab743f6c9437357057f9f9f08ea)
        pbs-masterkey: 1

Any help appreciated, thanks!

 

[dietmar]

Seems there is a network problem during backup, maybe to much load on the network or on the backup server:

>ERROR: VM 201 qmp command 'backup' failed - backup connect failed: command error: http request timed out

 

[timdonovan]

Thanks @dietmar but as mentioned its SPECIFICALLY only this one VM. If I immeditely go backup another VM to the same PBS target (i.e. with the same network conditions), no issues at all. I'm backing up 7 other VM's in total and it's a repeatable consistent error only on this one VM.

Therefore it is not a network issue (despite what the error says).

 

[dcsapak]

INFO: VM Name: pihole-1

is that vm responsible for resolving the dns name of connecting to the pbs? if yes this will not work since the vm is paused for the backup start...
edit: resolving the dns would probably produce a different error

 

[timdonovan]

Hah..omg.

https://isitdns.com/

You're right. This VM is the primary DNS. I have a failover backup DNS and the switchover happens automatically in < 10 seconds, but the snapshot and backup obviously pauses the VM and attempts to connect in a much faster time than that!

You're a genius @dcsapak thank you

To your point though, it's just DNS so maybe the error vzdump is reporting needs tweaking slightly!

 

[timdonovan]

I changed the storage server option to be IP based rather than hostname it fixed it. Cheers!