Backup snapshot of VM: "ERROR: got timeout"

[cesarpk]

Hi ti all

Anyone that can help me

I have PVE 2.3 in cluster, a Node make backup to other Node through of NFS.
The Servers are DELL

This is the message of Backup:
INFO: starting new backup job: vzdump 105 107 --quiet 1 --mailto tecnico@paraguaytrading.com.py,cesar@paraguaytrading.com.py --mode snapshot --compress gzip --storage Backups_on_kvm3 --node kvm4
INFO: Starting Backup of VM 105 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backups_on_kvm3/dump/vzdump-qemu-105-2013_09_05-00_00_02.vma.gz'
INFO: started backup task 'c4c711e6-ad8c-4bb5-9f19-47bb237377d6'
INFO: status: 7% (1199898624/16106127360), sparse 7% (1178247168), duration 3, 399/7 MB/s
...
etc
...
INFO: status: 100% (16106127360/16106127360), sparse 12% (2051940352), duration 777, 20/20 MB/s
INFO: transferred 16106 MB in 777 seconds (20 MB/s)
INFO: archive file size: 2.41GB
INFO: delete old backup '/mnt/pve/Backups_on_kvm3/dump/vzdump-qemu-105-2013_09_03-00_00_02.vma.gz'
INFO: Finished Backup of VM 105 (00:13:01)
INFO: Starting Backup of VM 107 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backups_on_kvm3/dump/vzdump-
qemu-107-2013_09_05-00_13_03.vma.gz'
ERROR: got timeout
INFO: aborting backup job
ERROR: Backup of VM 107 failed - got timeout
INFO: Backup job finished with errors
TASK ERROR: job errors <----- at 00:13:08 am

The config of VM 107 (that have the problem of backup):
boot: c
bootdisk: virtio0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 36864
name: Win2008R2-SQL
net0: virtio=72:60:0D:E5:29:65,bridge=vmbr0
net1: virtio=DE:B36:AB:B5:6B,bridge=vmbr1
ostype: win7
sockets: 1
virtio0: local:107/vm-107-disk-1.raw,size=350G
virtio1: local:107/vm-107-disk-2.raw,size=32G
virtio2: local:107/vm-107-disk-3.raw,size=150G
virtio3: local:107/vm-107-disk-4.qcow2,size=50G

Maybe much space on virtual disks for create the backup image?

I will be very grateful to anyone who can help me

Cesar

 

[dietmar]

Can you trigger that if you only backup VM 107 manually? Does it work if you backup to local storage instead of NFS?

 

[dietmar]

Oh, you run an old version. It would be great if you test with latest 3.1 code instead.

 

[cesarpk]

Oh, you run an old version. It would be great if you test with latest 3.1 code instead.

Hi Dietmar and thanks for your attention

For now i can't upgrade PVE, but for many months these backups works very well, and of a day to other i get the error messages without make any structural change

Due that the PVE Nodes are never turned off or restart it for very months, may be that the PVE Nodes needs to be restarted it for get stability?

Any idea for solved without upgrade PVE?

Best regards
Cesar

 

[dietmar]

It is probably not a big problem if it only happens sometimes?

 

[cesarpk]

It is probably not a big problem if it only happens sometimes?

Hi Dietmar

Yesterday i only restarted the PVE Host and now see on the GUI that the backups are ok

My conclusion:
PVE needs more stability

Thanks for your attention

Ahh!, please tell me step by step how i add "SOLVED PARTIALLY" into the text of Thread, for if this help to other users

Best regards
Cesar

 

[Sarvesh Goburdhun]

Hello Proxmox Team,

We currently have the same backup timeout issue with 2 VMs to be backuped on a remote NFS:

2nd VM info:

qm config 116
bootdisk: scsi0
cores: 1
cpu: host
ide2: none,media=cdrom
memory: 2048
name: vm1
net0: virtio=52:54:00:00:a4:22,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: ceph-host1:vm-116-disk-0,cache=writeback,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=f764d04a-3733-4d5d-98ff-a81c5338c786
sockets: 1

* Backup Issue:

2019-11-04 01:13:14 INFO: Starting Backup of VM 116 (qemu)
2019-11-04 01:13:14 INFO: status = running
2019-11-04 01:13:15 INFO: update VM 116: -lock backup
2019-11-04 01:13:15 INFO: VM Name: vm1
2019-11-04 01:13:15 INFO: include disk 'scsi0' 'ceph-host1:vm-116-disk-0' 32G
2019-11-04 01:13:15 INFO: backup mode: snapshot
2019-11-04 01:13:15 INFO: ionice priority: 7
2019-11-04 01:13:15 INFO: creating archive '/mnt/pve/backupNfs/dump/vzdump-qemu-116-2019_11_04-01_13_14.vma.lzo'
2019-11-04 01:13:18 ERROR: got timeout
2019-11-04 01:13:18 INFO: aborting backup job
2019-11-04 01:23:18 ERROR: VM 116 qmp command 'backup-cancel' failed - got timeout
2019-11-04 01:23:19 ERROR: Backup of VM 116 failed - got timeout

2nd VM info:

 qm config 118
bootdisk: scsi0
cores: 12
cpu: host
ide2: none,media=cdrom
memory: 16384
name: vm2
net0: virtio=1E:82:BB:C1:B6:B9,bridge=vmbr1,firewall=1
net1: virtio=00:50:56:01:98:58,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: ceph-host1:vm-118-disk-0,cache=writeback,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=bbbb7982-b19a-4ac1-923c-ed7046563d20
sockets: 1
vmgenid: e95469c7-c7ce-4426-b5e1-1391b81392ec

Backup Issue:

2019-11-04 01:23:19 INFO: Starting Backup of VM 118 (qemu)
2019-11-04 01:23:19 INFO: status = running
2019-11-04 01:23:19 INFO: update VM 118: -lock backup
2019-11-04 01:23:19 INFO: VM Name: vm2
2019-11-04 01:23:19 INFO: include disk 'scsi0' 'ceph-host1:vm-118-disk-0' 16G
2019-11-04 01:23:19 INFO: backup mode: snapshot
2019-11-04 01:23:19 INFO: ionice priority: 7
2019-11-04 01:23:19 INFO: creating archive '/mnt/pve/backupNfs/dump/vzdump-qemu-118-2019_11_04-01_23_19.vma.lzo'
2019-11-04 01:23:22 ERROR: got timeout
2019-11-04 01:23:22 INFO: aborting backup job
2019-11-04 01:33:22 ERROR: VM 118 qmp command 'backup-cancel' failed - got timeout
2019-11-04 01:33:23 ERROR: Backup of VM 118 failed - got timeout

1. Proxmox Version :

pve-manager/6.0-9/508dcee0 (running kernel: 5.0.21-2-pve)

2. A Proxmox cluster of 3 Hosts with Ceph.
3. The vzdump (snapshot) is done via cron job:

0 0 * * *           root vzdump --mailto xxx@xxx.com --all 1 --storage backupNfs --mode snapshot --quiet 1 --mailnotification always --compress lzo

When the vzdump is run manually on local storage or nfs, same backup timeout error message is obtained.

From our understanding, there might be a timeout of 3 seconds before initiating a backup in the source code. Can this timeout value be increased or any other workaround without stopping the VM to do a backup?

From the proxmox forum, there was an article indicating that there must at least be a successful backup before running the automatic vzdump (when maxfiles is set to 3).

nfs: backupNfs
    export /data/backups/new-host1-cluster
    path /mnt/pve/backupNfs
    server xxxx.xxx.com
    content backup
    maxfiles 3

I am not sure how far this statement is correct. Can you please advise?

Kindly waiting for your return.