Backup freezes eternally

[Nam Trần]

After some upgrades to pve-manager 5.1.36, the backup function always ends in eternal freeze. For example:

INFO: starting new backup job: vzdump 603 --mode snapshot --compress lzo --dumpdir /tmp/ --storage backup
INFO: Starting Backup of VM 603 (qemu)
INFO: status = running
INFO: update VM 603: -lock backup
INFO: VM Name: DCRepBT2
INFO: include disk 'virtio0' 'local-lvm:vm-603-disk-1' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating archive '/mnt/backup/dump/vzdump-qemu-603-2017_10_26-22_07_12.vma.lzo'

Is there any problem with the vzdump command and arguments? I could not be able to backup a very important VM as stated above.

Any help? Thanks!

 

[Nam Trần]

I let the vzdump run during night and found out that it had finished in the morning. It seems that vzdump freezes for hours before executing. The same problem happen among all Proxmox Servers of two different clusters. It used to be very fast!

 

[LnxBil]

A full log would be good. Could you provide one?

I notice also some lags due to freezing timeouts on some machines, yet I did not find any solution yet. Sometimes it only tooks quite some time, sometimes it completely fails and I need to reset the VM because the guest os looses its hard disks due to the freeze. This is an excerpt of my log:

Oct 13 18:46:07 INFO: creating archive '/mnt/pve/backup-zfs/dump/vzdump-qemu-1018-2017_10_13-18_46_03.vma.lzo'
Oct 13 19:46:07 ERROR: VM 1018 qmp command 'guest-fsfreeze-freeze' failed - got timeout
Oct 13 19:46:17 ERROR: VM 1018 qmp command 'guest-fsfreeze-thaw' failed - unable to connect to VM 1018 qga socket - timeout after 101 retries
Oct 13 19:46:17 INFO: started backup task 'a8c65f8c-7dea-4655-9544-514852fde091'

So, a timeout occurs after one hour according to the timestamp. The guest does nothing disk related. It is a router, yet the hanging machines are not deterministic. The number of occurrences get less and less after each PVE update, yet once in every 100 backups or so I see this problem.

 

[Nam Trần]

A full log would be good. Could you provide one?

I notice also some lags due to freezing timeouts on some machines, yet I did not find any solution yet. Sometimes it only tooks quite some time, sometimes it completely fails and I need to reset the VM because the guest os looses its hard disks due to the freeze. This is an excerpt of my log:

Oct 13 18:46:07 INFO: creating archive '/mnt/pve/backup-zfs/dump/vzdump-qemu-1018-2017_10_13-18_46_03.vma.lzo'
Oct 13 19:46:07 ERROR: VM 1018 qmp command 'guest-fsfreeze-freeze' failed - got timeout
Oct 13 19:46:17 ERROR: VM 1018 qmp command 'guest-fsfreeze-thaw' failed - unable to connect to VM 1018 qga socket - timeout after 101 retries
Oct 13 19:46:17 INFO: started backup task 'a8c65f8c-7dea-4655-9544-514852fde091'

So, a timeout occurs after one hour according to the timestamp. The guest does nothing disk related. It is a router, yet the hanging machines are not deterministic. The number of occurrences get less and less after each PVE update, yet once in every 100 backups or so I see this problem.

Thanks for your sharings. I have found out that the broken DNS service has caused such weird problem. After fixing the DNS service, everything works just fine as expected.

 

[Andrew Holybee]

Thanks for your sharings. I have found out that the broken DNS service has caused such weird problem. After fixing the DNS service, everything works just fine as expected.

What was broken about with your DNS service I have this issue where it stalls for 1 hour every night but only on our DNS server VM