Backup freezes eternally

Nam Trần

Active Member
Jun 29, 2016
31
1
28
36
After some upgrades to pve-manager 5.1.36, the backup function always ends in eternal freeze. For example:

Code:
INFO: starting new backup job: vzdump 603 --mode snapshot --compress lzo --dumpdir /tmp/ --storage backup
INFO: Starting Backup of VM 603 (qemu)
INFO: status = running
INFO: update VM 603: -lock backup
INFO: VM Name: DCRepBT2
INFO: include disk 'virtio0' 'local-lvm:vm-603-disk-1' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating archive '/mnt/backup/dump/vzdump-qemu-603-2017_10_26-22_07_12.vma.lzo'

Is there any problem with the vzdump command and arguments? I could not be able to backup a very important VM as stated above.

Any help? Thanks!
 

Nam Trần

Active Member
Jun 29, 2016
31
1
28
36
I let the vzdump run during night and found out that it had finished in the morning. It seems that vzdump freezes for hours before executing. The same problem happen among all Proxmox Servers of two different clusters. It used to be very fast!
 

LnxBil

Famous Member
Feb 21, 2015
6,788
900
173
Saarland, Germany
A full log would be good. Could you provide one?

I notice also some lags due to freezing timeouts on some machines, yet I did not find any solution yet. Sometimes it only tooks quite some time, sometimes it completely fails and I need to reset the VM because the guest os looses its hard disks due to the freeze. This is an excerpt of my log:

Code:
Oct 13 18:46:07 INFO: creating archive '/mnt/pve/backup-zfs/dump/vzdump-qemu-1018-2017_10_13-18_46_03.vma.lzo'
Oct 13 19:46:07 ERROR: VM 1018 qmp command 'guest-fsfreeze-freeze' failed - got timeout
Oct 13 19:46:17 ERROR: VM 1018 qmp command 'guest-fsfreeze-thaw' failed - unable to connect to VM 1018 qga socket - timeout after 101 retries
Oct 13 19:46:17 INFO: started backup task 'a8c65f8c-7dea-4655-9544-514852fde091'

So, a timeout occurs after one hour according to the timestamp. The guest does nothing disk related. It is a router, yet the hanging machines are not deterministic. The number of occurrences get less and less after each PVE update, yet once in every 100 backups or so I see this problem.
 
  • Like
Reactions: Nam Trần

Nam Trần

Active Member
Jun 29, 2016
31
1
28
36
A full log would be good. Could you provide one?

I notice also some lags due to freezing timeouts on some machines, yet I did not find any solution yet. Sometimes it only tooks quite some time, sometimes it completely fails and I need to reset the VM because the guest os looses its hard disks due to the freeze. This is an excerpt of my log:

Code:
Oct 13 18:46:07 INFO: creating archive '/mnt/pve/backup-zfs/dump/vzdump-qemu-1018-2017_10_13-18_46_03.vma.lzo'
Oct 13 19:46:07 ERROR: VM 1018 qmp command 'guest-fsfreeze-freeze' failed - got timeout
Oct 13 19:46:17 ERROR: VM 1018 qmp command 'guest-fsfreeze-thaw' failed - unable to connect to VM 1018 qga socket - timeout after 101 retries
Oct 13 19:46:17 INFO: started backup task 'a8c65f8c-7dea-4655-9544-514852fde091'

So, a timeout occurs after one hour according to the timestamp. The guest does nothing disk related. It is a router, yet the hanging machines are not deterministic. The number of occurrences get less and less after each PVE update, yet once in every 100 backups or so I see this problem.

Thanks for your sharings. I have found out that the broken DNS service has caused such weird problem. After fixing the DNS service, everything works just fine as expected.
 

Andrew Holybee

Active Member
Mar 27, 2017
52
0
26
42
Thanks for your sharings. I have found out that the broken DNS service has caused such weird problem. After fixing the DNS service, everything works just fine as expected.

What was broken about with your DNS service I have this issue where it stalls for 1 hour every night but only on our DNS server VM
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!