vm hang up on backup and sporadic

[informant]

Hi all,

we have on one our nodes a problem. VMs hang up sporadic and on backups (sporadic). No error codes are listen. no log entrys are listen. cpu load goes down to 0%.

On backup:
backup starts automatical
vm was locked on backup
INFO: started backup task '1fe4ca08-c704-497d-9bb6-8d0b7630baa2'
INFO: status: 0% (0/107374182400), sparse 0% (0), duration 3, 0/0 MB/s
vm hang up on backup and is not more reachable
here we must unlock vm on ssh and stop start the vm
this entry we have found on syslog: dec 12 08:00:02 node12 vzdump[993084]: ERROR: can't aquire lock '/var/run/vzdump.lock' - got timeout

and we have find a log entry on node Dec 12 01:30:01 titan /USR/SBIN/CRON[964371]: (root) CMD (vzdump 5138 but this vm is not on this node and no backup shob was on this node in backuplist found? why this entry comes? can we show backup processes on ssh on this node - /etc/cron.d/vzdump here are all jobs of all nodes listeing?

if vm is nor reachabl sporadical without backup, we can connect to ssh but if we login with passwort the vm does not response, we must here stop and start vm to get online.

what we can do to fix this issue. any ideas for us?

very thanks and best regards.

 

[informant]

Hi, anywhere a idea?

The problem comes sporadical on a backup snapshot with tar.gz
vm is not reachable, backupprocess is running, ip is not reachable,
we must stop vm start vm and vm is back online
statistic and cpu ram ... in webpanel is normal, but after restart vm is is all before 0% after reboot again its normal limits

a other information, a first backup of the node run and the 2nd does stop on 0% and this vm hang up.


any ideas to solve or fix?

regards

 

[dietmar]

No real idea, but maybe it helps if you change the backup target? What target storage do you use currently?

 

[informant]

hi dietmar, we use a external storage with nfs, this is mounted on all our nodes,
the problem is only on one node.

we have start a hdd check: ok
raid: ok
ram check: ok
cpu: ok

backup starts normal of 1st vm:

INFO: starting new backup job: vzdump 1134 1131 --quiet 1 --mailto backup@mydomain.de --mode snapshot --compress gzip --storage S001 --node titan
INFO: Starting Backup of VM 1131 (qemu)
INFO: status = running
INFO: update VM 1131: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/S001/dump/vzdump-qemu-1131-2015_01_28-03_00_02.vma.gz'
INFO: started backup task '4c9c88a9-79f3-45f0-b6ae-6114a811dea1'
INFO: status: 0% (144441344/53687091200), sparse 0% (134422528), duration 3, 48/3 MB/s
INFO: status: 1% (543555584/53687091200), sparse 0% (173281280), duration 36, 12/10 MB/s
INFO: status: 2% (1075707904/53687091200), sparse 0% (207503360), duration 76, 13/12 MB/s
INFO: status: 3% (1620377600/53687091200), sparse 0% (214945792), duration 115, 13/13 MB/s
...
INFO: transferred 53687 MB in 4117 seconds (13 MB/s)
INFO: archive file size: 17.05GB
INFO: delete old backup '/mnt/pve/S001/dump/vzdump-qemu-1131-2015_01_25-03_00_02.vma.gz'
INFO: Finished Backup of VM 1131 (01:08:40)
INFO: Starting Backup of VM 1134 (qemu)
INFO: status = running
INFO: update VM 1134: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/S001/dump/vzdump-qemu-1134-2015_01_28-04_08_42.vma.gz'
INFO: started backup task '85f9fd87-07fa-4be9-91dd-187b35580467'
INFO: status: 0% (0/107374182400), sparse 0% (0), duration 3, 0/0 MB/s
hang on this position... and vm was not reachable...
is backup hang, we stop it
than comes
ERROR: interrupted by signal
INFO: aborting backup job
than we must unlock vm on node ssh
than we can stop vm and start it again

after this, backups works normal to a sporadical time and same issue comes.
same issu, on this node, if we use a other storage (internal and external).

regards