Backup Trouble after upgrade 3.0 -> 3.1

gmed · Dec 20, 2013

Hi, we moved forward from 3.0 to 3.1 because of the spice feature.
Unfortunally now the old backup-events wont' work anymore.
I used local storage on the same RAID1 as the vms are and a cifs-share for external storage.
Both went fine. After the upgrade, each backup-job has started, but never has endet. No backup-data has been written neiter on local storage nor on cifs-share.
Some days ago i plugged an 3TB USB Drive on the server, now some vms could got a backup, some not.
I'm highly confused, because since starting with V2.x Backup was a fire-and-forget thingi for me?

Any hints welcome, because of the coming holydays i may have time for testing.

dietmar · Dec 21, 2013

Any hint in the backup logs?

gmed · Dec 21, 2013

Hi

Following the Info from Task History:

Backup-Job for KVM-Guest running BSD, backing up to USB-Drive:

Stop-Mode

INFO: starting new backup job: vzdump 102 --quiet 1 --mode suspend --compress lzo --storage USB-Sich
INFO: Starting Backup of VM 102 (qemu)
INFO: status = running
INFO: VM is locked (backup)
ERROR: Backup of VM 102 failed - command 'qm set 102 --lock backup' failed: exit code 25
INFO: Backup job finished with errors
TASK ERROR: job errors

Susbend Mode

INFO: starting new backup job: vzdump 102 --quiet 1 --mode suspend --compress lzo --storage USB-Sich
INFO: Starting Backup of VM 102 (qemu)
INFO: status = running
INFO: VM is locked (backup)
ERROR: Backup of VM 102 failed - command 'qm set 102 --lock backup' failed: exit code 25
INFO: Backup job finished with errors
TASK ERROR: job errors

One example for Win2003-Guest backing up to CIFS-Share

INFO: starting new backup job: vzdump 101 --quiet 1 --mode suspend --compress lzo --storage SMB-Backup --node proxmox
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: update VM 101: -lock backup
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: suspend vm
INFO: skip unused drive 'RAID-RZ:101/vm-101-disk-1.qcow2' (not included into backup)
INFO: creating archive '/media/DASI/dump/vzdump-qemu-101-2013_12_16-02_25_01.vma.lzo'
INFO: started backup task '9d999fb0-e5d7-4ef1-92cd-4e129186769f'
INFO: status: 0% (500498432/816041820160), sparse 0% (4173824), duration 3, 166/165 MB/s

This was created: /media/DASI/dump/vzdump-qemu-101-2013_12_16-02_25_01.vma.lzo
But no data furthermore?
hanging for hours

One Example for Win2008-Guest onto USB-Drive running fine

INFO: starting new backup job: vzdump 104 --quiet 1 --mode suspend --compress lzo --storage USB-Sich --node proxmox
INFO: Starting Backup of VM 104 (qemu)
INFO: status = running
INFO: update VM 104: -lock backup
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: suspend vm
INFO: creating archive '/media/USB1/dump/vzdump-qemu-104-2013_12_16-00_30_02.vma.lzo'
INFO: started backup task '4d431e68-4882-464e-8137-52de928306ea'
INFO: status: 0% (109379584/536870912000), sparse 0% (87998464), duration 3, 36/7 MB/s
INFO: status: 1% (5380636672/536870912000), sparse 0% (318988288), duration 249, 21/20 MB/s
INFO: status: 2% (10808852480/536870912000), sparse 0% (388018176), duration 449, 27/26 MB/s
INFO: status: 3% (16190472192/536870912000), sparse 0% (473538560), duration 569, 44/44 MB/s
INFO: status: 4% (21488009216/536870912000), sparse 0% (2464030720), duration 616, 112/70 MB/s
INFO: status: 5% (26894073856/536870912000), sparse 0% (2569568256), duration 714, 55/54 MB/s
INFO: status: 6% (32233226240/536870912000), sparse 0% (2672017408), duration 798, 63/62 MB/s
...
INFO: status: 98% (526979497984/536870912000), sparse 48% (259674619904), duration 6333, 1007/0 MB/s
INFO: status: 99% (532022689792/536870912000), sparse 49% (264717811712), duration 6338, 1008/0 MB/s
INFO: status: 100% (536870912000/536870912000), sparse 50% (269566029824), duration 6343, 969/0 MB/s
INFO: transferred 536870 MB in 6343 seconds (84 MB/s)
INFO: archive file size: 168.40GB
INFO: resume vm
INFO: vm is online again after 6345 seconds
INFO: Finished Backup of VM 104 (01:45:46)
INFO: Backup job finished successfully
TASK OK

If you need some more data, let me know.
And many thanks for helping us.

dietmar · Dec 21, 2013

Unlock the VM, then try again.

# qm unlock 102

gmed · Dec 21, 2013

Ok

With guest 102 BSD it was my fault in not seeing 102 beeing locked...acutally running manual backup onto USB-Drive.

But there´s maybe still the issue with automated backups to CIFS-Share. We will see.

I´m interessted in getting commercial support. Maybe you could give my mail address to one of your collegues to contact me?
Would be nice.

dietmar · Dec 21, 2013

gmed said:
I´m interessted in getting commercial support.

http://pve.proxmox.com/wiki/Get_support

gmed · Dec 22, 2013

Bump.... Tonights Backup-Job failed using USB-Device, which worked fine manually

INFO: starting new backup job: vzdump 103 --remove 0 --mode snapshot --compress lzo --storage USB-Sich --node proxmox
INFO: Starting Backup of VM 103 (qemu)
INFO: status = running
INFO: update VM 103: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/media/USB1/dump/vzdump-qemu-103-2013_12_21-21_12_18.vma.lzo'
ERROR: got timeout
INFO: aborting backup job
ERROR: Backup of VM 103 failed - got timeout
INFO: Backup job finished with errors
TASK ERROR: job errors

not amusing...

Nachtfalke · Dec 25, 2013

Hi gmed,

I probably have the same problem as you and posted this some weeks ago here in the forum. There are others which seem to have the same problem with CIFS.
http://forum.proxmox.com/threads/16746-NFS-Performance-Cifs-3-1
In the thread I posted we narrowed a date when a change in proxmox or some modules happend and since then the backup feature with CIFS didn't work anymore.

Unfortunately several hints, tips and tricks did not help to solve the problem and then after some time there wasn't any feedback and suggestions to solve this problem anymore. Hopefully you can provide better/more information and this could be solved with an update.

Good luck!

gmed · Dec 25, 2013

My last check before holidays was a "manual" backup with rsync.
I startet a rsync job at the console via ssh from local storage to the mounted cifs-share.
Surprise surprise, rsync got stuck after 3-4 minutes.
So it seems for me a more general Problem, maybe related to the kernel ore the cifs modul in wheezy.
I have another wheezy station and will try to check wether pure Debian shows the same.

Nachtfalke said:
Hi gmed,

I probably have the same problem as you and posted this some weeks ago here in the forum. There are others which seem to have the same problem with CIFS.
http://forum.proxmox.com/threads/16746-NFS-Performance-Cifs-3-1
In the thread I posted we narrowed a date when a change in proxmox or some modules happend and since then the backup feature with CIFS didn't work anymore.

Unfortunately several hints, tips and tricks did not help to solve the problem and then after some time there wasn't any feedback and suggestions to solve this problem anymore. Hopefully you can provide better/more information and this could be solved with an update.

Good luck!

philbrearley · Dec 30, 2013

I have been suffering from the same problem - timeout backing up to CIFS
Kernel in use at the time was 2.6.32-26
I reverted to the 2.6.32-23 kernel and backups worked again.

Hope this helps others.

k11a · Mar 12, 2014

Anyone tested this problem on Proxmox 3.2?

philbrearley · Mar 18, 2014

Tested in Proxmox 3.2

Same result,

INFO: status: 15% (82566119424/536870912000), sparse 15% (82110078976), duration 27, 3736/14 MB/s
INFO: status: 16% (87087448064/536870912000), sparse 16% (85914935296), duration 37, 452/71 MB/s
INFO: status: 18% (99618586624/536870912000), sparse 18% (98440892416), duration 40, 4177/1 MB/s
INFO: status: 19% (102032605184/536870912000), sparse 18% (99552583680), duration 58, 134/72 MB/s
ERROR: VM 101 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: Backup of VM 101 failed - VM 101 qmp command 'query-backup' failed - got timeout

dietmar · Mar 18, 2014

philbrearley said:
Tested in Proxmox 3.2

Same result,

What kind of backup storage do you use?

philbrearley · Mar 21, 2014

This was a backup to a CIFS storage device (Zyxel NSA310)
As stated earlier in the post, all worked well until upgrading to 3.1
Backing off the kernel to 2.6.32-23 cured the problem and backup worked again.
Have now retested after upgrading to 3.2 and backups fail again with timeout.

I have been suffering from the same problem - timeout backing up to CIFS
Kernel in use at the time was 2.6.32-26
I reverted to the 2.6.32-23 kernel and backups worked again.

Phil Brearley

dietmar · Mar 22, 2014

Seems CIFS is seriously broken in newer 2.6.32 kernel (but works with 3.10 kernel).

m.ardito · Mar 22, 2014

so could a machine (vm o phisical) mounting cifs with a not broken kernel, and sharing it with nfs be a working backup storage alternative for those who have a cifs only device and still need pve 2.6.32 for openvz?

Marco

mir · Mar 22, 2014

But why use cifs if it is broken?

dietmar · Mar 22, 2014

We should fix it. But so far I have no clue whats wrong.

m.ardito · Mar 22, 2014

mir said:
But why use cifs if it is broken?

with the right kernel, cifs shoud work afaik

about why, well the device could have only cifs. that zyxel could have nfs, too but cifs is easier to share with other windows hosts

Marco

Backup Trouble after upgrade 3.0 -> 3.1

Well-Known Member

Proxmox Staff Member

Well-Known Member

Proxmox Staff Member

Well-Known Member

Proxmox Staff Member

Well-Known Member

Member

Well-Known Member

New Member

Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

Active Member

Famous Member

Proxmox Staff Member

Active Member