VM halts on backup

[lister]

Hi, ive been using proxmox for almost a year and everething was OK, but after upgrade the shuddled backup option started to give errors.
I have an identical machine HP ML 350 with 5-th raid but with a proxmox without upgrade and backup option works perfectly.
here are the logs...

INFO: starting new backup job: vzdump 100 --remove 0 --mode snapshot --compress lzo --storage backup --node proxmox
INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: update VM 100: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/backup/dump/vzdump-qemu-100-2014_01_16-14_40_02.vma.lzo'
INFO: started backup task '777de1f8-e4ba-45b7-a572-071f6ab26acc'
INFO: status: 1% (224985088/16106127360), sparse 0% (128503808), duration 3, 74/32 MB/s
INFO: status: 2% (452460544/16106127360), sparse 1% (233869312), duration 6, 75/40 MB/s
...
INFO: status: 32% (5184356352/16106127360), sparse 11% (1903751168), duration 98, 44/32 MB/s
INFO: status: 33% (5378605056/16106127360), sparse 12% (1960067072), duration 101, 64/45 MB/s
INFO: status: 34% (5529927680/16106127360), sparse 12% (2017292288), duration 104, 50/31 MB/s
INFO: status: 35% (5732958208/16106127360), sparse 12% (2089041920), duration 107, 67/43 MB/s
INFO: status: 36% (5930483712/16106127360), sparse 13% (2148683776), duration 110, 65/45 MB/s
INFO: status: 37% (6113787904/16106127360), sparse 13% (2204942336), duration 113, 61/42 MB/s
INFO: status: 38% (6134956032/16106127360), sparse 13% (2205147136), duration 116, 7/6 MB/s
after it stalls with the error
ERROR: VM 100 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: Backup of VM 100 failed - VM 100 qmp command 'query-backup' failed - got timeout

root@proxmox:~# pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-24 (running version: 3.1-24/060bd5a6)
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-9
libpve-access-control: 3.0-8
libpve-storage-perl: 3.0-18
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-6
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1

the only one significant differences from two of this servers is in pve-kernel-2.6.32-26-pve: 2.6.32-114 vs proxmox-ve-2.6.32-23-pve from august 6 2013

The backup jub stores archives on mounted windows share folder. i think this is due to something in pve kernel, cose ive reinstalled proxmox with a fresh install but the problem persists, so its not in the hardware, to be sure i need to install proxmox-ve-2.6.32-23-pve kernel and start server with it, how can i do that?

 

[reno]

I've the same issue on 2 VMs.
Everything was OK before upgrading.

 

[lister]

OK, so ive got the answer. I dont know for sure but theres something wrong with the pve-kernel module pve-kernel-2.6.32-26-pve: 2.6.32-114, cose ive installed the 23-pve kernel and everething works for now perfectly. The backup job runs great and conform shuduller. if anyone has the same problem you should do next steps:

nano /etc/apt/sources.list

# PVE pve-no-subscription repository provided by proxmox.com, NOT recommended
deb http://download.proxmox.com/debian wheezy pve-no-subscription

aptitude install pve-kernel-2.6.32-23-pve

after those steps reboot the proxmox and select kernel indicated above. TRY IT THIS SHOULD BE INUF TO EVERETHING WORK PERFECTLY, in any other case just load the default conf , cose we didnt chenge the grub config file.
if the problem is solved chenge the default load on the grub ang you are the KING.

 

[lister]

So, im back one more time, this time just to update the topic, the problem persist even on the latest hardware HP ml 350 gen 8 its the same
do not forget to chenge grub.cfg after install the older core

menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 2.6.32-23-pve' --class proxmox --class gnu-linux --class gnu --class os {
load_video
insmod gzio
insmod part_gpt
insmod ext2
set root='(hd0,gpt2)'
search --no-floppy --fs-uuid --set=root blablabla
echo 'Loading Linux 2.6.32-23-pve ...'
linux /vmlinuz-2.6.32-23-pve root=/dev/mapper/pve-root ro quiet
echo 'Loading initial ramdisk ...'
initrd /initrd.img-2.6.32-23-pve
}
menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 2.6.32-23-pve (recovery mode)' --class proxmox --class gnu-linux --class gnu --class os {
load_video
insmod gzio
insmod part_gpt
insmod ext2
set root='(hd0,gpt2)'
search --no-floppy --fs-uuid --set=root blablabla
echo 'Loading Linux 2.6.32-23-pve ...'
linux /vmlinuz-2.6.32-23-pve root=/dev/mapper/pve-root ro single
echo 'Loading initial ramdisk ...'
initrd /initrd.img-2.6.32-23-pve
}

in my case

 

[spirit]

Hi, which version of proxmox do you use?

I think that this backup crash bug has been fixed in proxmox 3.2.

(backup should fail if backup storage is too slow but should not crash the vm)

 

[lister]

Hi, which version of proxmox do you use?

I think that this backup crash bug has been fixed in proxmox 3.2.

(backup should fail if backup storage is too slow but should not crash the vm)

nope its still here
proxmox version 3.2-4/e24a91c1

 

[dietmar]

proxmox version 3.2-4/e24a91c1

Please update and test again.

 

[lister]

Please update and test again.

made the update,PROXMOX version 3.3-1/a06c9f73 . After restarted the prox ..... problem still there

INFO: starting new backup job: vzdump 110 --remove 0 --mode snapshot --compress lzo --storage backup --node 350p
INFO: Starting Backup of VM 110 (qemu)
INFO: status = running
INFO: update VM 110: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/backup/dump/vzdump-qemu-110-2014_09_19-06_51_57.vma.lzo'
INFO: started backup task '6aa24286-8b78-4c10-a227-fec1ae4926c5'
INFO: status: 1% (319291392/21474836480), sparse 0% (3723264), duration 3, 106/105 MB/s
INFO: status: 2% (630980608/21474836480), sparse 0% (7360512), duration 6, 103/102 MB/s
INFO: status: 4% (1008271360/21474836480), sparse 0% (11145216), duration 9, 125/124 MB/s
INFO: status: 6% (1353580544/21474836480), sparse 0% (11714560), duration 12, 115/114 MB/s
INFO: status: 7% (1615527936/21474836480), sparse 0% (12165120), duration 15, 87/87 MB/s
INFO: status: 9% (1934753792/21474836480), sparse 0% (12906496), duration 18, 106/106 MB/s
INFO: status: 10% (2286944256/21474836480), sparse 0% (16830464), duration 21, 117/116 MB/s
INFO: status: 12% (2774794240/21474836480), sparse 0% (18554880), duration 24, 162/162 MB/s
INFO: status: 15% (3246129152/21474836480), sparse 0% (47030272), duration 27, 157/147 MB/s
INFO: status: 17% (3725066240/21474836480), sparse 0% (47030272), duration 31, 119/119 MB/s
INFO: status: 19% (4200202240/21474836480), sparse 0% (47030272), duration 34, 158/158 MB/s
INFO: status: 21% (4576509952/21474836480), sparse 0% (48963584), duration 37, 125/124 MB/s
INFO: status: 23% (5047844864/21474836480), sparse 0% (50237440), duration 40, 157/156 MB/s
INFO: status: 33% (7267745792/21474836480), sparse 9% (1987710976), duration 43, 739/94 MB/s
INFO: status: 36% (7799832576/21474836480), sparse 9% (1998925824), duration 46, 177/173 MB/s
INFO: status: 38% (8267366400/21474836480), sparse 9% (2011803648), duration 49, 155/151 MB/s
INFO: status: 40% (8639873024/21474836480), sparse 9% (2011983872), duration 52, 124/124 MB/s
INFO: status: 41% (9012379648/21474836480), sparse 9% (2030968832), duration 55, 124/117 MB/s
INFO: status: 43% (9255649280/21474836480), sparse 9% (2036162560), duration 58, 81/79 MB/s
INFO: status: 44% (9609150464/21474836480), sparse 9% (2036973568), duration 61, 117/117 MB/s
INFO: status: 46% (9977856000/21474836480), sparse 9% (2036973568), duration 64, 122/122 MB/s
INFO: status: 48% (10357964800/21474836480), sparse 9% (2036973568), duration 67, 126/126 MB/s
INFO: status: 50% (10737483776/21474836480), sparse 9% (2036973568), duration 70, 126/126 MB/s
INFO: status: 52% (11175198720/21474836480), sparse 9% (2037010432), duration 73, 145/145 MB/s

after this it could take for 2-3 hours and then gives the error

ERROR: VM 110 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: Backup of VM 110 failed - VM 110 qmp command 'query-backup' failed - got timeout

The backup is been made to a NFS share on win xp PC with full access, the folder is mounted with the manual from on site. In the graphic ui it shows the size and content normal.
The restore procidure is ok and speed is about 60-110 mb/s
even after upgrade ive loaded the older kernel and backup went well....... any ideas????

 

[lister]

made the update,PROXMOX version 3.3-1/a06c9f73 . After restarted the prox ..... problem still there

little update, when the backup operation is in progress these no data transfer to the remote folder till ERROR, but if during the backup i press STOP button ,before the error, the file on the remote host begins to grow (raze in size).
Sorry for my English

 

[term]

Can you try backing up over NFS to a linux machine instead? What NFS stack are you running on windows xp?

 

[lister]

Ive try ed, its the same on unix and windows machines, the result is the same ERROR, what do you mean NFS stack??? its just a simple shared folder with full permission

 

[lister]

Please update and test again.

any ideas?

 

[lister]

tryed the newly released version of proxmox, the problem persists