QEMU backup task never ends

G

GerShock

New Member
Feb 17, 2014
7
0
1
Hi, i'm having a problem after more than 8 successfully backups this way. The last backup of a QEMU VM never ends:

Code: 
[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 106 --quiet 1 --mailto xxxxx@xxxxxx.xxx --mode snapshot --compress lzo --storage bckServer --node pve01[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 106 (qemu)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: update VM 106: -lock backup[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: backup mode: snapshot[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: ionice priority: 7[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating archive '/mnt/bckServer//dump/vzdump-qemu-106-2014_02_15-22_00_02.vma.lzo'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: started backup task '5a745787-a809-4de7-8b6f-def27b35a8fb'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 0% (339410944/171798691840), sparse 0% (139231232), duration 3, 113/66 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 1% (1856110592/171798691840), sparse 0% (245825536), duration 41, 39/37 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 2% (4121755648/171798691840), sparse 0% (1606180864), duration 54, 174/69 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 3% (5909250048/171798691840), sparse 1% (3202113536), duration 58, 446/47 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 4% (7845838848/171798691840), sparse 2% (4997861376), duration 61, 645/46 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 6% (11043667968/171798691840), sparse 4% (8143081472), duration 64, 1065/17 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 8% (14966194176/171798691840), sparse 7% (12056051712), duration 67, 1307/3 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 11% (19341049856/171798691840), sparse 9% (16430833664), duration 70, 1458/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 13% (23496491008/171798691840), sparse 11% (20585984000), duration 73, 1385/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 16% (27968929792/171798691840), sparse 14% (25058398208), duration 76, 1490/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 17% (30358044672/171798691840), sparse 15% (27305852928), duration 79, 796/47 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 18% (31949651968/171798691840), sparse 16% (28503879680), duration 84, 318/78 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 21% (36264804352/171798691840), sparse 19% (32808570880), duration 87, 1438/3 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 23% (40565866496/171798691840), sparse 21% (37109624832), duration 90, 1433/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 26% (45171408896/171798691840), sparse 24% (41714954240), duration 93, 1535/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 28% (49730748416/171798691840), sparse 26% (46274240512), duration 96, 1519/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 31% (54031810560/171798691840), sparse 29% (50575294464), duration 99, 1433/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 34% (59808350208/171798691840), sparse 32% (56351825920), duration 102, 1925/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 37% (64102596608/171798691840), sparse 35% (60629471232), duration 105, 1431/5 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 39% (68701126656/171798691840), sparse 37% (65227993088), duration 108, 1532/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 42% (73217277952/171798691840), sparse 40% (69744119808), duration 111, 1505/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 45% (78034960384/171798691840), sparse 43% (74561794048), duration 114, 1605/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 48% (82973491200/171798691840), sparse 46% (79500316672), duration 117, 1646/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 49% (84523810816/171798691840), sparse 47% (80917082112), duration 120, 516/44 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 51% (89176801280/171798691840), sparse 49% (85570064384), duration 123, 1550/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 53% (92316172288/171798691840), sparse 51% (88709431296), duration 126, 1046/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 56% (96654721024/171798691840), sparse 54% (93047963648), duration 129, 1446/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 57% (99097903104/171798691840), sparse 55% (95321165824), duration 132, 814/56 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 58% (99652993024/171798691840), sparse 55% (95368151040), duration 138, 92/84 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 59% (102719684608/171798691840), sparse 57% (98288070656), duration 142, 766/36 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 62% (107337547776/171798691840), sparse 59% (102905860096), duration 145, 1539/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 65% (111737307136/171798691840), sparse 62% (107305607168), duration 148, 1466/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 67% (116299726848/171798691840), sparse 65% (111868018688), duration 151, 1520/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 70% (120843730944/171798691840), sparse 67% (116412014592), duration 154, 1514/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 71% (122770751488/171798691840), sparse 68% (118106071040), duration 157, 642/77 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 72% (124580659200/171798691840), sparse 69% (119514116096), duration 162, 361/80 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 73% (126884577280/171798691840), sparse 70% (121653415936), duration 165, 767/54 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 75% (128989528064/171798691840), sparse 71% (123643895808), duration 168, 701/38 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 76% (131738238976/171798691840), sparse 73% (126251343872), duration 171, 916/47 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 78% (135436959744/171798691840), sparse 75% (129899053056), duration 174, 1232/17 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 80% (138208411648/171798691840), sparse 77% (132532342784), duration 177, 923/46 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 81% (139749097472/171798691840), sparse 78% (134016049152), duration 180, 513/18 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 82% (142218166272/171798691840), sparse 79% (136368041984), duration 183, 823/39 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 84% (145709531136/171798691840), sparse 81% (139788365824), duration 186, 1163/23 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 86% (148368261120/171798691840), sparse 82% (142296539136), duration 189, 886/50 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 88% (152188616704/171798691840), sparse 85% (146066956288), duration 192, 1273/16 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 90% (155882881024/171798691840), sparse 87% (149715927040), duration 195, 1231/15 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 92% (159105810432/171798691840), sparse 88% (152878116864), duration 198, 1074/20 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 94% (162641346560/171798691840), sparse 91% (156374978560), duration 201, 1178/12 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 95% (163416375296/171798691840), sparse 91% (157088083968), duration 204, 258/20 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 96% (165561368576/171798691840), sparse 92% (159205728256), duration 207, 714/9 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 98% (170038525952/171798691840), sparse 95% (163670646784), duration 210, 1492/4 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 100% (171798691840/171798691840), sparse 96% (165430812672), duration 211, 1760/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: transferred 171798 MB in 211 seconds (814 MB/s)

It still in there during 2 days and i tried to kill the process from Web and command line without lucky. Can someone give me a clue? I'm without backups since last 2 days.

Additional information:
[/FONT][/COLOR]
Code: 
root@pve01:~# pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-24 (running version: 3.1-24/060bd5a6)
pve-kernel-2.6.32-24-pve: 2.6.32-111
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-9
libpve-access-control: 3.0-8
libpve-storage-perl: 3.0-18
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-6
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1

Thanks.

 
dietmar said:
What kind of backup storage is that - NFS?
Click to expand...

this came to my mind just now: could it be possibile to use local (temp) storage to create and complete the lzo.vma (or gz.vma), and only then move that completed backup file to the backup storage (so, just store it there), at least as an option? would this help many nfs backup (or similar storage performance problems) issues?

Marco
 
Marco's idea is the right one. I did something like you did for a while. After having to reboot the host a few times, I gave up and started doing local backups and copying them off.
 
I meant that trick as a possible (optional, even default) way for pve to cut out all external storage performance problems I read alot this days. But enough local space is needed, though.
but obviously you can always script you custom backups on the CLI & cron them (or even usgin hooks, see http://pve.proxmox.com/wiki/Vzdump_manual)

@GerShock. I have not such problems, using nfs server. You could also try using NFS and a quicker storage system.
Probably with such a low performance ftp storage (due to network, server or mount method/options) some timeout is involved and the script just hangs. check out relevant system logs too.

Marco
 
term said:
Marco's idea is the right one. I did something like you did for a while. After having to reboot the host a few times, I gave up and started doing local backups and copying them off.
Click to expand...

Thanks, that idea is in course. I wrote a script to run with a cron to do the copy from the local storage to the remote one, but now I can't stop the "zombie" backup task and the reboot the proxmox server is not viable for a time.

I need a way to kill the backup task which never ends.
 
Thanks! But I have to tell you the same as I said to @term... If you know how to kill the backup task, I'll be really grateful.


Greetings!
 
GerShock,

That's what I was getting at. The process is stuck in the kernel and cannot be killed. I had to wait several days until I could reboot the host, and during that time proxmox backups would not work. It's the same problem that people face with NFS backup locations. I spent way too much time trying to figure out how to kill the stuck process, it simply cannot be done in some cases. If you have already tried kill -9 with no success, then all you can do is work on a alternative way of getting your backups done and wait for the next maintenance window. Sorry I don't have any better answers.
 
try also to force umount with lazy option, as last resort. It will be autoremounted when needed.
perhaps will work...

#umount -f -l /mnt/bckServer
Marco
 
Thanks to everyone. I didn't have lucky trying to kill the backup task without rebooting the server.


The "solution" will be:
1- Change backup tasks doing it over a local storage.
2- Reboot the server.
3- Run periodicaly a script that moves the backups from local to remote storage. (maybe through a cron job that make a copy and then deletes the original backup file)


Hope this helps!


Greetings...
 
Hi,

I have just setup an FTP backup with Curlftpfs as above!
And I am facing the exact same issue..

This is all i have found regarding a simmilar setup so far!
and i was wondering, have anyone found a way to fix this, that does NOT include using local storage first and moving them afterwards?

Thankyou :)
Kindly
Niels
 
so now is year 2023 with proxmox v7.4-3
i backing up to this added storage in pve host

this usb storage passthrough to a Ubuntu VM and setup as SMB Share. then in pve host, is added as storage.

stuck at INFO: transferred xxx MB in xxx seconds.
reboot host and it get stuck without able to reboot. need a hard reset computer
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back