vzdump.conf tweaking - backups failing

[ttist25]

Hello,

I'm having some trouble with backups hanging. The backups were working brilliantly until a few weeks ago. I've been pouring over forum posts trying to figure out what to do to get things working again but I can't seem to figure it out. Once I do I'd like to put a post together like I did for migrating VirtualBox VM's to ProxMox. That seems to have helped a few people and my head feels now the way it did before I figured that one out. Maybe I can collect everything into a step-by-step kind of troubleshooting/tweaking guide once I get it all figured out. It's the least I could do to try and contribute to this amazing body of work.

Anyway - I'm trying to tweak my vzdump.conf in hopes that it might resolve my backup issue. I have already switched from an NFS share to a CIFS share as I've seen others have had success with this. This didn't resolve my issue. I also tried adding "size: 4096" to my vzdump.conf but that doesn't seem to have made any difference either. I have also seen reference to doubling that number but I'm not really sure what the repercussions are as I don't have a firm understanding of the value.

So three questions will get us started with tweaking the vzdump.conf:

1. First, how would one calculate a good "size:" value and what does it actually represent (I know it has something to do with the LVM size but not sure what really)
2. Next, how would one calculate a good "bwlimit" value and what is the default value (I thought I saw 10Mbps in a post)
3. Finally, how would one calculate a good "ionice" value. Is this a "priority" in context of all processes running on the VE? I believe the default is 7.

Thanks in advance for any help.

 

[tom]

before you change vzdump.conf settings, you need to analyse the problem. why is the backup failing? post the full backup log.

 

[ttist25]

before you change vzdump.conf settings, you need to analyse the problem. why is the backup failing? post the full backup log.

Hi Tom.

Thanks for your help and sorry for not replying sooner - I was looking in the forum for my post and didn't see your reply until I checked my email this morning.

In reference to the backup log, which log are you looking for? The one that's created on the destination or somewhere else? Although I've come a long way, I'm still a Linux novice at best so please don't assume any knowledge on my part. Although I have done some wonderful fun things with Linux, I still have gaping holes in my knowledge base.

As an aside - the automatic backup kicked off last night (02:00) and is currently still running (13:15) but it looks to be progressing. It's usually hung up on one of the VM's by now. Perhaps the size change did have an effect after all?

Sorry for being daft. Thanks again for your generous help.

 

[tom]

on the bottom of the GUI, you will see "tasks" - double click the "backup" log and post it here.

 

[ttist25]

on the bottom of the GUI, you will see "tasks" - double click the "backup" log and post it here.

Ok thanks Tom.

Here is a log from the 2nd:
INFO: starting new backup job: vzdump 100 101 103 106 104 102 108 109 110 --quiet 1 --mailto obfuscated@nospam.com --mode snapshot --compress lzo --storage proxbackup-smb
INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-100-2013_01_03-02_00_01.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-100-2013_01_03-02_00_01.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/100/vm-100-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 268435459072 (33.20 MiB/s)
INFO: archive file size: 25.90GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-100-2013_01_02-02_00_01.tar.lzo'
INFO: Finished Backup of VM 100 (02:08:38)
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-101-2013_01_03-04_08_39.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-101-2013_01_03-04_08_39.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 268435459072 (95.67 MiB/s)
INFO: archive file size: 18.94GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-101-2013_01_02-04_08_39.tar.lzo'
INFO: Finished Backup of VM 101 (00:44:44)
INFO: Starting Backup of VM 102 (qemu)
INFO: status = stopped
INFO: VM is locked (backup)
ERROR: Backup of VM 102 failed - command 'qm set 102 --lock backup' failed: exit code 25
INFO: Starting Backup of VM 103 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-103-2013_01_03-04_53_23.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-103-2013_01_03-04_53_23.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/103/vm-103-disk-1.raw' to archive ('vm-disk-ide0.raw')

The error in the GUI shows "Error: unexpected status". There are other log entries but I've been manually stopping the process during the day to try to backup my file server VM etc. so they may not necessarily be representative of what's actually causing the issue.

Here is the log for the job currently running - it still seems to be progressing (keeping fingers crossed):
INFO: starting new backup job: vzdump 100 101 103 106 104 102 108 109 110 --quiet 1 --mailto obfuscated@forspam.com --mode snapshot --compress lzo --storage proxbackup-smb
INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-100-2013_01_06-02_00_02.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-100-2013_01_06-02_00_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/100/vm-100-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 268435459072 (31.28 MiB/s)
INFO: archive file size: 25.89GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-100-2013_01_03-02_00_01.tar.lzo'
INFO: Finished Backup of VM 100 (02:16:40)
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-101-2013_01_06-04_16_42.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-101-2013_01_06-04_16_42.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 268435459072 (94.33 MiB/s)
INFO: archive file size: 18.89GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-101-2013_01_05-23_54_57.tar.lzo'
INFO: Finished Backup of VM 101 (00:45:28)
INFO: Starting Backup of VM 102 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-102-2013_01_06-05_02_10.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-102-2013_01_06-05_02_10.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/102/vm-102-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 161061276160 (28.77 MiB/s)
INFO: archive file size: 17.75GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-102-2013_01_03-10_49_21.tar.lzo'
INFO: Finished Backup of VM 102 (01:29:11)
INFO: Starting Backup of VM 103 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-103-2013_01_06-06_31_21.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-103-2013_01_06-06_31_21.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/103/vm-103-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 268435458560 (33.79 MiB/s)
INFO: archive file size: 21.09GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-103-2013_01_02-04_51_33.tar.lzo'
INFO: Finished Backup of VM 103 (02:06:31)
INFO: Starting Backup of VM 104 (qemu)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-104-2013_01_06-08_37_52.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-104-2013_01_06-08_37_52.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/var/lib/vz/images/104/vm-104-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 161061276160 (101.25 MiB/s)
INFO: archive file size: 10.65GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-104-2013_01_02-06_29_00.tar.lzo'
INFO: Finished Backup of VM 104 (00:25:20)
INFO: Starting Backup of VM 106 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-106-2013_01_06-09_03_12.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-106-2013_01_06-09_03_12.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/106/vm-106-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 268435458560 (32.67 MiB/s)
INFO: archive file size: 19.61GB
INFO: delete old backup '/mnt/proxbackup-smb/dump/vzdump-qemu-106-2013_01_02-07_22_43.tar.lzo'
INFO: Finished Backup of VM 106 (02:10:52)
INFO: Starting Backup of VM 108 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-108-2013_01_06-11_14_04.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-108-2013_01_06-11_14_04.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/108/vm-108-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 268435458560 (34.60 MiB/s)
INFO: archive file size: 21.02GB
INFO: Finished Backup of VM 108 (02:03:31)
INFO: Starting Backup of VM 109 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxvs1-0" created
INFO: creating archive '/mnt/proxbackup-smb/dump/vzdump-qemu-109-2013_01_06-13_17_35.tar.lzo'
INFO: adding '/mnt/proxbackup-smb/dump/vzdump-qemu-109-2013_01_06-13_17_35.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/109/vm-109-disk-1.raw' to archive ('vm-disk-ide0.raw')

I'm comfortable with SSH and can cat any of the log files that might be in there. The GUI only shows logs back to 1/2/13 and as I said, I've been stopping and running them manually recently. If there's something older you'd like me to try to get just say the word and point me to it.

Can't thank you enough.

 

[tom]

maybe our run out of snapshot space, but then your backup should stop with an error. but only if you use latest version.

upgrade to latest and try again. (to validate, post the output of "pveversion -v")

if you want to increase snapshot size you need to check first how big is your free space in the volume group. post the output of "pvdisplay"

 

[ttist25]

Hi Tom. The backup finished (and I got an email which is a first). Im out right now but will post pvdisplay as soon as I get back. My version is 2.1. Ive been too chicken to update.

 

[mir]

maybe our run out of snapshot space, but then your backup should stop with an error. but only if you use latest version.

Could it be an idea in the future to have a separate VG created for backup purpose etc? It seems to be a returning problem hitting users.

 

[tom]

AW: vzdump.conf tweaking - backups failing

Backup in 2.3 will work without lvm, Problem solved.

 

[mir]

Re: AW: vzdump.conf tweaking - backups failing

Backup in 2.3 will work without lvm, Problem solved.

An the ability to make snapshots with all storage backends is also a welcome new feature

 

[ttist25]

maybe our run out of snapshot space, but then your backup should stop with an error. but only if you use latest version.

upgrade to latest and try again. (to validate, post the output of "pveversion -v")

if you want to increase snapshot size you need to check first how big is your free space in the volume group. post the output of "pvdisplay"

Sorry for the delay -here's the output of pvdisplay:

root@proxvs1:~# vgdisplay
--- Volume group ---
VG Name pve
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1064
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.64 TiB
PE Size 4.00 MiB
Total PE 428703
Alloc PE / Size 424608 / 1.62 TiB
Free PE / Size 4095 / 16.00 GiB

I'm really hesitant to update this box if I don't have to. It's working brilliantly except for the backup issue.

Thanks again for your help.

 

[tom]

AW: vzdump.conf tweaking - backups failing

pvdisplay, not vgdisplay

 

[ttist25]

Re: AW: vzdump.conf tweaking - backups failing

pvdisplay, not vgdisplay

Doh! Sorry.

root@proxvs1:~# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name pve
PV Size 1.64 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 428703
Free PE 3071
Allocated PE 425632

 

[ttist25]

maybe our run out of snapshot space...

Hi Tom,

This seems to be working again since I changed the "size:" value in vzdump.conf to 4096. I'm guessing that's the cause for the fix but I'm still not certain. I'm having trouble understanding how I would go about calculating the proper size value for a specific host. It sounds like the information might be located in the output of "pvdisplay". Is that correct? If so, what would I look for?

Thanks again for all of your help.

 

[tom]

Re: AW: vzdump.conf tweaking - backups failing

...
PE Size 4.00 MiB
Total PE 428703
Free PE 3071
...

this means you have 3071 free. 3071 * 4 MB = 12284 MB.

so you can create snapshots up to 12284 MB. if you set 4096 in vzdump.conf there is still space. (default is 1024)