Backup error -5 input/output error

[terminus]

I get this error when backing up one of my machines:

INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --compress lzo --storage usb --node orac
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: update VM 101: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/media/backup/dump/vzdump-qemu-101-2015_08_22-16_37_56.vma.lzo'
INFO: started backup task '1c994266-b1ba-4988-acbb-a8a08ab0891a'
INFO: status: 0% (23855104/107374182400), sparse 0% (2519040), duration 3, 7/7 MB/s
INFO: status: 1% (1087111168/107374182400), sparse 0% (228904960), duration 85, 12/10 MB/s
INFO: status: 2% (2157182976/107374182400), sparse 0% (237760512), duration 168, 12/12 MB/s
INFO: status: 3% (3235250176/107374182400), sparse 0% (331046912), duration 233, 16/15 MB/s
INFO: status: 4% (4310761472/107374182400), sparse 0% (347684864), duration 311, 13/13 MB/s
INFO: status: 5% (5413666816/107374182400), sparse 0% (425197568), duration 352, 26/25 MB/s
INFO: status: 6% (6474563584/107374182400), sparse 0% (425197568), duration 384, 33/33 MB/s
INFO: status: 7% (7517896704/107374182400), sparse 0% (522604544), duration 425, 25/23 MB/s
INFO: status: 8% (8590721024/107374182400), sparse 0% (522604544), duration 505, 13/13 MB/s
INFO: status: 9% (9666428928/107374182400), sparse 0% (615350272), duration 576, 15/13 MB/s
INFO: status: 10% (10738073600/107374182400), sparse 0% (615464960), duration 648, 14/14 MB/s
INFO: status: 11% (11820924928/107374182400), sparse 0% (713023488), duration 720, 15/13 MB/s
INFO: status: 12% (12897091584/107374182400), sparse 0% (713023488), duration 804, 12/12 MB/s
INFO: status: 13% (13969653760/107374182400), sparse 0% (806354944), duration 871, 16/14 MB/s
INFO: status: 14% (15039594496/107374182400), sparse 0% (806354944), duration 948, 13/13 MB/s
INFO: status: 15% (16109993984/107374182400), sparse 0% (900157440), duration 1017, 15/14 MB/s
INFO: status: 16% (17183277056/107374182400), sparse 0% (908517376), duration 1090, 14/14 MB/s
INFO: status: 17% (18260164608/107374182400), sparse 0% (1002291200), duration 1158, 15/14 MB/s
INFO: status: 18% (19331743744/107374182400), sparse 0% (1002291200), duration 1233, 14/14 MB/s
INFO: status: 19% (20413284352/107374182400), sparse 1% (1095925760), duration 1296, 17/15 MB/s
INFO: status: 20% (21485977600/107374182400), sparse 1% (1095925760), duration 1362, 16/16 MB/s
INFO: status: 21% (22576431104/107374182400), sparse 1% (1189421056), duration 1404, 25/23 MB/s
INFO: status: 22% (23657971712/107374182400), sparse 1% (1189421056), duration 1437, 32/32 MB/s
INFO: status: 23% (24704057344/107374182400), sparse 1% (1282719744), duration 1464, 38/35 MB/s
INFO: status: 24% (25792675840/107374182400), sparse 1% (1282719744), duration 1495, 35/35 MB/s
INFO: status: 25% (26896957440/107374182400), sparse 1% (1375895552), duration 1540, 24/22 MB/s
INFO: status: 26% (27922792448/107374182400), sparse 1% (1375895552), duration 1574, 30/30 MB/s
INFO: status: 27% (29049618432/107374182400), sparse 1% (1491714048), duration 1600, 43/38 MB/s
INFO: status: 28% (30070407168/107374182400), sparse 1% (1509806080), duration 1636, 28/27 MB/s
INFO: status: 29% (31153389568/107374182400), sparse 1% (1603641344), duration 1679, 25/23 MB/s
INFO: status: 30% (32227459072/107374182400), sparse 1% (1603641344), duration 1709, 35/35 MB/s
INFO: status: 31% (33313062912/107374182400), sparse 1% (1702100992), duration 1751, 25/23 MB/s
INFO: status: 32% (34371600384/107374182400), sparse 1% (1709268992), duration 1800, 21/21 MB/s
INFO: status: 33% (35469262848/107374182400), sparse 1% (1808715776), duration 1860, 18/16 MB/s
INFO: status: 34% (36523802624/107374182400), sparse 1% (1809051648), duration 1908, 21/21 MB/s
INFO: status: 34% (37403885568/107374182400), sparse 1% (1880457216), duration 1947, 22/20 MB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
ERROR: Backup of VM 101 failed - job failed with err -5 - Input/output error
INFO: Backup job finished with errors
TASK ERROR: job errors

I tried increasing the "size" value in vzdump.conf to 108000 (my disk is 100Gb) but it didn't help. I also did a "badblocks" scan of the backup disk, but nothing showed up and the same disk successfully backs up two other machines (one of them bigger).

Help!

 

[dietmar]

Is there enough free space on the target storage (usb)?

 

[terminus]

Is there enough free space on the target storage (usb)?

Yes, there is 2.8Tb available and I'm only backing up 100Gb.

 

[Vvvs]

Hi, i'm too hawe errors:
an 18 03:13:36 INFO: status: 63% (158408048640/251255586816), sparse 48% (121108713472), duration 812, 123/46 MB/s
Jan 18 03:14:02 INFO: status: 63% (160368033792/251255586816), sparse 48% (122277289984), duration 838, 75/30 MB/s
Jan 18 03:14:02 ERROR: job failed with err -5 - Input/output error
Jan 18 03:14:02 INFO: aborting backup job
Jan 18 03:14:04 INFO: resume vm
Jan 18 03:14:04 INFO: vm is online again after 841 seconds
Jan 18 03:14:05 ERROR: Backup of VM 1001 failed - job failed with err -5 - Input/output error

Please help me to understand

 

[sahostking]

Not sure if its the same issue I had but I seemed to have solved my error like yours above doing the following:

Ran vgdisplay

vgdisplay
--- Volume group ---
VG Name pve
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 5.45 TiB
PE Size 4.00 MiB
Total PE 1429763
Alloc PE / Size 1301764 / 4.97 TiB
Free PE / Size 127999 / 500.00 GiB

The noticed I had 500GB of Free PE:

So did this :

nano /etc/vzdump.conf

changed size variable to:

size: 409600

All my backups awesome after that

As I said not sure if its realted to yours but I also got Input / Output errors.

 

[terminus]

This didn't work for me. I changed my size variable to 15000 which was about the amount of free PE that I had as per vgdisplay:

  --- Volume group ---
  VG Name  pve
  System ID  
  Format  lvm2
  Metadata Areas  1
  Metadata Sequence No  204
  VG Access  read/write
  VG Status  resizable
  MAX LV  0
  Cur LV  3
  Open LV  3
  Max PV  0
  Cur PV  1
  Act PV  1
  VG Size  2.27 TiB
  PE Size  4.00 MiB
  Total PE  596006
  Alloc PE / Size  591912 / 2.26 TiB
  Free  PE / Size  4094 / 15.99 GiB

Any other ideas?

 

[sahostking]

Your PE size may then be too small I think. You'll need to find someway of increasing it, thats my opinion.

Do you have a test server you can install Proxmox on again with larger free PE size like say 100GB, I set mine to 600GB on all our servers now as we have some large VMs that are 800GB. Never had issue since increasing the Free PE size.

 

[Vvvs]

Hi.
terminus,
will it help if I change
changed size variable to:
size: 409600
????

My system
Linux pve 2.6.32-39-pve #1 SMP Fri May 8 11:27:35 CEST 2015 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

root@pve:~# vgdisplay
--- Volume group ---
VG Name pve
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 465.63 GiB
PE Size 4.00 MiB
Total PE 119202
Alloc PE / Size 115108 / 449.64 GiB
Free PE / Size 4094 / 15.99 GiB

 

[terminus]

Thanks for the help. I don't have a test server and would rather not go to the trouble of reinstalling to increase the free PE size, unless I'm sure your suggestion would work. I'm not saying it won't, but I need to understand why the free PE size should matter. What is it used for? It is basically like a cache used for PEs read from the LV, before they are copied to the backup storage? If so, how many PEs are cached there? I have room for 4094 of them, it seems surprising if that wouldn't be enough. Sorry if some of this should be obvious from the documentation.

 

[terminus]

Hi.
terminus,
will it help if I change
changed size variable to:
size: 409600
????

I don't think so, 4094 is the number of 4Mb chunks (PEs) that you've got. It's not the total free space in those PEs. The total is the figure that comes after that (15.99Gb). Interestingly yours is the same as mine. So you should rather try changing it to 15000 or less. (But what do I know, it's still not working for me.)

 

[Vvvs]

# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
size: 15000
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST

???