Backup job failed with err -5 - Input/output error

[Cerias]

Hey folks 0/

after our upgrade last weekend from proxmox 3.4 to 4.1 we got only by 2 vm's strange errors wich has no clear hint for me.

Any time at the same position for this vm it throws an error. The VM has 3 vd's in the raw format. Other VM with more vd's have no problems.

It's enough space on the target left.

Mar 03 00:00:02 INFO: Starting Backup of VM 101 (qemu)
Mar 03 00:00:02 INFO: status = running
Mar 03 00:00:02 INFO: update VM 101: -lock backup
Mar 03 00:00:03 INFO: backup mode: snapshot
Mar 03 00:00:03 INFO: ionice priority: 7
Mar 03 00:00:03 INFO: creating archive '/mnt/pve/nas2-backup/dump/vzdump-qemu-101-2016_03_03-00_00_02.vma.lzo'
Mar 03 00:00:03 INFO: started backup task '8c63a7bf-0e5b-4280-a828-844368d3ca15'
Mar 03 00:00:06 INFO: status: 0% (369229824/216409309184), sparse 0% (4096000), duration 3, 123/121 MB/s
Mar 03 00:00:25 INFO: status: 1% (2170421248/216409309184), sparse 0% (11292672), duration 22, 94/94 MB/s

Spoiler: proccessing...

Mar 03 00:00:53 INFO: status: 2% (4365221888/216409309184), sparse 0% (20099072), duration 50, 78/78 MB/s
Mar 03 00:02:17 INFO: status: 3% (6508904448/216409309184), sparse 0% (46469120), duration 134, 25/25 MB/s
Mar 03 00:04:10 INFO: status: 4% (8659664896/216409309184), sparse 0% (74747904), duration 247, 19/18 MB/s
Mar 03 00:05:57 INFO: status: 5% (10837360640/216409309184), sparse 0% (101425152), duration 354, 20/20 MB/s
Mar 03 00:07:32 INFO: status: 6% (13009616896/216409309184), sparse 0% (132214784), duration 449, 22/22 MB/s
Mar 03 00:08:18 INFO: status: 7% (15196291072/216409309184), sparse 0% (136908800), duration 495, 47/47 MB/s
Mar 03 00:09:02 INFO: status: 8% (17320968192/216409309184), sparse 0% (199540736), duration 539, 48/46 MB/s
Mar 03 00:09:46 INFO: status: 9% (19480576000/216409309184), sparse 0% (209289216), duration 583, 49/48 MB/s
Mar 03 00:10:26 INFO: status: 10% (21700411392/216409309184), sparse 0% (248827904), duration 623, 55/54 MB/s
Mar 03 00:11:00 INFO: status: 11% (23904256000/216409309184), sparse 0% (282615808), duration 657, 64/63 MB/s
Mar 03 00:11:27 INFO: status: 12% (26051084288/216409309184), sparse 0% (330317824), duration 684, 79/77 MB/s
Mar 03 00:11:46 INFO: status: 13% (28226879488/216409309184), sparse 0% (362209280), duration 703, 114/112 MB/s
Mar 03 00:12:19 INFO: status: 14% (30375346176/216409309184), sparse 0% (396156928), duration 736, 65/64 MB/s
Mar 03 00:12:43 INFO: status: 15% (32470204416/216409309184), sparse 0% (416141312), duration 760, 87/86 MB/s
Mar 03 00:13:04 INFO: status: 16% (34720776192/216409309184), sparse 0% (460230656), duration 781, 107/105 MB/s
Mar 03 00:13:32 INFO: status: 17% (36813602816/216409309184), sparse 0% (493768704), duration 809, 74/73 MB/s
Mar 03 00:13:59 INFO: status: 18% (38984155136/216409309184), sparse 0% (535425024), duration 836, 80/78 MB/s
Mar 03 00:14:21 INFO: status: 19% (41210937344/216409309184), sparse 0% (625389568), duration 858, 101/97 MB/s
Mar 03 00:14:47 INFO: status: 20% (43292098560/216409309184), sparse 0% (786464768), duration 884, 80/73 MB/s
Mar 03 00:15:15 INFO: status: 21% (45486768128/216409309184), sparse 0% (842862592), duration 912, 78/76 MB/s
Mar 03 00:15:47 INFO: status: 22% (47665512448/216409309184), sparse 0% (876154880), duration 944, 68/67 MB/s

Mar 03 00:16:20 INFO: status: 23% (49839865856/216409309184), sparse 0% (903819264), duration 977, 65/65 MB/s
Mar 03 00:16:36 INFO: status: 23% (51058769920/216409309184), sparse 0% (906022912), duration 993, 76/76 MB/s
Mar 03 00:16:45 ERROR: job failed with err -5 - Input/output error
Mar 03 00:16:45 INFO: aborting backup job
Mar 03 00:16:48 ERROR: Backup of VM 101 failed - job failed with err -5 - Input/output error

There are no future configs changes on the /etc/vzdump.conf.

Could anyone give me some hints whats broken?

 

[benitoll]

Hello, I have this very same issue, filesystem is ZFS (VM on zvol and backup storage either the very same ZFS filesystem or NFS, there is sufficient space in both and always fail at exactly the same percentaje in every attempt).

This is a serious problem for me right now because my dedicated server expires tonight and need to move the VM (luckily only one but a big one) to another server, so please if anyone could help I'd be very grateful, I will try converting the VDI to qcow2 and manually backing it up and the conf file, but It will mean a long(er) downtime so leave that for last resort.

The weirdest thing is that previous backups of smaller sized VMs worked, I just realized of this incoherence while typing this post so maybe it's filling the RAM? there IS enough space in the filesystems and disks...

Another weird thing is that apt-get can't update for some reason (I don't care about fixing it unless it's the reason for the backups failing, as this server will be cancelled).

Spoiler: Backup fail example

It doesn't matter stop/snapshot to ZFS/NFS, the result is always the same: failing at 28%.

INFO: starting new backup job: vzdump 200 --storage local --compress lzo --remove 0 --node nodo3 --mode snapshot
INFO: Starting Backup of VM 200 (qemu)
INFO: status = running
INFO: update VM 200: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-200-2016_04_12-00_53_43.vma.lzo'
INFO: started backup task '78a8e4af-20db-4da4-a8d2-501469ed9e07'
INFO: status: 0% (103481344/161061273600), sparse 0% (10551296), duration 3, 34/30 MB/s
INFO: status: 1% (1620180992/161061273600), sparse 0% (32456704), duration 47, 34/33 MB/s
INFO: status: 2% (3223322624/161061273600), sparse 0% (57962496), duration 95, 33/32 MB/s
INFO: status: 3% (4993122304/161061273600), sparse 0% (58019840), duration 108, 136/136 MB/s
INFO: status: 4% (6449856512/161061273600), sparse 0% (587362304), duration 117, 161/103 MB/s
INFO: status: 5% (8153333760/161061273600), sparse 0% (631779328), duration 130, 131/127 MB/s
INFO: status: 6% (9736159232/161061273600), sparse 0% (654086144), duration 142, 131/130 MB/s
INFO: status: 7% (11352735744/161061273600), sparse 1% (1996222464), duration 233, 17/3 MB/s
INFO: status: 8% (12933136384/161061273600), sparse 2% (3576229888), duration 245, 131/0 MB/s
INFO: status: 9% (14496825344/161061273600), sparse 2% (4117659648), duration 257, 130/85 MB/s
INFO: status: 10% (16212623360/161061273600), sparse 2% (4123185152), duration 270, 131/131 MB/s
INFO: status: 11% (17793744896/161061273600), sparse 2% (4471955456), duration 282, 131/102 MB/s
INFO: status: 12% (19363004416/161061273600), sparse 3% (6038790144), duration 294, 130/0 MB/s
INFO: status: 13% (20940193792/161061273600), sparse 4% (7126253568), duration 306, 131/40 MB/s
INFO: status: 14% (22650683392/161061273600), sparse 4% (7127633920), duration 363, 30/29 MB/s
INFO: status: 15% (24257101824/161061273600), sparse 4% (7505887232), duration 375, 133/102 MB/s
INFO: status: 16% (25831473152/161061273600), sparse 5% (8947953664), duration 387, 131/11 MB/s
INFO: status: 17% (27407089664/161061273600), sparse 6% (10516275200), duration 399, 131/0 MB/s
INFO: status: 18% (29101129728/161061273600), sparse 6% (10816176128), duration 412, 130/107 MB/s
INFO: status: 19% (30700929024/161061273600), sparse 6% (11033890816), duration 424, 133/115 MB/s
INFO: status: 20% (32269795328/161061273600), sparse 6% (11228897280), duration 436, 130/114 MB/s
INFO: status: 21% (33847312384/161061273600), sparse 6% (11244400640), duration 448, 131/130 MB/s
INFO: status: 22% (35555377152/161061273600), sparse 7% (11805040640), duration 461, 131/88 MB/s
INFO: status: 23% (37132828672/161061273600), sparse 7% (11859251200), duration 473, 131/126 MB/s
INFO: status: 24% (38773391360/161061273600), sparse 7% (11859488768), duration 621, 11/11 MB/s
INFO: status: 25% (40355233792/161061273600), sparse 7% (11874365440), duration 633, 131/130 MB/s
INFO: status: 26% (41891790848/161061273600), sparse 7% (11874779136), duration 668, 43/43 MB/s
INFO: status: 27% (43584520192/161061273600), sparse 7% (11875360768), duration 817, 11/11 MB/s
INFO: status: 28% (45145194496/161061273600), sparse 7% (11876577280), duration 829, 130/129 MB/s
INFO: status: 28% (45902528512/161061273600), sparse 7% (11881664512), duration 1046, 3/3 MB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
ERROR: Backup of VM 200 failed - job failed with err -5 - Input/output error
INFO: Backup job finished with errors
TASK ERROR: job errors

Spoiler: aptitude failing to update

root@nodo3:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  faketime libfaketime thin-provisioning-tools
The following packages will be upgraded:
  apt-transport-https apt-utils bind9-host dnsutils git git-man initramfs-tools
  libapt-inst1.5 libbind9-90 libcairo2 libdns-export100 libdns100 libglib2.0-0
  libgraphite2-3 libgtk2.0-0 libgtk2.0-common libhogweed2 libirs-export91 libisc-export95
  libisc95 libisccc90 libisccfg-export90 libisccfg90 libjasper1 liblwres90 libnettle4
  libpam-modules libpam-modules-bin libpam0g libpcre3 libpve-access-control
  libpve-common-perl libpve-storage-perl libsmbclient libsndfile1 libssl1.0.0 libsystemd0
  libwbclient0 locales lxcfs multiarch-support openssl proxmox-ve pve-cluster pve-container
  pve-firewall pve-ha-manager pve-kernel-4.2.8-1-pve pve-manager pve-qemu-kvm qemu-server
  samba-common samba-libs smbclient systemd systemd-sysv tzdata
57 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/84.3 MB of archives.
After this operation, 18.5 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Reading changelogs... Done
Extracting templates from packages: 100%
Preconfiguring packages ...
Setting up udev (215-17+deb8u4) ...
addgroup: The group `input' already exists as a system group. Exiting.
update-initramfs: deferring update (trigger activated)
insserv: Service zfs-mount has to be enabled to start service udev-finish
insserv: exiting now!
update-rc.d: error: insserv rejected the script header
dpkg: error processing package udev (--configure):
subprocess installed post-installation script returned error exit status 1
Processing triggers for initramfs-tools (0.120) ...
update-initramfs: Generating /boot/initrd.img-4.2.8-1-pve
Errors were encountered while processing:
udev
E: Sub-process /usr/bin/dpkg returned an error code (1)

 

[Cerias]

@benitoll

my -5 error while backups is gone after i completly shutdown the VM and restarted it.

I hope this will also work for you.

 

[benitoll]

@Cerias thanks for the answer, but unfortunately that was not the problem. I couldn't even really fix it (even converting the image failed, and after filling some disk space inside the VM it "input/output error" crashed, so I think the disk image was somehow corrupt/faulty).

I could save the data in the hardest way (reinstalling the VM in the new server and moving manual backups of the data), I haven't slept yet but it's almost over, and just in time (by experience, the server will be disconnected in 2 hours).

 

[grl]

Anyone has an idea on that problem? Having the same problem here.

Our play-around-pve-installation failed. So we installed a new one using the recent ISO from the page (4.2).
As the disk images come from an external storage via NFS we recreated the VMs and then changed the harddisk images to the ones on the NFS-Server by editing the conf by hand.

The VMs boot as expected. Most of them have only really small Harddisks (around 10G) - they can be backed up using vzdump.
There is just one VM with a 60G HDD. If I try to do vzdump here it always stops at 78% with and error 5 input/output error.
I tried with VM running, VM stopped, all different compression levels, backing up to local disk (ext4) or to an NFS share - every time it stops at 78%

I tried to boot the VM from a systemresq-CD and did an e2fsck - everything is fine. Just vzdump fails.

Anyone an Idea what to do?

Thanks
Luggi

pveversion
pve-manager/4.2-5/7cf09667 (running kernel: 4.4.8-1-pve)
root@vmint01:/mnt/ssd/dump# pveversion -v
proxmox-ve: 4.2-51 (running kernel: 4.4.8-1-pve)
pve-manager: 4.2-5 (running version: 4.2-5/7cf09667)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.8-1-pve: 4.4.8-51
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-39
qemu-server: 4.0-75
pve-firmware: 1.1-8
libpve-common-perl: 4.0-62
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-50
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-17
pve-container: 1.0-64
pve-firewall: 2.0-27
pve-ha-manager: 1.0-31
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve9~jessie