[SOLVED] Read error at byte 0, while reading X bytes: Input/output error

R

Razva

Renowned Member
Dec 3, 2013
250
8
83
Romania
cncted.com
Hey,

Last night's backup posted this error:
Code: 
INFO: starting new backup job: vzdump 101 100 --quiet 1 --mode snapshot --compress gzip --storage pmx1-web1 --node mh-pmx1
INFO: filesystem type on dumpdir is 'fuse.sshfs' -using /var/tmp/vzdumptmp2445 for temporary files
INFO: Starting Backup of VM 100 (openvz)
INFO: CTID 100 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-mh-pmx1-0')
INFO:   Logical volume "vzsnap-mh-pmx1-0" created
INFO: creating archive '/nfs/web1/dump/vzdump-openvz-100-2014_11_12-02_00_02.tar.gz'
INFO: Total bytes written: 35348459520 (33GiB, 20MiB/s)
INFO: archive file size: 19.47GB
INFO: Finished Backup of VM 100 (00:29:38)
INFO: filesystem type on dumpdir is 'fuse.sshfs' -using /var/tmp/vzdumptmp2445 for temporary files
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-mh-pmx1-0')
INFO:   Logical volume "vzsnap-mh-pmx1-0" created
INFO: creating archive '/nfs/web1/dump/vzdump-openvz-101-2014_11_12-02_29_40.tar.gz'
INFO: tar: ./home/mysite/descarca/revista_2.pdf: File shrank by 5920717 bytes; padding with zeros
INFO: tar: ./home/mysite/descarca/revista_3.pdf: Read error at byte 0, while reading 10240 bytes: Input/output error
INFO: tar: ./home/mysite/site/: Cannot savedir: Input/output error
INFO: tar: ./home/mysite/site/phpinfo.php: Read error at byte 0, while reading 19 bytes: Input/output error
INFO: tar: ./home/mysite/css/: Cannot savedir: Input/output error
INFO: tar: ./home/mysite/css/sequencejs-theme.modern-slide-in.css: Read error at byte 0, while reading 3584 bytes: Input/output error
INFO: tar: ./home/mysite/css/jquery.fancybox.css: Read error at byte 0, while reading 8192 bytes: Input/output error
INFO: tar: ./home/mysite/css/fancybox-y.png: Read error at byte 0, while reading 176 bytes: Input/output error
INFO: tar: ./home/mysite/css/sequencejs-theme.modern-slide-in.ie.css: Read error at byte 0, while reading 29 bytes: Input/output error
INFO: tar: ./home/mysite/css/style.css: Read error at byte 0, while reading 7168 bytes: Input/output error
INFO: tar: ./home/mysite/css/fancybox.png: Read error at byte 0, while reading 5120 bytes: Input/output error
INFO: tar: ./home/mysite/css/normalize.css: Read error at byte 0, while reading 9004 bytes: Input/output error
INFO: tar: ./home/mysite/css/fancybox-x.png: Read error at byte 0, while reading 203 bytes: Input/output error
INFO: tar: ./home/mysite/downloads.php: Read error at byte 0, while reading 392 bytes: Input/output error
INFO: tar: ./home/mysite/search.php: Read error at byte 0, while reading 1586 bytes: Input/output error
INFO: tar: ./home/mysite/resetare-parola.php: Read error at byte 0, while reading 1823 bytes: Input/output error
INFO: tar: ./home/mysite/googlebbb5b9e4e5511985.html: Read error at byte 0, while reading 53 bytes: Input/output error
INFO: tar: ./home/mysite/blogroll.php: Read error at byte 0, while reading 2048 bytes: Input/output error
INFO: tar: ./home/mysite/mysql.php: Read error at byte 0, while reading 1327 bytes: Input/output error
INFO: tar: ./home/mysite/emails/: Cannot savedir: Input/output error
INFO: tar: ./home/mysite/emails/trimitere-factura.php: Read error at byte 0, while reading 206 bytes: Input/output error
INFO: tar: ./home/mysite/emails/factura.php: Read error at byte 0, while reading 5632 bytes: Input/output error
INFO: tar: ./home/mysite/emails/lost-password.php: Read error at byte 0, while reading 141 bytes: Input/output error
INFO: tar: ./home/mysite/emails/contact.php: Read error at byte 0, while reading 370 bytes: Input/output error
INFO: tar: ./home/mysite/emails/comanda-online-aprobata.php: Read error at byte 0, while reading 353 bytes: Input/output error
INFO: tar: ./home/mysite/epayment/: Cannot savedir: Input/output error
INFO: tar: ./home/mysite/epayment/function.php: Read error at byte 0, while reading 512 bytes: Input/output error
INFO: tar: ./home/mysite/epayment/pay.php: Read error at byte 0, while reading 4840 bytes: Input/output error
INFO: tar: ./home/mysite/epayment/ep_reply.php: Read error at byte 0, while reading 2357 bytes: Input/output error
INFO: tar: ./home/mysite/nws.php: Read error at byte 0, while reading 284 bytes: Input/output error
INFO: tar: ./home/mysite/plata.php: Read error at byte 0, while reading 370 bytes: Input/output error
INFO: tar: ./home/mysite/comanda.php: Read error at byte 0, while reading 5258 bytes: Input/output error
INFO: tar: ./home/mysite/articol.php: Read error at byte 0, while reading 2048 bytes: Input/output error
...with "Read error at byte 0..." repeating for every other file on *all* CTs.

Please note that:

  • before using GZIP we've used LZO for backups and had no issues
  • the NFS share is mounted and fully functional
  • node and attached NFS have enough space
  • node and attached NFS use new SSDs and have 0 corrupted sectors
Can you please give me a hint?

Thanks,
Razvan
 
The log ends with this:
INFO: tar: ./sys/: Cannot savedir: Input/output error
INFO: Total bytes written: 77390366720 (73GiB, 18MiB/s)
INFO: tar: Exiting with failure status due to previous errors
ERROR: Backup of VM 101 failed - command '(cd /mnt/vzsnap0/private/101;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system --null -T -|gzip) >/nfs/web1/dump/vzdump-openvz-101-2014_11_12-02_29_40.tar.dat' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors
Click to expand...
 
Any hint in /var/log/syslog why sending that backup mail failed?

Those Input/output error usually indicates that you run out of LVM snapshot space, see

# man vzdump

related parameter is called '--size'
 
Here's the log: http://pastebin.com/AXH2ytJU

I've separated the backup part from the other info (search for ==========).

I can see these errors repeating:
BUG: Quota files for 101 are broken: no quota engine running
EXT4-fs (dm-3): ext4_orphan_cleanup: deleting unreferenced inode X
VZDQ: Tried to clean orphans on qmblk with 1 state
Click to expand...

And after that:
Buffer I/O error on device dm-3, logical block 52428918
lost page write due to I/O error on dm-3
Click to expand...
It may be more so please be so kind and check the log.

Any idea how can I fix it?

Thank you very much for your time!
 
I've read the vzdump manual but it's not so "explicit":
-size integer (500 - N)
LVM snapshot size im MB.
Click to expand...
So this is the maximum size of a snapshot?

Thinking at the fact that I have no idea what size really represents, I've set "size: 204800" in /etc/vzdump.conf (max. CT on the node has ~200gb).

Any solution please?

Later edit: this might be useful?

root@mh-pmx1:~# pvdisplay
--- Physical volume ---
PV Name /dev/sda3
VG Name pve
PV Size 1.82 TiB / not usable 3.98 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 476543
Free PE 4095
Allocated PE 472448
Click to expand...
Thanks!
 
Last edited:
root@mh-pmx1:~# vgdisplay
--- Volume group ---
VG Name pve
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 20
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.82 TiB
PE Size 4.00 MiB
Total PE 476543
Alloc PE / Size 472448 / 1.80 TiB
Free PE / Size 4095 / 16.00 GiB
VG UUID 0XcFaC-tYIl-6nSh-7gWm-sger-ilXG-7laPfT
Click to expand...
Tonight's backup error is this:
INFO: starting new backup job: vzdump 101 100 --quiet 1 --mode snapshot --compress gzip --storage pmx1-web1 --node mh-pmx1
INFO: filesystem type on dumpdir is 'fuse.sshfs' -using /var/tmp/vzdumptmp570529 for temporary files
INFO: Starting Backup of VM 100 (openvz)
INFO: CTID 100 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-mh-pmx1-0')
INFO: Volume group "pve" has insufficient free space (4095 extents): 51200 required.
ERROR: Backup of VM 100 failed - command 'lvcreate --size 204800M --snapshot --name vzsnap-mh-pmx1-0 /dev/pve/data' failed: exit code 5
INFO: filesystem type on dumpdir is 'fuse.sshfs' -using /var/tmp/vzdumptmp570529 for temporary files
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-mh-pmx1-0')
INFO: Volume group "pve" has insufficient free space (4095 extents): 51200 required.
ERROR: Backup of VM 101 failed - command 'lvcreate --size 204800M --snapshot --name vzsnap-mh-pmx1-0 /dev/pve/data' failed: exit code 5
INFO: Backup job finished with errors
TASK ERROR: job errors
Click to expand...
So I suppose it's something related to free space...?

Thanks!
 
Understood. Well I didn't really touched anything when installing Proxmox, so the partitioning is "default".

Is there any documentation regarding VG? I've used "8192" and seems to work for some small CTs (I'll wait till tonight to make backups of the bigger ones). Is this value too low, thinking at the fact that the node has a CTs with sizes that vary between 10gb - 200gb?

Thanks!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back