Backup Container error input/output

[Trương Ngọc Ẩn]

Proxmox community,
I have error during backup a container.
PVE version is 3.1.3
I try to change size in /etc/vzdump.conf but not effect.
Here is my info:

root@prxmxlab4:~# pvdisplay
--- Physical volume ---
PV Name /dev/sda3
VG Name pve
PV Size 2.73 TiB / not usable 3.98 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 714411
Free PE 4095
Allocated PE 710316
PV UUID zxPPQk-OXOF-sDyW-m3kf-Hyph-Buoa-61PPjj

root@prxmxlab4:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 2.73t 16.00g

This is my log

INFO: starting new backup job: vzdump 3510 --remove 0 --mode snapshot --compress lzo --node prxmxlab4 --storage snapshot
INFO: Starting Backup of VM 3510 (openvz)
INFO: CTID 3510 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-prxmxlab4-0')
INFO: Logical volume "vzsnap-prxmxlab4-0" created
INFO: creating archive '/mnt/pve/snapshot/dump/vzdump-openvz-3510-2015_12_25-14_22_35.tar.lzo'
INFO: find: `./var/spool/clientmqueue/qft2CGj1M6009801': Input/output error
INFO: find: `./var/spool/clientmqueue/qft2CGj1M6009801': Input/output error
INFO: find: `./var/spool/clientmqueue/dft2CGo1At010781': Input/output error
INFO: find: `./var/spool/clientmqueue/qft2CH51Du013536': Input/output error
INFO: find: `./var/spool/clientmqueue/qft2CH71kD013934': Input/output error
INFO: find: `./var/spool/clientmqueue/qft2CH81w3014133': Input/output error
.......
NFO: tar: ./etc/csh.cshrc: Cannot stat: Input/output error
INFO: tar: ./etc/rc6.d: Cannot stat: Input/output error
INFO: tar: ./srv/: Cannot savedir: Input/output error
INFO: Total bytes written: 73976852480 (69GiB, 459MiB/s)
INFO: tar: Exiting with failure status due to previous errors
ERROR: Backup of VM 3510 failed - command '(cd /mnt/vzsnap0/private/3510;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system --null -T -|lzop) >/mnt/pve/snapshot/dump/vzdump-openvz-3510-2015_12_25-14_22_35.tar.dat' failed: exit code 2
INFO: Backup job finished with errors

 

[dietmar]

Looks you run out of LVM snapshot space? Please check using 'lvs' while backup is running.

 

[Trương Ngọc Ẩn]

Looks you run out of LVM snapshot space? Please check using 'lvs' while backup is running.

Hi Dietmar,
This is content of 'lvs' when backup is running:

root@prxmxlab4:~# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
data pve owi-aos-- 2.49t
root pve -wi-ao--- 96.00g
swap pve -wi-ao--- 125.00g
vzsnap-prxmxlab4-0 pve swi-aos-- 4.00g data 1.52

 

[dietmar]

vzsnap-prxmxlab4-0 pve swi-aos-- 4.00g data 1.52

Does data usage increase up to full 4GB during backup?

 

[Trương Ngọc Ẩn]

Does data usage increase up to full 4GB during backup?

Hi Dietmar,

The disk sized of VM is 100G, so backup at least is half of this size.
When backup process get about 20% data, the error log begin appear.

LV VG Attr LSize Pool Origin Data% Move Log Copy% C
onvert
data pve owi-aos-- 2.49t

root pve -wi-ao--- 96.00g

swap pve -wi-ao--- 125.00g

vzsnap-prxmxlab4-0 pve swi-aos-- 4.00g data 41.11

 

[dietmar]

Maybe the disk is damaged? Or do you use some kind of RAID?

 

[Trương Ngọc Ẩn]

Maybe the disk is damaged? Or do you use some kind of RAID?

Thank Dietmer,
I check some files from logs on this VM, and i can delete this file, error input/out put.
So, is this root cause that make bacup failed?

 

[dietmar]

and i can delete this file, error input/out put.
So, is this root cause that make bacup failed?

Seems so.

 

[Trương Ngọc Ẩn]

Seems so.

I can not delete these file. Maybe must run fsck, but in CT just has a /dev/simfs.
So, Could you please help me to run this command to fix this incident ?