Backup failure

K

k0nsl

Member
Nov 17, 2013
15
0
21
Denmark, Slangerup
k0nsl.org
I've tried to backup one KVM for several days and it always end bad with a log entry such as the following:

Code: 
Nov 02 10:25:54 INFO: Starting Backup of VM 104 (qemu)
Nov 02 10:25:54 INFO: status = running
Nov 02 10:25:55 INFO: update VM 104: -lock backup
Nov 02 10:25:55 INFO: VM Name: srv51-kvm
Nov 02 10:25:55 INFO: include disk 'virtio0' 'storage:vm-104-disk-1'
Nov 02 10:25:55 INFO: include disk 'virtio1' 'storage:vm-104-disk-2'
Nov 02 10:25:55 INFO: backup mode: snapshot
Nov 02 10:25:55 INFO: ionice priority: 7
Nov 02 10:25:55 INFO: pending configuration changes found (not included into backup)
Nov 02 10:25:56 INFO: creating archive '/mnt/pve/Gluster01/dump/vzdump-qemu-104-2016_11_02-10_25_54.vma.gz'
Nov 02 10:25:56 INFO: started backup task '38c0ef86-ae59-41c7-aa85-c1131c6da678'
Nov 02 10:25:59 INFO: status: 0% (254672896/1395864371200), sparse 0% (38039552), duration 3, 84/72 MB/s
Nov 02 10:28:02 INFO: status: 1% (14035976192/1395864371200), sparse 0% (2577686528), duration 126, 112/91 MB/s
Nov 02 10:31:21 INFO: status: 2% (27961065472/1395864371200), sparse 0% (3589496832), duration 325, 69/64 MB/s
Nov 02 10:34:15 INFO: status: 3% (41905618944/1395864371200), sparse 0% (4800688128), duration 499, 80/73 MB/s
Nov 02 10:37:07 INFO: status: 4% (55902601216/1395864371200), sparse 0% (7773581312), duration 671, 81/64 MB/s
Nov 02 10:40:23 INFO: status: 5% (69834375168/1395864371200), sparse 0% (8411758592), duration 867, 71/67 MB/s
Nov 02 10:43:47 INFO: status: 6% (83801210880/1395864371200), sparse 0% (9264824320), duration 1071, 68/64 MB/s
Nov 02 10:46:54 INFO: status: 7% (97748975616/1395864371200), sparse 0% (9663406080), duration 1258, 74/72 MB/s
Nov 02 10:49:20 INFO: status: 8% (111721578496/1395864371200), sparse 0% (9895198720), duration 1404, 95/94 MB/s
Nov 02 10:52:31 INFO: status: 9% (125656825856/1395864371200), sparse 0% (10142330880), duration 1595, 72/71 MB/s
Nov 02 10:57:45 INFO: status: 10% (139674779648/1395864371200), sparse 0% (10558214144), duration 1909, 44/43 MB/s
Nov 02 11:00:14 INFO: status: 11% (153552814080/1395864371200), sparse 0% (11356491776), duration 2058, 93/87 MB/s
Nov 02 11:02:55 INFO: status: 12% (167548616704/1395864371200), sparse 0% (12643295232), duration 2219, 86/78 MB/s
Nov 02 11:05:44 INFO: status: 13% (181480194048/1395864371200), sparse 1% (14067425280), duration 2388, 82/74 MB/s
Nov 02 11:09:00 INFO: status: 14% (195468787712/1395864371200), sparse 1% (14918557696), duration 2584, 71/67 MB/s
Nov 02 11:12:50 INFO: status: 15% (209425661952/1395864371200), sparse 1% (15529914368), duration 2814, 60/58 MB/s
Nov 02 11:15:32 INFO: status: 16% (223360843776/1395864371200), sparse 1% (15826960384), duration 2976, 86/84 MB/s
Nov 02 11:16:12 INFO: status: 16% (225911373824/1395864371200), sparse 1% (15860338688), duration 3016, 63/62 MB/s
Nov 02 11:16:12 ERROR: job failed with err -5 - Input/output error
Nov 02 11:16:12 INFO: aborting backup job
Nov 02 11:16:17 ERROR: Backup of VM 104 failed - job failed with err -5 - Input/output error

There is 1.8T available in backup location so I don't understand what the problem could be. I never had this problem before.

Any ideas? I'm running latest version available in the "non-subscription" repository. The output of "pveperf" shows:
Code: 
root@edelweiss:~# pveperf
CPU BOGOMIPS:      54397.20
REGEX/SECOND:      2470805
HD SIZE:           94.37 GB (/dev/dm-0)
BUFFERED READS:    19.89 MB/sec
AVERAGE SEEK TIME: 20.02 ms
FSYNCS/SECOND:     22.68
DNS EXT:           211.67 ms
DNS INT:           142.13 ms (k0nsl.org)

Thanks in advance for any hints on how to solve this. I really need to get this backed up, ASAP.

Best wishes,
-k0nsl
 
I have figured out why it fails. The disk is rapidly failing.

Here are some additional information:

Code: 
root@edelweiss:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               pve
  PV Size               1.82 TiB / not usable 1.07 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              476900
  Free PE               4039
  Allocated PE          472861
  PV UUID               QVyybl-qwjB-QTJC-f8gj-akbz-KpYC-zS5aYK
   
root@edelweiss:~# vgdisplay
  --- Volume group ---
  VG Name               pve
  System ID            
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  39
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                9
  Open LV               4
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TiB
  PE Size               4.00 MiB
  Total PE              476900
  Alloc PE / Size       472861 / 1.80 TiB
  Free  PE / Size       4039 / 15.78 GiB
  VG UUID               RFhNxU-uj1x-D0Pi-EAxJ-T74u-ovCg-RBuCTM
   
root@edelweiss:~# lvdisplay
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                DTG56N-L30R-Xqlb-wukp-V6fb-1y8O-vYA3e8
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-09-20 22:15:15 -0900
  LV Status              available
  # open                 2
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:1
   
  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                GDwdZa-mOHR-vVQn-0Bsf-ePlp-5yL3-ygX91o
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-09-20 22:15:16 -0900
  LV Status              available
  # open                 1
  LV Size                96.00 GiB
  Current LE             24576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:0
   
  --- Logical volume ---
  LV Name                data
  VG Name                pve
  LV UUID                x0T9HD-6htn-B4Eg-KB5Z-rmsd-GQH7-5kcOLy
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-09-20 22:15:17 -0900
  LV Pool metadata       data_tmeta
  LV Pool data           data_tdata
  LV Status              available
  # open                 7
  LV Size                1.70 TiB
  Allocated pool data    55.06%
  Allocated metadata     26.63%
  Current LE             446181
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:4
   
  --- Logical volume ---
  LV Path                /dev/pve/vm-100-disk-1
  LV Name                vm-100-disk-1
  VG Name                pve
  LV UUID                gdxGOM-Pcsa-Suf4-66Y3-fp6H-bz9H-GErG36
  LV Write Access        read/write
  LV Creation host, time edelweiss, 2016-09-21 13:11:06 -0900
  LV Pool name           data
  LV Status              available
  # open                 0
  LV Size                45.00 GiB
  Mapped size            100.00%
  Current LE             11520
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:6
   
  --- Logical volume ---
  LV Path                /dev/pve/vm-102-disk-1
  LV Name                vm-102-disk-1
  VG Name                pve
  LV UUID                dVbn0I-owWP-F3pL-5Nsy-OlTH-b0F9-QDjwHH
  LV Write Access        read/write
  LV Creation host, time edelweiss, 2016-09-22 05:32:47 -0900
  LV Pool name           data
  LV Status              available
  # open                 0
  LV Size                10.00 GiB
  Mapped size            42.27%
  Current LE             2560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:7
   
  --- Logical volume ---
  LV Path                /dev/pve/vm-103-disk-1
  LV Name                vm-103-disk-1
  VG Name                pve
  LV UUID                jUxeOI-XCzT-aDf1-0Lej-mJ69-kAdy-sRC9FX
  LV Write Access        read/write
  LV Creation host, time edelweiss, 2016-09-23 23:56:20 -0900
  LV Pool name           data
  LV Status              available
  # open                 0
  LV Size                10.00 GiB
  Mapped size            25.68%
  Current LE             2560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:8
   
  --- Logical volume ---
  LV Path                /dev/pve/vm-104-disk-1
  LV Name                vm-104-disk-1
  VG Name                pve
  LV UUID                ZussfF-8edD-fthO-DA2O-xr9J-HOOD-Wc0TIe
  LV Write Access        read/write
  LV Creation host, time edelweiss, 2016-09-28 13:04:18 -0900
  LV Pool name           data
  LV Status              available
  # open                 1
  LV Size                800.00 GiB
  Mapped size            92.25%
  Current LE             204800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:9
   
  --- Logical volume ---
  LV Path                /dev/pve/vm-104-disk-2
  LV Name                vm-104-disk-2
  VG Name                pve
  LV UUID                9b71NQ-ZNI2-t5Jx-d9pY-DC9e-U1z3-1JGFgm
  LV Write Access        read/write
  LV Creation host, time edelweiss, 2016-09-28 13:04:19 -0900
  LV Pool name           data
  LV Status              available
  # open                 1
  LV Size                500.00 GiB
  Mapped size            27.56%
  Current LE             128000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:10
   
  --- Logical volume ---
  LV Path                /dev/pve/vm-104-disk-3
  LV Name                vm-104-disk-3
  VG Name                pve
  LV UUID                Uxo23I-MJYj-AfNI-tTyO-PdZf-l9wU-cfMcuJ
  LV Write Access        read/write
  LV Creation host, time edelweiss, 2016-11-03 08:12:33 -0900
  LV Pool name           data
  LV Status              available
  # open                 0
  LV Size                500.00 GiB
  Mapped size            6.40%
  Current LE             128000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:11
   
root@edelweiss:~#

I have only two logical volumes on that disk which I care about and those are "/dev/pve/vm-104-disk-1" and "/dev/pve/vm-104-disk-2". Can somebody recommend a safe way to safe them? I have another disk mounted which is functioning and could be used for that purpose. But what is the best/safest way to do it?

Any help is appreciated.

Best wishes,
-k0nsl
 
You must log in or register to reply here.
Top Bottom