Backup error again

[lumox]

Hi,
I was trying to backup one of my VMs created in Proxmox, but I got this error message:

INFO: starting new backup job: vzdump 100 --compress zstd --node pve --storage ISOs --remove 0 --mode snapshot
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-06-17 19:48:28
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: OMV5
INFO: include disk 'scsi0' 'VM:vm-100-disk-0' 32G
INFO: include disk 'scsi1' 'VM:vm-100-disk-1' 50G
INFO: creating archive '/mnt/pve/ISOs/dump/vzdump-qemu-100-2020_06_17-19_48_28.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '45ac8f3c-7abc-47b9-bb06-fc2f40289b8a'
INFO: status: 0% (181075968/88046829568), sparse 0% (44548096), duration 3, read/write 60/45 MB/s
INFO: status: 1% (927465472/88046829568), sparse 0% (54583296), duration 20, read/write 43/43 MB/s
INFO: status: 2% (1801650176/88046829568), sparse 0% (92299264), duration 40, read/write 43/41 MB/s
INFO: status: 3% (2672164864/88046829568), sparse 0% (138153984), duration 62, read/write 39/37 MB/s
INFO: status: 4% (3554017280/88046829568), sparse 0% (139894784), duration 84, read/write 40/40 MB/s
INFO: status: 5% (4421779456/88046829568), sparse 0% (176861184), duration 104, read/write 43/41 MB/s
INFO: status: 6% (5283512320/88046829568), sparse 0% (179769344), duration 126, read/write 39/39 MB/s
INFO: status: 7% (6182928384/88046829568), sparse 0% (181485568), duration 148, read/write 40/40 MB/s
INFO: status: 8% (7052001280/88046829568), sparse 0% (215744512), duration 168, read/write 43/41 MB/s
INFO: status: 9% (7967080448/88046829568), sparse 0% (218370048), duration 189, read/write 43/43 MB/s
INFO: status: 10% (8829927424/88046829568), sparse 0% (252628992), duration 211, read/write 39/37 MB/s
INFO: status: 11% (9723183104/88046829568), sparse 0% (282165248), duration 230, read/write 47/45 MB/s
INFO: status: 12% (10573119488/88046829568), sparse 0% (300232704), duration 249, read/write 44/43 MB/s
INFO: status: 13% (11483086848/88046829568), sparse 0% (335122432), duration 269, read/write 45/43 MB/s
INFO: status: 14% (12342132736/88046829568), sparse 0% (344285184), duration 292, read/write 37/36 MB/s
INFO: status: 15% (13248561152/88046829568), sparse 0% (379813888), duration 314, read/write 41/39 MB/s
INFO: status: 16% (14090633216/88046829568), sparse 0% (394428416), duration 335, read/write 40/39 MB/s
INFO: status: 17% (14991491072/88046829568), sparse 0% (394440704), duration 356, read/write 42/42 MB/s
INFO: status: 18% (15849226240/88046829568), sparse 0% (429092864), duration 375, read/write 45/43 MB/s
INFO: status: 19% (16755195904/88046829568), sparse 0% (441503744), duration 396, read/write 43/42 MB/s
INFO: status: 20% (17619615744/88046829568), sparse 0% (482037760), duration 414, read/write 48/45 MB/s
INFO: status: 21% (18515099648/88046829568), sparse 0% (493228032), duration 433, read/write 47/46 MB/s
INFO: status: 22% (19412156416/88046829568), sparse 0% (537645056), duration 453, read/write 44/42 MB/s
INFO: status: 23% (20303380480/88046829568), sparse 0% (551505920), duration 470, read/write 52/51 MB/s
INFO: status: 24% (21158690816/88046829568), sparse 0% (559722496), duration 489, read/write 45/44 MB/s
INFO: status: 25% (22015901696/88046829568), sparse 0% (602918912), duration 506, read/write 50/47 MB/s
INFO: status: 26% (22912958464/88046829568), sparse 0% (614809600), duration 528, read/write 40/40 MB/s
INFO: status: 27% (23813816320/88046829568), sparse 0% (651984896), duration 547, read/write 47/45 MB/s
INFO: status: 28% (24682037248/88046829568), sparse 0% (660799488), duration 566, read/write 45/45 MB/s
INFO: status: 29% (25539510272/88046829568), sparse 0% (665915392), duration 586, read/write 42/42 MB/s
INFO: status: 30% (26459373568/88046829568), sparse 0% (701595648), duration 606, read/write 45/44 MB/s
INFO: status: 31% (27299414016/88046829568), sparse 0% (715202560), duration 624, read/write 46/45 MB/s
INFO: status: 32% (28194504704/88046829568), sparse 0% (763826176), duration 644, read/write 44/42 MB/s
INFO: status: 33% (29062791168/88046829568), sparse 0% (769404928), duration 663, read/write 45/45 MB/s
INFO: status: 34% (29941170176/88046829568), sparse 0% (769474560), duration 680, read/write 51/51 MB/s
INFO: status: 35% (30855331840/88046829568), sparse 0% (803037184), duration 698, read/write 50/48 MB/s
INFO: status: 36% (31719292928/88046829568), sparse 0% (803205120), duration 716, read/write 47/47 MB/s
INFO: status: 37% (32597409792/88046829568), sparse 0% (803274752), duration 733, read/write 51/51 MB/s
INFO: status: 38% (33502789632/88046829568), sparse 0% (804073472), duration 751, read/write 50/50 MB/s
INFO: status: 39% (34360066048/88046829568), sparse 0% (804667392), duration 770, read/write 45/45 MB/s
INFO: status: 40% (35228483584/88046829568), sparse 0% (840523776), duration 787, read/write 51/48 MB/s
INFO: status: 41% (36102733824/88046829568), sparse 0% (854581248), duration 805, read/write 48/47 MB/s
INFO: status: 42% (37028495360/88046829568), sparse 1% (892211200), duration 824, read/write 48/46 MB/s
INFO: status: 43% (37904449536/88046829568), sparse 1% (905076736), duration 844, read/write 43/43 MB/s
INFO: status: 44% (38786301952/88046829568), sparse 1% (938631168), duration 863, read/write 46/44 MB/s
INFO: status: 45% (39622541312/88046829568), sparse 1% (939470848), duration 880, read/write 49/49 MB/s
INFO: status: 46% (40546467840/88046829568), sparse 1% (947122176), duration 900, read/write 46/45 MB/s
INFO: status: 47% (41419603968/88046829568), sparse 1% (983601152), duration 916, read/write 54/52 MB/s
INFO: status: 48% (42294706176/88046829568), sparse 1% (985812992), duration 932, read/write 54/54 MB/s
INFO: status: 49% (43157553152/88046829568), sparse 1% (1023172608), duration 948, read/write 53/51 MB/s
INFO: status: 50% (44074860544/88046829568), sparse 1% (1027837952), duration 965, read/write 53/53 MB/s
INFO: status: 51% (44921257984/88046829568), sparse 1% (1028136960), duration 985, read/write 42/42 MB/s
INFO: status: 52% (45803110400/88046829568), sparse 1% (1061691392), duration 1002, read/write 51/49 MB/s
INFO: status: 53% (46719172608/88046829568), sparse 1% (1061691392), duration 1019, read/write 53/53 MB/s
INFO: status: 54% (47582019584/88046829568), sparse 1% (1095245824), duration 1036, read/write 50/48 MB/s
INFO: status: 55% (48463872000/88046829568), sparse 1% (1095544832), duration 1054, read/write 48/48 MB/s
INFO: status: 56% (49334321152/88046829568), sparse 1% (1099706368), duration 1075, read/write 41/41 MB/s
INFO: status: 57% (50219974656/88046829568), sparse 1% (1165119488), duration 1094, read/write 46/43 MB/s
INFO: status: 58% (51086622720/88046829568), sparse 1% (1176358912), duration 1116, read/write 39/38 MB/s
INFO: status: 59% (51967688704/88046829568), sparse 1% (1298055168), duration 1135, read/write 46/39 MB/s
INFO: status: 60% (52850327552/88046829568), sparse 1% (1300140032), duration 1155, read/write 44/44 MB/s
INFO: status: 61% (53709373440/88046829568), sparse 1% (1329377280), duration 1174, read/write 45/43 MB/s
INFO: status: 62% (54621634560/88046829568), sparse 1% (1342578688), duration 1193, read/write 48/47 MB/s
INFO: status: 63% (55485530112/88046829568), sparse 1% (1359831040), duration 1212, read/write 45/44 MB/s
INFO: status: 64% (56381538304/88046829568), sparse 1% (1476874240), duration 1233, read/write 42/37 MB/s
INFO: status: 65% (57257558016/88046829568), sparse 1% (1590087680), duration 1251, read/write 48/42 MB/s
INFO: status: 66% (58137640960/88046829568), sparse 1% (1624375296), duration 1270, read/write 46/44 MB/s
INFO: status: 66% (58814365696/88046829568), sparse 1% (1701822464), duration 1292, read/write 30/27 MB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 100 failed - job failed with err -5 - Input/output error
INFO: Failed at 2020-06-17 20:10:08
INFO: Backup job finished with errors
TASK ERROR: job errors

I had no problem last time I did it.
What happened?
Thanks

 

[Moayad]

Hi,

Is this problem with all VMs or just VMID 100?

EDIT:

ERROR: Backup of VM 100 failed - job failed with err -5 - Input/output error

This error most often points to a faulty storage/disk

check if all VMs so and try backup on a different storage

 

[lumox]

Hi,

Is this problem with all VMs or just VMID 100?

EDIT:


This error most often points to a faulty storage/disk

check if all VMs so and try backup on a different storage

I ran a backup of another VM and it went smoothly, and as I already said, I did it for the same VM too a week ago, and everything went ok on the same storage. Strange
Thanks

 

[Moayad]

Please post config of VM 100 and cat /etc/pve/storage.cfg as well.

 

[lumox]

Please post config of VM 100 and cat /etc/pve/storage.cfg as well.

root@pve:/etc/pve# cat storage.cfg
dir: local
        path /var/lib/vz
        content iso,backup,vztmpl

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

lvm: VM
        vgname VM
        content rootdir,images
        nodes pve
        shared 0

dir: ISOs
        path /mnt/pve/ISOs
        content rootdir,images,iso,snippets,vztmpl,backup
        is_mountpoint 1
        maxfiles 2
        nodes pve
        shared 0

and 100 VM config:

root@pve:/etc/pve# qm config 100
bootdisk: scsi0
cores: 1
ide2: none,media=cdrom
memory: 2048
name: OMV5
net0: virtio=AE:DD:62:6D:96:BD,bridge=vmbr1
numa: 0
ostype: l26
scsi0: VM:vm-100-disk-0,size=32G
scsi1: VM:vm-100-disk-1,size=50G
scsihw: virtio-scsi-pci
smbios1: uuid=e1709470-a622-4425-97b3-e6b2be85d451
sockets: 1
vmgenid: e7d0d8b3-f55a-41b3-92e2-e0c4295c32b3

Thank you

 

[Moayad]

Hi,

Please post output of pveversion -v

I recommend check the health of your disk using smartctl or you can do that from GUI go to Node -> Disks, also check the syslog/journal or dmesg for IO errors on specific devices during the Backup time

 

[lumox]

Hi,

Please post output of pveversion -v

I recommend check the health of your disk using smartctl or you can do that from GUI go to Node -> Disks, also check the syslog/journal or dmesg for IO errors on specific devices during the Backup time

here it is:

root@pve:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-1
pve-kernel-helper: 6.2-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.3
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-5
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Thanks

 

[Moayad]

Hi,

Your Proxmox VE versions looks up to date. So as i said check your Disks

 ERROR: Backup of VM 100 failed - job failed with err -5 - Input/output error

because this error points to a faulty storage/disk.

 

[lumox]

 ERROR: Backup of VM 100 failed - job failed with err -5 - Input/output error

because this error points to a faulty storage/disk.

Yes, you are most probably right.
Thank you very much