Intermittent ERROR: job failed with err -5 - Input/output error

[LooneyTunes]

Hi,
I have a backup job configured which I also run manually at times. Looking at the logs the past backups have failed as well. Sometimes one VM, sometimes a few. I have checked the SMARTs of the disk and wear is currently at 1% and no other indicators are shown. It is fairly new and works well otherwise as far as I can tell. The error I am getting is "ERROR: job failed with err -5 - Input/output error".

What can I do to diagnose? Verbose logging? The error here is all I get unfortunately. Server is not under any load what-so-ever, adequate ventilation, feels cool. Diskusage is about 48% full.

PVE running is fully updated v 7.1-10. Any ideas?

Thanks

 

[mira]

Please provide the output of pveversion -v, qm config <VMID>, cat /etc/pve/storage.cfg and the full task log of a backup that ran into this issue.

 

[LooneyTunes]

Hi,
this is the log of a failed one;

INFO: starting new backup job: vzdump --mode snapshot --storage local --node pve --compress zstd --prune-backups 'keep-last=3,keep-monthly=1' --mailnotification always --all 1
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-03-05 09:36:12
INFO: status = running
INFO: VM Name: ubuntu
INFO: include disk 'virtio0' 'local-lvm:vm-100-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-100-2022_03_05-09_36_12.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '2022608c-b0a1-44e2-83cc-67b615b36aa3'
INFO: resuming VM again
INFO:   2% (656.1 MiB of 32.0 GiB) in 3s, read: 218.7 MiB/s, write: 134.1 MiB/s
INFO:   3% (1.1 GiB of 32.0 GiB) in 6s, read: 171.6 MiB/s, write: 124.9 MiB/s
INFO:   5% (1.6 GiB of 32.0 GiB) in 9s, read: 164.3 MiB/s, write: 160.3 MiB/s
INFO:   6% (2.2 GiB of 32.0 GiB) in 12s, read: 180.0 MiB/s, write: 167.5 MiB/s
INFO:   8% (2.8 GiB of 32.0 GiB) in 15s, read: 216.3 MiB/s, write: 181.7 MiB/s
INFO:   9% (3.2 GiB of 32.0 GiB) in 18s, read: 139.5 MiB/s, write: 133.8 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - job failed with err -5 - Input/output error
INFO: Failed at 2022-03-05 09:36:31

pveversion -v

pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-7
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

qm config 100

agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 2
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=6.1.0,ctime=1642250390
name: ubuntu
net0: virtio=<mac removed>,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
protection: 1
scsihw: virtio-scsi-pci
smbios1: uuid=<id removed>
sockets: 1
virtio0: local-lvm:vm-100-disk-0,size=32G
vmgenid: <id removed>

cat /etc/pve/storage.cfg

dir: local
    path /var/lib/vz
    content iso,vztmpl,backup

lvmthin: local-lvm
    thinpool data
    vgname pve
    content images,rootdir

dir: local-USB
    path /mnt/usb-drive
    content backup
    prune-backups keep-last=3,keep-monthly=1
    shared 0

 

[mira]

How much space do you have available on root (/)? df -h
You create the backup in /var/lib/vz, could it be that it runs out of space?

 

[LooneyTunes]

How much space do you have available on root (/)? df -h
You create the backup in /var/lib/vz, could it be that it runs out of space?

This is what it looks like, should be plenty of space;

Filesystem            Size  Used Avail Use% Mounted on
udev                   16G     0   16G   0% /dev
tmpfs                 3.1G  1.2M  3.1G   1% /run
/dev/mapper/pve-root   94G   45G   45G  51% /
tmpfs                  16G   49M   16G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sda2             511M  328K  511M   1% /boot/efi
/dev/fuse             128M   28K  128M   1% /etc/pve
tmpfs                 3.1G     0  3.1G   0% /run/user/0

 

[mira]

Does it always fail at the exact same size?

What disk type and model do you use?

 

[LooneyTunes]

Does it always fail at the exact same size?

What disk type and model do you use?

Hi,
sorry for the delay, been a busy few days. Yes, when it fails, it always happens in the beginning of a backup, approx at 10%, so not the "exact" size, but close to. Disk in the machine is a "Samsung_SSD_870_EVO_500GB", connected through S-ATA.

For unknown reason the last set of backups have completed without errors. I'll keep monitor a just a little while longer before letting it go for now

 

[Brij Kishore]

Hi,
sorry for the delay, been a busy few days. Yes, when it fails, it always happens in the beginning of a backup, approx at 10%, so not the "exact" size, but close to. Disk in the machine is a "Samsung_SSD_870_EVO_500GB", connected through S-ATA.

For unknown reason the last set of backups have completed without errors. I'll keep monitor a just a little while longer before letting it go for now

I am facing same issue while backing up VM on "Samsung_SSD_870_EVO_500GB". Not sure what to do.