Error I/O during backup

[Lollo91]

Hi all,
I'm tring to perform a backup of all my VM with Proxmox Backup Server.
When I lunch it for every VM, I received following output:

INFO: Starting Backup of VM xx (qemu)
INFO: Backup started at 2024-05-01 05:22:23
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: xxxx
INFO: include disk 'scsi0' 'Storage1T:xxx/vm-xxx-disk-1.qcow2' 10G
INFO: creating vzdump archive '/mnt/xxxxxx/vzdump-qemu-xxx-2024_05_01-05_22_23.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '0650c3f4-a4eb-4c76-8e82-d7b1b86b63cb'
INFO: 16% (1.6 GiB of 10.0 GiB) in 3s, read: 549.7 MiB/s, write: 161.0 MiB/s
INFO: 16% (1.6 GiB of 10.0 GiB) in 4s, read: 2.0 MiB/s, write: 2.0 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM xxx failed - job failed with err -5 - Input/output error
INFO: Failed at 2024-05-01 05:22:28
INFO: Backup job finished with errors
TASK ERROR: job errors
So, I've tried to perform the backup on different destination and result is the same.
Looking the /var/log/syslog I've notice:

SErr 0x40000 action 0x0
ata3.00: irq_stat 0x40000008
ata3: SError: { CommWake }
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/80:88:80:3e:43/02:00:03:00:00/40 tag 17 ncq dma 327680 in
res 41/40:80:00:40:43/00:02:03:00:00/00 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: supports DRM functions and may not be fully accessible
ata3.00: supports DRM functions and may not be fully accessible
ata3.00: configured for UDMA/133
sd 2:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cm
sd 2:0:0:0: [sda] tag#17 Sense Key : Medium Error [current]
sd 2:0:0:0: [sda] tag#17 Add. Sense: Unrecovered read error - auto reallocate f
sd 2:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 03 43 3e 80 00 02 80 00
blk_update_request: I/O error, dev sda, sector 54738944 op 0x0READ) flags 0x0

This is the pveversion -v output:

proxmox-ve: 7.4-1 (running kernel: 5.15.149-1-pve)
pve-manager: 7.4-17 (running version: 7.4-17/513c62be)
pve-kernel-5.15: 7.4-12
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.149-1-pve: 5.15.149-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.3
libpve-apiclient-perl: 3.2-2
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.6-1
proxmox-backup-file-restore: 2.4.6-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+3
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.10-1
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-5
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.15-pve1

How can I fix it? It should be an hw problem? If yes, how can I perform a backup due to replace the hw if it's needed to replace it?

Thanks

 

[bbgeek17]

Your disk is failing. It can't be read, so a full backup is no longer possible.
You can try to "dd" to another disk while "ignoring errors". You may end up with a little data loss, or the file could be completely unusable.
You can try "ddrescue" for a bit more intelligent recovery.
There is a small chance that it's a cable or a controller, and moving the disk elsewhere may allow you to continue.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Lollo91]

Thank you so much. I was able to recover disk data using ddrescue!