Failing HDD - Backup fails too - what to do?

[torontob]

I have a failing HDD and a new backup HDD was added but when trying to BACKUP all fails with errors. Some did backup and some didn't. These are LXC Containers. What can I do to save those LXC Containers that do not copy?

Thanks,




Task viewer: Backup

OutputStatus

Stop
INFO: starting new backup job: vzdump 20101 --node pve1 --mode stop --storage TB2 --compress lzo --remove 0
INFO: Starting Backup of VM 20101 (qemu)
INFO: status = stopped
INFO: update VM 20101: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/2tb/dump/vzdump-qemu-20101-2019_10_09-22_55_17.vma.lzo'
INFO: starting kvm to execute backup task
Running as unit 20101.scope.
INFO: started backup task 'b18afce9-67c6-4ea2-8821-c3f6218263eb'
INFO: status: 0% (149159936/107374182400), sparse 0% (103575552), duration 3, 49/15 MB/s
INFO: status: 1% (1147076608/107374182400), sparse 0% (1000132608), duration 21, 55/5 MB/s
INFO: status: 2% (2169044992/107374182400), sparse 1% (2022035456), duration 33, 85/0 MB/s
INFO: status: 3% (3311730688/107374182400), sparse 2% (3158122496), duration 46, 87/0 MB/s
INFO: status: 4% (4321312768/107374182400), sparse 3% (4160315392), duration 57, 91/0 MB/s
INFO: status: 5% (5378605056/107374182400), sparse 4% (5217570816), duration 75, 58/0 MB/s
INFO: status: 6% (6522667008/107374182400), sparse 5% (6361600000), duration 94, 60/0 MB/s
INFO: status: 7% (7576748032/107374182400), sparse 6% (7415615488), duration 106, 87/0 MB/s
INFO: status: 8% (8686862336/107374182400), sparse 7% (8525697024), duration 125, 58/0 MB/s
INFO: status: 9% (9719578624/107374182400), sparse 8% (9558380544), duration 139, 73/0 MB/s
INFO: status: 10% (10805116928/107374182400), sparse 9% (10643886080), duration 156, 63/0 MB/s
INFO: status: 11% (11896094720/107374182400), sparse 10% (11734798336), duration 168, 90/0 MB/s
INFO: status: 12% (12927827968/107374182400), sparse 11% (12766498816), duration 186, 57/0 MB/s
INFO: status: 13% (14043971584/107374182400), sparse 12% (13882609664), duration 200, 79/0 MB/s
INFO: status: 14% (15131869184/107374182400), sparse 13% (14970474496), duration 217, 63/0 MB/s
INFO: status: 15% (16199647232/107374182400), sparse 14% (16038219776), duration 230, 82/0 MB/s
INFO: status: 16% (17203593216/107374182400), sparse 15% (16907452416), duration 244, 71/9 MB/s
INFO: status: 17% (18313904128/107374182400), sparse 16% (18017730560), duration 265, 52/0 MB/s
INFO: status: 18% (19337052160/107374182400), sparse 17% (19040845824), duration 277, 85/0 MB/s
INFO: status: 19% (20459159552/107374182400), sparse 18% (20162920448), duration 297, 56/0 MB/s
INFO: status: 20% (21489057792/107374182400), sparse 19% (21192785920), duration 312, 68/0 MB/s
INFO: status: 21% (22575120384/107374182400), sparse 20% (22278815744), duration 329, 63/0 MB/s
INFO: status: 22% (23644209152/107374182400), sparse 21% (23347871744), duration 343, 76/0 MB/s
INFO: status: 23% (24761663488/107374182400), sparse 22% (24465293312), duration 374, 36/0 MB/s
INFO: status: 24% (25770852352/107374182400), sparse 23% (25474437120), duration 414, 25/0 MB/s
INFO: status: 25% (26918191104/107374182400), sparse 24% (26621743104), duration 431, 67/0 MB/s
INFO: status: 26% (27921809408/107374182400), sparse 25% (27625328640), duration 448, 59/0 MB/s
INFO: status: 27% (29007413248/107374182400), sparse 26% (28710899712), duration 460, 90/0 MB/s
INFO: status: 28% (30110121984/107374182400), sparse 27% (29813575680), duration 473, 84/0 MB/s
INFO: status: 29% (31234654208/107374182400), sparse 28% (30937903104), duration 492, 59/0 MB/s
INFO: status: 30% (32263897088/107374182400), sparse 29% (31967113216), duration 511, 54/0 MB/s
INFO: status: 31% (33352515584/107374182400), sparse 30% (33055666176), duration 528, 64/0 MB/s
INFO: status: 32% (34381103104/107374182400), sparse 31% (34084220928), duration 549, 48/0 MB/s
INFO: status: 33% (35527196672/107374182400), sparse 32% (35230236672), duration 562, 88/0 MB/s
INFO: status: 34% (36599300096/107374182400), sparse 33% (36302286848), duration 575, 82/0 MB/s
INFO: status: 35% (37621727232/107374182400), sparse 34% (37324681216), duration 587, 85/0 MB/s
INFO: status: 36% (38734594048/107374182400), sparse 35% (38437515264), duration 599, 92/0 MB/s
INFO: status: 37% (39748370432/107374182400), sparse 36% (39450783744), duration 623, 42/0 MB/s
INFO: status: 38% (40853176320/107374182400), sparse 37% (40555290624), duration 636, 84/0 MB/s
INFO: status: 39% (41886547968/107374182400), sparse 38% (41588629504), duration 650, 73/0 MB/s
INFO: status: 40% (43056168960/107374182400), sparse 39% (42758217728), duration 667, 68/0 MB/s
INFO: status: 41% (44072763392/107374182400), sparse 40% (43774779392), duration 684, 59/0 MB/s
INFO: status: 42% (45112557568/107374182400), sparse 41% (44814540800), duration 699, 69/0 MB/s
INFO: status: 43% (46204977152/107374182400), sparse 42% (45906894848), duration 717, 60/0 MB/s
INFO: status: 43% (46897692672/107374182400), sparse 43% (46537363456), duration 747, 23/2 MB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 20101 failed - job failed with err -5 - Input/output error
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[Alwin]

I have a failing HDD and a new backup HDD

As your HDD is failing, some part of it is already damaged. Sadly it happens to be on the part where your CTs/VMs are. vzdump reads every block of the VM to get a backup, it fails to read blocks and gives you the input/output error. As creating a backup doesn't work, try to run an imaging software (eg. clonezilla) that can ignore failed blocks. This will give you at least some of the data, but most likely you will need to re-install the failed VMs anyhow and restore missing or corrupt data from backup.

 

[torontob]

Clonezilla installs on Proxmox mother one?

So it's safe to say that data lost is unrecoverable for good and forget about trying to recover since Proxmox is ultimate source for backing and since it fails there is no other alternative?

 

[Alwin]

Clonezilla installs on Proxmox mother one?

Clonezilla runs as a live CD/USB stick, no install needed.

So it's safe to say that data lost is unrecoverable for good and forget about trying to recover since Proxmox is ultimate source for backing and since it fails there is no other alternative?

Clonezilla can create an image of the disk in question (skipping bad blocks) and from there you can try to recover as much data as possible.