Back up always fails on 42% single node only.

[marcus_ph]

hi sirs, maybe you could help me out,
after a power failure while i was backing up one of my node.
the node got locked, i unlock it using qm unlock vmid.
but after that, the back up always fails with error input output error.

my version is 4.2-2

i have tried to restart my proxmox and tried to back up serveral times already but same error once it reach 42%.
also, after the error the web gui goes down as well, so i have to reboot the system (although other nodes can be accessed).

 

[mira]

Please post the complete log of the backup.
You can't access the GUI on the node in question but all others in the cluster still work? Perhaps check your disk for errors.

 

[marcus_ph]

just have the problem backing up this node... the other nodes backup with out any problem


INFO: starting new backup job: vzdump 102 --storage mybackup --mode stop --remove 0 --node pve --compress lzo
INFO: Starting Backup of VM 102 (qemu)
INFO: status = running
INFO: update VM 102: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: stopping vm
INFO: creating archive '/mybackup/dump/vzdump-qemu-102-2019_02_26-00_19_29.vma.lzo'
INFO: starting kvm to execute backup task
Running as unit 102.scope.
INFO: started backup task '688a6a98-b851-484d-a579-059504f08e22'
INFO: resume VM
INFO: status: 0% (491716608/536870912000), sparse 0% (157876224), duration 3, 163/111 MB/s
INFO: status: 1% (5368971264/536870912000), sparse 0% (389910528), duration 301, 16/15 MB/s
INFO: status: 2% (10740236288/536870912000), sparse 0% (1019801600), duration 797, 10/9 MB/s
INFO: status: 3% (16140795904/536870912000), sparse 0% (1728925696), duration 1113, 17/14 MB/s
INFO: status: 4% (23183425536/536870912000), sparse 1% (8760840192), duration 1118, 1408/2 MB/s
INFO: status: 6% (32492093440/536870912000), sparse 3% (18069483520), duration 1121, 3102/0 MB/s
INFO: status: 7% (37730975744/536870912000), sparse 4% (23124226048), duration 1124, 1746/61 MB/s
INFO: status: 8% (44903432192/536870912000), sparse 5% (30147514368), duration 1129, 1434/29 MB/s
INFO: status: 10% (54270296064/536870912000), sparse 7% (39514222592), duration 1132, 3122/0 MB/s
INFO: status: 11% (62729355264/536870912000), sparse 8% (47972810752), duration 1135, 2819/0 MB/s
INFO: status: 13% (70354206720/536870912000), sparse 10% (55573848064), duration 1138, 2541/7 MB/s
INFO: status: 14% (76938674176/536870912000), sparse 11% (61972398080), duration 1142, 1646/46 MB/s
INFO: status: 15% (83378962432/536870912000), sparse 12% (68357615616), duration 1145, 2146/18 MB/s
INFO: status: 16% (87751262208/536870912000), sparse 13% (72624775168), duration 1148, 1457/35 MB/s
INFO: status: 17% (93566730240/536870912000), sparse 14% (78348742656), duration 1151, 1938/30 MB/s
INFO: status: 18% (100454236160/536870912000), sparse 15% (85169696768), duration 1154, 2295/22 MB/s
INFO: status: 20% (107663392768/536870912000), sparse 17% (92344479744), duration 1157, 2403/11 MB/s
INFO: status: 21% (116091977728/536870912000), sparse 18% (100740714496), duration 1160, 2809/10 MB/s
INFO: status: 23% (123602272256/536870912000), sparse 20% (108207898624), duration 1163, 2503/14 MB/s
INFO: status: 24% (129216151552/536870912000), sparse 21% (113336709120), duration 1175, 467/40 MB/s
INFO: status: 25% (134593773568/536870912000), sparse 22% (118662885376), duration 1179, 1344/12 MB/s
INFO: status: 26% (140781420544/536870912000), sparse 23% (124709007360), duration 1182, 2062/47 MB/s
INFO: status: 27% (148756824064/536870912000), sparse 24% (132684365824), duration 1185, 2658/0 MB/s
INFO: status: 29% (155807055872/536870912000), sparse 26% (139699548160), duration 1188, 2350/11 MB/s
INFO: status: 30% (162741485568/536870912000), sparse 27% (146539331584), duration 1191, 2311/31 MB/s
INFO: status: 31% (166841679872/536870912000), sparse 28% (150493749248), duration 1194, 1366/48 MB/s
INFO: status: 32% (172479741952/536870912000), sparse 29% (156036448256), duration 1197, 1879/31 MB/s
INFO: status: 33% (178719490048/536870912000), sparse 30% (162226966528), duration 1200, 2079/16 MB/s
INFO: status: 34% (186112737280/536870912000), sparse 31% (169537146880), duration 1203, 2464/27 MB/s
INFO: status: 35% (192486375424/536870912000), sparse 32% (175830777856), duration 1206, 2124/26 MB/s
INFO: status: 36% (198284541952/536870912000), sparse 33% (181565902848), duration 1209, 1932/21 MB/s
INFO: status: 38% (204120653824/536870912000), sparse 34% (187239092224), duration 1212, 1945/54 MB/s
INFO: status: 39% (210048843776/536870912000), sparse 35% (193114308608), duration 1215, 1976/17 MB/s
INFO: status: 40% (218255130624/536870912000), sparse 37% (201300697088), duration 1218, 2735/6 MB/s
INFO: status: 41% (224710033408/536870912000), sparse 38% (207678259200), duration 1221, 2151/25 MB/s
INFO: status: 42% (227003990016/536870912000), sparse 39% (209835339776), duration 1224, 764/45 MB/s
INFO: status: 42% (227288612864/536870912000), sparse 39% (209865252864), duration 1232, 35/31 MB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: vm is online again after 1330 seconds
ERROR: Backup of VM 102 failed - job failed with err -5 - Input/output error
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[marcus_ph]

Please post the complete log of the backup.
You can't access the GUI on the node in question but all others in the cluster still work? Perhaps check your disk for errors.

thank you for your reply sir,

please see the full error details. thanks

 

[mira]

This is most likely related to a faulty disk. Please check all disks involved in the backup.
Just to be sure please post your storage config (/etc/pve/storage.cfg).

 

[marcus_ph]

This is most likely related to a faulty disk. Please check all disks involved in the backup.
Just to be sure please post your storage config (/etc/pve/storage.cfg).

hi sir, here is the storage config

 

[mira]

Please check your disks as mentioned previously (with smartctl for example). Check your dmesg for any errors. You can also run a filesystem check.

 

[marcus_ph]

Please check your disks as mentioned previously (with smartctl for example). Check your dmesg for any errors. You can also run a filesystem check.

hi, do i check the proxmox hdd or the files on the node?

please do note that i only have one hdd for both back up and local.

other nodes, back up with no problem. only this particular node is having this problem.

 

[mira]

Check both the filesystem in the VM as well as the physical disk.

 

[marcus_ph]

Check both the filesystem in the VM as well as the physical disk.

noted thanks, ill do this tonight.