Can't move CT volume to new disk due to input/output errors.

[LunaXQ]

Hello.

I am trying to move some containers to a new disk but when trying to move the volume i get the following output

Logical volume "vm-610-disk-0" created.
mke2fs 1.43.4 (31-Jan-2017)
Discarding device blocks:     4096/87818240 4722688/8781824012062720/8781824018878464/8781824025694208/8781824033034240/8781824039849984/8781824046665728/8781824054005760/8781824060821504/8781824067637248/8781824074452992/8781824081793024/87818240                 done                         
Creating filesystem with 87818240 4k blocks and 21954560 inodes
Filesystem UUID: 4337687c-3c05-4b09-aeed-7f28e843afec
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Allocating group tables:    0/2680         done                         
Writing inode tables:    0/2680         done                         
Creating journal (262144 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information:    0/2680         done

rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk00887.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk00950.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01003.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01004.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01022.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01099.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01104.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01151.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01219.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/chainstate/3464560.ldb": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk00887.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk00950.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01003.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01004.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01022.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01099.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01104.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01151.dat": Input/output error (5)
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/blocks/blk01219.dat": Input/output error (5)
ERROR: blocks/blk00887.dat failed verification -- update discarded.
ERROR: blocks/blk00950.dat failed verification -- update discarded.
rsync: read errors mapping "/var/lib/lxc/610/.copy-volume-2/chainstate/3464560.ldb": Input/output error (5)
ERROR: blocks/blk01003.dat failed verification -- update discarded.
ERROR: blocks/blk01004.dat failed verification -- update discarded.
ERROR: blocks/blk01022.dat failed verification -- update discarded.
ERROR: blocks/blk01099.dat failed verification -- update discarded.
ERROR: blocks/blk01104.dat failed verification -- update discarded.
ERROR: blocks/blk01151.dat failed verification -- update discarded.
ERROR: blocks/blk01219.dat failed verification -- update discarded.
ERROR: chainstate/3464560.ldb failed verification -- update discarded.

Number of files: 6,529 (reg: 6,524, dir: 5)
Number of created files: 6,527 (reg: 6,524, dir: 3)
Number of deleted files: 0
Number of regular files transferred: 6,534
Total file size: 331,788,304,278 bytes
Total transferred file size: 332,991,362,192 bytes
Literal data: 332,991,362,192 bytes
Matched data: 0 bytes
File list size: 131,062
File list generation time: 0.005 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 333,073,097,682
Total bytes received: 124,942

sent 333,073,097,682 bytes  received 124,942 bytes  102,499,837.71 bytes/sec
total size is 331,788,304,278  speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
  Logical volume "vm-610-disk-0" successfully removed
TASK ERROR: command '/usr/bin/rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system /var/lib/lxc/610/.copy-volume-2/ /var/lib/lxc/610/.copy-volume-1' failed: exit code 23

I think the filesystems in the containers are corrupted in some way, i also get similar issues when trying to make a backup of the container so i can try restoring it to the new disk. I've tried to repair the file system with fsck and it says errors were fixed but still have the same issues. I'm not so worried about the files it can't read being lost after the move if there's an option to make the move operation ignore errors but i can't find an option like this even when doing the move on the command line using pct move_voilume.

How do i repair the file system so the volume can be moved or how do i move it and ignore the errors?

Thanks!

 

[fiona]

Hi,
if fsck didn't help, it likely means that the errors come from the underlying storage/disks. Please check those and your /var/log/syslog for any other errors around the time of the move operation.

 

[LunaXQ]

Hi,
if fsck didn't help, it likely means that the errors come from the underlying storage/disks. Please check those and your /var/log/syslog for any other errors around the time of the move operation.

So i'm seeing a lot of errors in the syslog file like this

Mar 11 07:46:07 proxmox kernel: [ 1957.887350] ata1.00: exception Emask 0x0 SAct 0x1e000000 SErr 0x0 action 0x0
Mar 11 07:46:07 proxmox kernel: [ 1957.890197] ata1.00: cmd 60/08:c8:08:e1:28/00:00:0a:00:00/40 tag 25 ncq dma 4096 in
Mar 11 07:46:07 proxmox kernel: [ 1957.890197]          res 41/40:00:08:e1:28/00:00:0a:00:00/00 Emask 0x409 (media error) <F>
Mar 11 07:46:07 proxmox kernel: [ 1957.893133] ata1.00: error: { UNC }
Mar 11 07:46:07 proxmox kernel: [ 1957.899837] sd 0:0:0:0: [sda] tag#25 Sense Key : Medium Error [current]
Mar 11 07:46:07 proxmox kernel: [ 1957.900792] ata1: EH complete
Mar 11 07:46:07 proxmox kernel: [ 1957.943354] ata1.00: exception Emask 0x0 SAct 0x1840000 SErr 0x0 action 0x0
Mar 11 07:46:07 proxmox kernel: [ 1957.946141] ata1.00: cmd 60/08:90:08:e1:28/00:00:0a:00:00/40 tag 18 ncq dma 4096 in
Mar 11 07:46:07 proxmox kernel: [ 1957.946141]          res 41/40:00:08:e1:28/00:00:0a:00:00/00 Emask 0x409 (media error) <F>
Mar 11 07:46:07 proxmox kernel: [ 1957.948972] ata1.00: error: { UNC }
Mar 11 07:46:07 proxmox kernel: [ 1957.955668] ata1.00: configured for UDMA/133
Mar 11 07:46:07 proxmox kernel: [ 1957.955693] sd 0:0:0:0: [sda] tag#18 CDB: Read(10) 28 00 0a 28 e1 08 00 00 08 00
Mar 11 07:46:07 proxmox kernel: [ 1958.255304] ata1.00: exception Emask 0x0 SAct 0x180 SErr 0x0 action 0x0
Mar 11 07:46:07 proxmox kernel: [ 1958.261086] ata1.00: error: { UNC }
Mar 11 07:46:07 proxmox kernel: [ 1958.267785] ata1.00: configured for UDMA/133
Mar 11 07:46:07 proxmox kernel: [ 1958.267794] sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 11 07:46:07 proxmox kernel: [ 1958.267796] sd 0:0:0:0: [sda] tag#7 Sense Key : Medium Error [current]
Mar 11 07:46:07 proxmox kernel: [ 1958.267798] sd 0:0:0:0: [sda] tag#7 Add. Sense: Unrecovered read error - auto reallocate failed
Mar 11 07:46:07 proxmox kernel: [ 1958.267800] sd 0:0:0:0: [sda] tag#7 CDB: Read(10) 28 00 0a 28 d0 e0 00 01 00 00
Mar 11 07:46:07 proxmox kernel: [ 1958.267801] print_req_error: I/O error, dev sda, sector 170447104
Mar 11 07:46:07 proxmox kernel: [ 1958.268726] ata1: EH complete
Mar 11 07:46:07 proxmox kernel: [ 1958.323333] ata1.00: exception Emask 0x0 SAct 0x4 SErr 0x0 action 0x0
Mar 11 07:46:07 proxmox kernel: [ 1958.324254] ata1.00: irq_stat 0x40000008
Mar 11 07:46:07 proxmox kernel: [ 1958.325183] ata1.00: failed command: READ FPDMA QUEUED
Mar 11 07:46:07 proxmox kernel: [ 1958.326121] ata1.00: cmd 60/08:10:00:d1:28/00:00:0a:00:00/40 tag 2 ncq dma 4096 in
Mar 11 07:46:07 proxmox kernel: [ 1958.326121]          res 41/40:00:00:d1:28/00:00:0a:00:00/00 Emask 0x409 (media error) <F>
Mar 11 07:46:07 proxmox kernel: [ 1958.328000] ata1.00: status: { DRDY ERR }
Mar 11 07:46:07 proxmox kernel: [ 1958.328949] ata1.00: error: { UNC }
Mar 11 07:46:07 proxmox kernel: [ 1958.335664] sd 0:0:0:0: [sda] tag#2 Sense Key : Medium Error [current]

Is there a way to repair the disk themselves and is there a risk of losing all the data on the disk if i do this? Currently the CTs im trying to move all boot and run perfectly fine and it's only when i try to move them that i run into issues.

 

[fiona]

Seems like the disk might be dying. I'd take a look at tools like ddrescue for how to get a copy of the data that still can be saved.