[SOLVED] .raw images corrupt?

N

NomarMora

New Member
Aug 14, 2015
27
0
1
Good day:

This is the situation, dmesg is reporting errors on loop devices (complete log on attachment), fsck on images show errors like this:

Code: 
root@proxmox4-eu3:~# e2fsck -fyv /var/lib/vz/images/107/vm-107-disk-1.raw
e2fsck 1.42.12 (29-Aug-2014)
MMP interval is 10 seconds and total wait time is 42 seconds. Please wait...
/var/lib/vz/images/107/vm-107-disk-1.raw: recovering journal
Error writing block 9015 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9017 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9027 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9029 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9002 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9003 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9008 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9009 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9022 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9023 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9024 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9025 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 774 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 523 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Pass 1: Checking inodes, blocks, and sizes
Deleted inode 3651 has zero dtime.  Fix? yes

Pass 2: Checking directory structure
Directory inode 3623, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3623.
Fix? yes

Setting filetype for entry '.' in ??? (3623) to 2.
Missing '..' in directory inode 3623.
Fix? yes

Setting filetype for entry '..' in ??? (3623) to 2.
Directory inode 3628, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3628.
Fix? yes

Setting filetype for entry '.' in ??? (3628) to 2.
Missing '..' in directory inode 3628.
Fix? yes

Setting filetype for entry '..' in ??? (3628) to 2.
Directory inode 3645, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3645.
Fix? yes

Setting filetype for entry '.' in ??? (3645) to 2.
Missing '..' in directory inode 3645.
Fix? yes

Setting filetype for entry '..' in ??? (3645) to 2.
Directory inode 3647, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3647.
Fix? yes

Setting filetype for entry '.' in ??? (3647) to 2.
Missing '..' in directory inode 3647.
Fix? yes

Setting filetype for entry '..' in ??? (3647) to 2.
Directory inode 3629, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3629.
Fix? yes

Setting filetype for entry '.' in ??? (3629) to 2.
Missing '..' in directory inode 3629.
Fix? yes

Setting filetype for entry '..' in ??? (3629) to 2.
Directory inode 3652, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3652.
Fix? yes

Setting filetype for entry '.' in Error writing block 9002 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

??? (3652) to 2.
Missing '..' in directory inode 3652.
Fix? yes

Setting filetype for entry '..' in ??? (3652) to 2.
Error writing block 9003 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Error writing block 9004 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Error writing block 9006 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Error writing block 9008 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3624, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3624.
Fix? yes

Setting filetype for entry '.' in ??? (3624) to 2.
Missing '..' in directory inode 3624.
Fix? yes

Setting filetype for entry '..' in ??? (3624) to 2.
Error writing block 9009 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3625, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3625.
Fix? yes

Setting filetype for entry '.' in ??? (3625) to 2.
Missing '..' in directory inode 3625.
Fix? yes

Setting filetype for entry '..' in ??? (3625) to 2.
Directory inode 3640, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3640.
Fix? yes

Setting filetype for entry '.' in ??? (3640) to 2.
Missing '..' in directory inode 3640.
Fix? yes

Setting filetype for entry '..' in ??? (3640) to 2.
Directory inode 3642, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3642.
Fix? yes

Setting filetype for entry '.' in ??? (3642) to 2.
Missing '..' in directory inode 3642.
Fix? yes

Setting filetype for entry '..' in ??? (3642) to 2.
Directory inode 3649, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3649.
Fix? yes

Setting filetype for entry '.' in ??? (3649) to 2.
Missing '..' in directory inode 3649.
Fix? yes

Setting filetype for entry '..' in ??? (3649) to 2.
Error writing block 9015 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3653, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3653.
Fix? yes

Setting filetype for entry '.' in ??? (3653) to 2.
Missing '..' in directory inode 3653.
Fix? yes

Setting filetype for entry '..' in ??? (3653) to 2.
Directory inode 3655, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3655.
Fix? yes

Setting filetype for entry '.' in ??? (3655) to 2.
Missing '..' in directory inode 3655.
Fix? yes

Setting filetype for entry '..' in ??? (3655) to 2.
Error writing block 9017 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3657, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3657.
Fix? yes

Setting filetype for entry '.' in ??? (3657) to 2.
Missing '..' in directory inode 3657.
Fix? yes

Setting filetype for entry '..' in ??? (3657) to 2.
Directory inode 3659, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3659.
Fix? yes

Setting filetype for entry '.' in ??? (3659) to 2.
Missing '..' in directory inode 3659.
Fix? yes

Setting filetype for entry '..' in ??? (3659) to 2.
Error writing block 9018 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3656, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3656.
Fix? yes

Setting filetype for entry '.' in ??? (3656) to 2.
Missing '..' in directory inode 3656.
Fix? yes

Setting filetype for entry '..' in ??? (3656) to 2.
Error writing block 9019 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3626, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3626.
Fix? yes

Setting filetype for entry '.' in ??? (3626) to 2.
Missing '..' in directory inode 3626.
Fix? yes

Setting filetype for entry '..' in ??? (3626) to 2.
Error writing block 9020 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3658, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3658.
Fix? yes

Setting filetype for entry '.' in Error writing block 9022 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

??? (3658) to 2.
Missing '..' in directory inode 3658.
Fix? yes

Setting filetype for entry '..' in ??? (3658) to 2.
Error writing block 9023 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Directory inode 3627, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 3627.
Fix? yes

Setting filetype for entry '.' in ??? (3627) to 2.
Missing '..' in directory inode 3627.
Fix? yes

Setting filetype for entry '..' in ??? (3627) to 2.
Error writing block 9024 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Error writing block 9025 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Error writing block 9026 (Attempt to write block to filesystem resulted in short write).  Ignore error? yes

Error writing block 9027 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Error writing block 9028 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Error writing block 9029 (Attempt to write block to filesystem resulted in short write) while reading directory block.  Ignore error? yes

Pass 3: Checking directory connectivity
'..' in /tmp/systemd-private-ef4ee2f1552240dba88d152c4a27ca10-varnish.service-OT9CgB (3623) is <The NULL inode> (0), should be /tmp (2736).
Fix? yes

Couldn't fix parent of inode 3623: EXT2 directory corrupted

'..' in /tmp/.ICE-unix (3624) is <The NULL inode> (0), should be /tmp (2736).
Fix? yes

Couldn't fix parent of inode 3624: EXT2 directory corrupted

'..' in /tmp/.XIM-unix (3625) is <The NULL inode> (0), should be /tmp (2736).
Fix? yes

Couldn't fix parent of inode 3625: EXT2 directory corrupted

'..' in /tmp/.font-unix (3626) is <The NULL inode> (0), should be /tmp (2736).
Fix? yes

Couldn't fix parent of inode 3626: EXT2 directory corrupted

'..' in /tmp/.Test-unix (3627) is <The NULL inode> (0), should be /tmp (2736).
Fix? yes

Couldn't fix parent of inode 3627: EXT2 directory corrupted

Unconnected directory inode 3628 (???)
Connect to /lost+found? yes

Couldn't fix parent of inode 3628: EXT2 directory corrupted

'..' in /var/tmp/systemd-private-ef4ee2f1552240dba88d152c4a27ca10-varnish.service-Ns2ltV (3629) is <The NULL inode> (0), should be /var/tmp (261).
Fix? yes

Couldn't fix parent of inode 3629: EXT2 directory corrupted

'..' in /lost+found/#3640 (3640) is <The NULL inode> (0), should be /lost+found (11).
Fix? yes

Couldn't fix parent of inode 3640: EXT2 directory corrupted

'..' in /var/tmp/systemd-private-ef4ee2f1552240dba88d152c4a27ca10-varnish.service-cLCRl2 (3642) is <The NULL inode> (0), should be /var/tmp (261).
Fix? yes

Couldn't fix parent of inode 3642: EXT2 directory corrupted

'..' in /lost+found/#3645 (3645) is <The NULL inode> (0), should be /lost+found (11).
Fix? yes

Couldn't fix parent of inode 3645: EXT2 directory corrupted

'..' in /lost+found/#3647 (3647) is <The NULL inode> (0), should be /lost+found (11).
Fix? yes

Couldn't fix parent of inode 3647: EXT2 directory corrupted

'..' in /lost+found/#3649 (3649) is <The NULL inode> (0), should be /lost+found (11).
Fix? yes

Couldn't fix parent of inode 3649: EXT2 directory corrupted

Unconnected directory inode 3652 (???)
Connect to /lost+found? yes

Couldn't fix parent of inode 3652: EXT2 directory corrupted

'..' in /tmp/systemd-private-ef4ee2f1552240dba88d152c4a27ca10-varnishncsa.service-MgZtGf (3653) is <The NULL inode> (0), should be /tmp (2736).
Fix? yes

Couldn't fix parent of inode 3653: EXT2 directory corrupted

Unconnected directory inode 3655 (???)
Connect to /lost+found? yes

Couldn't fix parent of inode 3655: EXT2 directory corrupted

'..' in /lost+found/#3656 (3656) is <The NULL inode> (0), should be /lost+found (11).
Fix? yes

Couldn't fix parent of inode 3656: EXT2 directory corrupted

'..' in /var/tmp/systemd-private-ef4ee2f1552240dba88d152c4a27ca10-varnishncsa.service-NfwCTz (3657) is <The NULL inode> (0), should be /var/tmp (261).
Fix? yes

Couldn't fix parent of inode 3657: EXT2 directory corrupted

'..' in /lost+found/#3658 (3658) is <The NULL inode> (0), should be /lost+found (11).
Fix? yes

Couldn't fix parent of inode 3658: EXT2 directory corrupted

Unconnected directory inode 3659 (???)
Connect to /lost+found? yes

Couldn't fix parent of inode 3659: EXT2 directory corrupted

Pass 4: Checking reference counts
Inode 2 ref count is 32, should be 41.  Fix? yes

Inode 261 ref count is 6, should be 4.  Fix? yes

Inode 2736 ref count is 9, should be 3.  Fix? yes

Inode 3623 ref count is 3, should be 2.  Fix? yes

Inode 3628 ref count is 3, should be 2.  Fix? yes

Inode 3629 ref count is 3, should be 2.  Fix? yes

Inode 3652 ref count is 3, should be 2.  Fix? yes

Inode 3653 ref count is 3, should be 2.  Fix? yes

Inode 3655 ref count is 3, should be 2.  Fix? yes

Inode 3657 ref count is 3, should be 2.  Fix? yes

Inode 3659 ref count is 3, should be 2.  Fix? yes

Pass 5: Checking group summary information
Free blocks count wrong for group #10 (12269, counted=32768).
Fix? yes

Free blocks count wrong (1706983, counted=1759728).
Fix? yes

Inode bitmap differences:  -3651
Fix? yes

Free inodes count wrong for group #0 (4556, counted=4557).
Fix? yes

Free inodes count wrong (497263, counted=497264).
Fix? yes


/var/lib/vz/images/107/vm-107-disk-1.raw: ***** FILE SYSTEM WAS MODIFIED *****

/var/lib/vz/images/107/vm-107-disk-1.raw: ********** WARNING: Filesystem still has errors **********


       27024 inodes used (5.15%, out of 524288)
          41 non-contiguous files (0.2%)
           7 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 22629/17
      337424 blocks used (16.09%, out of 2097152)
           0 bad blocks
           1 large file

       20297 regular files
        2191 directories
           2 character device files
           0 block device files
           0 fifos
          12 links
        4524 symbolic links (4367 fast symbolic links)
           1 socket
------------
       27027 files

But fsck on /dev/mapper/pve-data show NO errors!

No errors on smartctl tests, no errors on raid1 checks

Any advice?

Thanks in advance
Nomar
 

Attachments

  • dmesg.output.txt
    260 KB · Views: 4
Please keep in mind that the image does not contain a partition but a disk. So you have to decode the partition table and mount the desired partition before you can fsck it.
 
pabernethy said:
Please keep in mind that the image does not contain a partition but a disk. So you have to decode the partition table and mount the desired partition before you can fsck it.
Click to expand...

:-o

Thanks :)

How can I run the fsck command? How can I check?

Mounting the image and then runing fsck?

How can I mount the image? runing:

Code: 
mount -o loop ....

like a .iso image?

Thanks in advance
Nomar
 
You can scan the partition table of the image with
Code: 
# partx -va /var/lib/vz/images/107/vm-107-disk-1.raw
That will show you the partitions it found and also the loop device it uses for the image. You can then mount it with a regular mount, as the loop has already been taken care of. To fsck it you don't need to mount it, though.
 
Hello,

I stopped a container and issue the command, this is the response:

Code: 
root@proxmox4-eu3:~# partx -va /var/lib/vz/images/102/vm-102-disk-1.raw
partition: none, disk: /var/lib/vz/images/102/vm-102-disk-1.raw, lower: 0, upper: 0
Trying to use '/dev/loop9' for the loop device
partx: /dev/loop9: failed to read partition table

Did I do it the right way?

this is the output of losetup command:

Code: 
root@proxmox4-eu3:~# losetup --list
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         1  0 /var/lib/vz/images/100/vm-100-disk-1.raw
/dev/loop1         0      0         1  0 /var/lib/vz/images/101/vm-101-disk-1.raw
/dev/loop2         0      0         0  0 /var/lib/vz/images/102/vm-102-disk-1.raw
/dev/loop3         0      0         1  0 /var/lib/vz/images/103/vm-103-disk-1.raw
/dev/loop4         0      0         1  0 /var/lib/vz/images/104/vm-104-disk-1.raw
/dev/loop5         0      0         1  0 /var/lib/vz/images/105/vm-105-disk-1.raw
/dev/loop6         0      0         1  0 /var/lib/vz/images/106/vm-106-disk-1.raw
/dev/loop7         0      0         1  0 /var/lib/vz/images/107/vm-107-disk-1.raw
/dev/loop8         0      0         0  0 /var/lib/vz/images/107/vm-107-disk-1.raw
/dev/loop9         0      0         0  0 /var/lib/vz/images/102/vm-102-disk-1.raw
 
I tried and this is the result, obviously I must be doing it wrong...

Code: 
root@proxmox4-eu3:~# mount /dev/loop2 /mnt/
mount: /dev/loop2: can't read superblock

Because the pct start command start the container.
 
Oh, it's a container. In that case the image does contain a partition. :rolleyes: Sorry for the confusion. So yes, it seems like the FS took a hit. Did something happen to the container?
 
pabernethy said:
Oh, it's a container. In that case the image does contain a partition. :rolleyes: Sorry for the confusion. So yes, it seems like the FS took a hit. Did something happen to the container?
Click to expand...
:-D no problem :)

No, the hn uptime is
Code: 
uptime 
 11:34:30 up 117 days,  3:36,  2 users,  load average: 1.02, 0.92, 0.71

I did get alerted because the filesystems on several containers change to ro status, no restarts, no problems till this situation.

The fisical disks are ok! the smart test output is clean, no problem on mirror system. Even the fsck on the lvm partition is clean!

I don't know were to look at :-( My first thought was that the physical disk was damaged in some way, but with all the tests that I already did I was more confused.

Are there reasons for images get corrupted without corruption on the hn's file system?
 
Mistery solved :-( there was no space left on /var/lib/vz! :-(

The spaces was 0 on hn device.

But I swear there was space left no much time ago, the raw images grow over time?

I set the space for every image at the beginning the space should not change :-(
 
NomarMora said:
I set the space for every image at the beginning the space should not change :-(
Click to expand...
the files are sparse, so they appear as their set size, but use only the amount that is actually used
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back