I have problem with my ZFS pool

[biotim]

Hello,
I have problem with my ZFS pool. My RAM recently flipped bit and as a result my pool got corrupt by permanent error (zpool status -vx):

zpool status -v
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: ...
  scan: scrub repaired 0B in 60h35m with 7 errors on Tue Sep 11 12:59:58 2018
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0    10
          sda2      ONLINE       0     0    20

errors: Permanent errors have been detected in the following files:

        <0x118>:<0x1>
        //var/lib/vz/template/iso/ubuntu-18.04.1-live-server-amd64.iso
        //var/lib/vz/template/iso/CentOS-7-x86_64-Minimal-1804.iso
        rpool/data/vm-100-disk-1:<0x1>
        <0x9d>:<0x1>
        rpool/data/vm-101-disk-1@c6:<0x1>

Can you help me please?

 

[guletz]

Hi,

It is very hard for anybody to help in this case(data coruption) because your zfs pool do not have any parity(like a zfs mirror/raidz1/2/3). Your best bet is to recover your data from a backup, or from a zfs snapshot(if you have) for the:

- rpool/data/vm-100-disk-1

The rest of the data can be download again(*.iso), and I guess that rpool/data/vm-101-disk-1@c6 is a snapshot, so it not so important

 

[biotim]

Hi,

It is very hard for anybody to help in this case(data coruption) because your zfs pool do not have any parity(like a zfs mirror/raidz1/2/3). Your best bet is to recover your data from a backup, or from a zfs snapshot(if you have) for the:

- rpool/data/vm-100-disk-1

The rest of the data can be download again(*.iso), and I guess that rpool/data/vm-101-disk-1@c6 is a snapshot, so it not so important

thanks for the answer
Yes, vm-100-disk-1 has its own snapshot, which I can roll back, but the problem is that inside fs goes into read mode after 24 hours, which I do not understand, because of what.
after the rollback, I might need a second scrub?

 

[guletz]

is that inside fs goes into read mode after 24 hours

... because there are to many errors, and zfs can not recover the data, so for this reson the fs is go into read-only mode!

 

[guletz]

after the rollback, I might need a second scrub?

Yes, is highly recomended. If you do this, then after scrub is finish, you will see if your pool is healthy or not. But before start
the scrub you will need to clear the errors with:

zpool clear rpool

 

[biotim]

clear, clear, how to scrub run from under the console what kind of command is needed for this
last my scrub lasted 2.5 days

 

[guletz]

For run a scrub using console:

zpool scrub rpool

Yes it is possible that scrub need a long time finish, depending of your zfs data size, disk speed, iops server activity and so on.

 

[biotim]

For run a scrub using console:

zpool scrub rpool

Yes it is possible that scrub need a long time finish, depending of your zfs data size, disk speed, iops server activity and so on.

ok, thanks, but zpool clear rpool itself will not remove the virtual machine itself?

 

[guletz]

ok, thanks, but zpool clear rpool itself will not remove the virtual machine itself?

It only remove the errors counter show in the "zpool status -v"

 

[biotim]

ok. decided to run scrub twice in a row it will take 5 days then we'll see

 

[guletz]

Please be kind and post your results.

And ... most important a big good luck for you

 

[biotim]

 zpool status -v
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: ...
  scan: scrub repaired 0B in 67h46m with 7 errors on Fri Sep 21 01:03:58 2018
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0    13
          sda2      ONLINE       0     0    27

errors: Permanent errors have been detected in the following files:

        //var/lib/vz/template/iso/ubuntu-18.04.1-live-server-amd64.iso
        rpool/data/vm-100-disk-1@c1:<0x1>
        rpool/data/vm-100-disk-1:<0x1>
        rpool/data/vm-101-disk-1@c6:<0x1>

Strangely, but during the first scrub fs vm100 the system did not go into read mode