[SOLVED] ZFS pool error but VM fine

L

leaskovski

New Member
Jan 22, 2024
7
0
1
Evening all... so I am at a bit of a quandary... One of my pools is reporting an error, and when I get the details from it, it tells me that a particular VMs disk is in the firing line...
Code: 
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 03:04:19 with 2 errors on Sun Jan 14 03:28:24 2024
config:

        NAME                                        STATE     READ WRITE CKSUM
        rpool                                       ONLINE       0     0     0
          ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3  ONLINE       0     0     4
          ata-SAMSUNG_HD161GJ_S1VCJ9FZ318818-part3  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        rpool/data/vm-101-disk-0:<0x1>
Now that VM is a Zabbix server, so its not really the end of the world if it dies. I only loose historic data, and the config setup. I could backup the mariadb off to another location, and that means I don't even loose anything. This is my problem... The disk is obviously getting thrown up as some sort of checksum issue where it sits on the physical disk, I have a raid ) setup, so that checksum has replicated across disks, so not able to fix that problem, but the VM is actually working fine, without it reporting errors. I tried moving the disk off to another pool, but that failed half way through, so that kind of shows that there is some issue there.

What would you do? Ignore it? Rebuild the VM? Some how override the error so it doesn;t reappear after clearing and then scrubs?? What to do?
 
Reading the (entire) virtual disk of the that VM will fail with a read error (since ZFS knows some part of the data is corrupt), and that's why you cannot move the disk to another pool.
Find you which file is stored in the corrupt block(s) (I can't remember the details, sorry) and overwrite the bad part of the file (or delete and trim) to fix the permanent virtual disk error.
 
Yeah, thats what I thought I would have to do. I was fishing for commands though, and badblocks was what I was looking for. Here is my detail from badblocks..
Code: 
Checking for bad blocks (read-only test): 11024936
11024937
11024938
11024939
11024940
11024941
11024942
11024943
11024976
11024977
11024978
11024979
11024980
11024981
11024982
11024983
And my fdisk is...
Code: 
Disk /dev/sda: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xba9b24eb

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1  *        2048  1050623  1048576  512M 83 Linux
/dev/sda2        1050624  9439231  8388608    4G 83 Linux
/dev/sda3        9439232 11536383  2097152    1G 83 Linux
/dev/sda4       11536384 62914559 51378176 24.5G  5 Extended
/dev/sda5       11538432 62912511 51374080 24.5G 83 Linux
So it looks like all the badblocks are sitting in that 1G sda3 partition. Time to find out what it is for, lsblk for the rescure...
Code: 
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   30G  0 disk
├─sda1   8:1    0  512M  0 part /boot
├─sda2   8:2    0    4G  0 part /
├─sda3   8:3    0    1G  0 part /tmp
├─sda4   8:4    0    1K  0 part
└─sda5   8:5    0 24.5G  0 part /var/lib/mysql
Whoop! Its the /tmp. I guess I just need to blow out its contents to clear the error?
 
Ok, so I used the shred tool to wipe that partition that contained the bad blocks that seem to be a result of the error. After recreating the the partition and sorting out fstab on the box to be able to boot again (thanks uuid references in the fstab for zabbix appliance I downloaded!), the VM is back up and running again, but clearing and then doing a full scrub on the pool hasn't cleared the error.

I would have thought the shred would have gone over every block and do a write, thus doing what you suggested. Maybe not then.
 
One thing I have noticed from all this though... my rpool isnt a mirror o_O which I thought it was, so thats a whole other question incoming.
 
leaskovski said:
One thing I have noticed from all this though... my rpool isnt a mirror o_O which I thought it was, so thats a whole other question incoming.
Click to expand...
Right, the mirror-0 is missing in the output of zpool status, and therefore it's a stripe (RAID0). Nowadays with newer ZFS on Linux versions, you can remove a vdev from a pool, so that might work (but maybe take a while).
 
So, I spent a bit more time on this over the weekend. I have moved all my CT Volumes and VM Disks off of rpool to my NAS pool. For that errornous VM (Zabbix), I ended up exporting the data I wanted off of it, and just creating a new applience using the latest images. After I deleted the old VM, the hard disk was removed as well, so the rpool is now empty...

Screenshot 2024-01-29 094737.png

Oldly enough though, it still won't clear the error. -v gives me...
Code: 
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:09:56 with 0 errors on Sat Jan 27 23:03:06 2024
config:

        NAME                                        STATE     READ WRITE CKSUM
        rpool                                       ONLINE       0     0     0
          ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3  ONLINE       0     0     0
          ata-SAMSUNG_HD161GJ_S1VCJ9FZ318818-part3  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <0x4685>:<0x1>
I thought I would just try and delete the rpool and recreate it, but this time with mirroring enabled. However, Its not letting me, as the rpool is aparently in use...
Screenshot 2024-01-29 094805.png
/dev/sda3 and /dev/sdc3 are the offending partitions. Mount lists the following on the proxmox server...
Code: 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=8092820k,nr_inodes=2023205,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1625480k,mode=755,inode64)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13490)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
NAS on /NAS type zfs (rw,xattr,noacl)
NAS/subvol-105-disk-0 on /NAS/subvol-105-disk-0 type zfs (rw,xattr,posixacl)
NAS/subvol-106-disk-0 on /NAS/subvol-106-disk-0 type zfs (rw,xattr,posixacl)
rpool on /rpool type zfs (rw,relatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,relatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,relatime,xattr,noacl)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1625476k,nr_inodes=406369,mode=700,inode64)
It appears rpool is being mounted...
Code: 
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
rpool on /rpool type zfs (rw,relatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,relatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,relatime,xattr,noacl)
What would be the right way of ditching that rpool and creating it again, thus hopefully clearing out that error?
 
leaskovski said:
So, I spent a bit more time on this over the weekend. I have moved all my CT Volumes and VM Disks off of rpool to my NAS pool. For that errornous VM (Zabbix), I ended up exporting the data I wanted off of it, and just creating a new applience using the latest images. After I deleted the old VM, the hard disk was removed as well, so the rpool is now empty...
Click to expand...
It's not but good idea to save everything somewhere else.
leaskovski said:
Oldly enough though, it still won't clear the error. -v gives me...
Code: 
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:09:56 with 0 errors on Sat Jan 27 23:03:06 2024
config:

        NAME                                        STATE     READ WRITE CKSUM
        rpool                                       ONLINE       0     0     0
          ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3  ONLINE       0     0     0
          ata-SAMSUNG_HD161GJ_S1VCJ9FZ318818-part3  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <0x4685>:<0x1>
Click to expand...
That's because the corrupt block is still on the disk. You need to overwrite it (or delete and trim it, maybe).
leaskovski said:
I thought I would just try and delete the rpool and recreate it, but this time with mirroring enabled. However, Its not letting me, as the rpool is aparently in use...
Click to expand...
Your Proxmox installation is on the rpool! That's why it's not completely empty and in use.
leaskovski said:
What would be the right way of ditching that rpool and creating it again, thus hopefully clearing out that error?
Click to expand...
You could reinstall Proxmox. Save everything below /etc/pve/ before doing so and don't copy it back to the new installation but use it to lookup anything that you don't remember.

Or simply remove the broken vdev (I can't tell at the moment which one it is but make sure you remove the one with the corrupt block) from the rpool, and then add it again as a mirror.

EDIT: /dev/disk/by-id/ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3 is the one with the corrupt block.
 
Last edited:
leaskovski said:
errors: Permanent errors have been detected in the following files: <0x4685>:<0x1>
Click to expand...
There is no longer a name in the reference.
This means the reference is stale, it is probably deleted. Do a scrub, then this error should vanish.

rpool is the root pool of Proxmox VE, the OS resides on it. Deleting while using it with the running OS will you do no good.
 
Last edited:
leesteken said:
Or simply remove the broken vdev (I can't tell at the moment which one it is but make sure you remove the one with the corrupt block) from the rpool, and then add it again as a mirror.

EDIT: /dev/disk/by-id/ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3 is the one with the corrupt block.
Click to expand...
So looking at the possibility of doing this. You mentioned it first time, but I wasn't too sure what you meant, but a little reading, I think I am understanding your logic.

So use zpool remove (https://openzfs.github.io/openzfs-docs/man/master/8/zpool-remove.8.html) to remove the dodgy device. This should shrink the rpool down to half its size, then readd the same device back into the pool, but using the mirror option, i.e.

zpool remove rpool ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3
zpool add rpool mirror ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3
 
leaskovski said:
zpool remove rpool ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3
Click to expand...
I think this should be zpool remove rpool /dev/disk/by-id/ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3
leaskovski said:
zpool add rpool mirror ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3
Click to expand...
"zpool add" is exactly the mistake you probably made before, which gives you a stripe (RAID0) instead of a mirror.
I think this should be zpool attach rpool /dev/disk/by-id/ata-SAMSUNG_HD161GJ_S1VCJ9FZ318818-part3 /dev/disk/by-id/ata-SAMSUNG_HD161HJ_S14LJA0Q536960-part3 as you always need to specify which existing drive you want to mirror.
 
  • Like
Reactions: leaskovski
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom