ZFS Datapool Error - How to Access Data from Virtual Machine Disk from Proxmox?

N

nikhilbhalwankar

Active Member
Jun 8, 2019
14
1
43
42
Hi,

I have a JBOD server with below configurationn ->

- I have two datapools. Each are under RAID-5
- Proxmox VE is installed on the server and these two RAID-5 datapools are attached to it as ZFS shares DATAPOOL-1 and DATAPOOL-2.
- TrueNAS 13 is installed as a virtual machine over Proxmox VE with three disks attached to it disk1 for OS, disk2 from DATAPOOL-1 for data, disk3 from DATAPOOL-2 for data.
- I have some issue with DATAPOOL-2. Proxmox is showing below errors

Code: 
zpool status -v DATAPOOL-2
  pool: DATAPOOL-2
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 132h51m with 1367849 errors on Fri Aug 14 13:15:18 2026
config:

    NAME        STATE     READ WRITE CKSUM
    DATAPOOL-2  ONLINE       0     0 9.15M
      sdb       ONLINE       0     0 18.3M

errors: Permanent errors have been detected in the following files:

        DATAPOOL-2/vm-100-disk-0:<0x1>


- disk3 of the TrueNAS is from this error DATAPOOL-2.
- If I detach disk3 then Truenas boots up fast and correct.
- If I attach disk3 then Truenas hangs during boot due to Proxmox ZFS error. I can not access TrueNas even from ssh.

Findings-
- Both RAID-5 pools are online from JBOD level. No erorrs.
- "zpool status -v" at proxmox level shows attached errors.
- disk3 from TrueNAS being from DATAPOOL-2, can not be accessed to TrueNAS as TrueNAS hangs.

How can I access data from VM disk3 which is at Truenas level? How to fix Proxmox ZFS errors? Frankly I do not wish to use backup as the disk3 from VM is itself 20 TB+ in size. Is there any way to fix the proxmox disk errors so that I can safely mount the disk3 back to TrueNAS?
 
> I have two datapools. Each are under RAID-5
> - Proxmox VE is installed on the server and these two RAID-5 datapools are attached to it as ZFS shares DATAPOOL-1 and DATAPOOL-2.
> - TrueNAS 13 is installed as a virtual machine over Proxmox VE with three disks attached to it disk1 for OS, disk2 from DATAPOOL-1 for data, disk3
> from DATAPOOL-2 for data

Proxmox explicitly does not support or recommend ZFS on top of RAID-anything storage. ZFS has its own RAIDZx levels, DRAID, and mirrors.

[[
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs

Do not use ZFS on top of a hardware RAID controller which has its own cache management. ZFS needs to communicate directly with the disks. An HBA adapter or something like an LSI controller flashed in “IT” mode is more appropriate.
]]

The "proper" process to recover from this would be to get more disks, copy everything that you can't reproduce off the faulty pool, and rebuild it the right way.

Leaving truenas out of it and the VM powered down, try importing the pools at the proxmox host level read-only.

zpool import -d /dev/disk/by-id -f -o readonly=on poolname

Don't try a zfs send, with known pool errors it will likely abort. Use rsync / rclone (for parallel transfers) and when everything is copied over, destroy the zpool and rebuild it with direct hardware access as described above.

> scan: scrub repaired 0B in 132h51m with 1367849 errors on Fri Aug 14 13:15:18 2026

Your disk appears to be dying. Check SMART stats and invest in 2x more disks of same size or larger so you at least have self-healing scrubs.
 
Hi,

In my case, DATAPOOL-2 is already mounted on the Proxmox host ZFS pool. I can see the same using zfs list command. Can I directly mount the truenas disk DATAPOOL-2/vm-100-disk-0 in read only mode on proxmox itself for fetching data?
 
Last edited:
Probably yes (depending on the filesystem inside, and how badly corrupted the data is). It would look something like


Code: 
mkdir /tmp/mnt
mount /dev/zvol/DATAPOOL-2/vm-100-disk-0-part1 /tmp/mnt -o ro

Don't use /mnt (It is likely that some storage is mounted under /mnt already). There is a chance for that mounting to cause the host OS to crash. If you do this inside another VM instead of doing it on the host you can reduce the risk of downtime for the other VMs on that host.

Also, you should seriously consider just restoring that disk from a backup instead. You already know some of the data it contains is corrupt.

If you are super lucky the corruption will be in blocks that are free space in the filesystem within, and copying the files will go fine, but it is far more likely you will get read errors reading some file(s) out of that virtual disk.

You can minimize what you need to read from this volume by restoring from a known good backup first. Restore from a backup, and then mount both the restored and the current volume, and then use rsync to copy just what is new since that backup into the restored disk will probably only need to copy a small fraction of the files from the corrupt volume, and you might get lucky and avoid trying to read from the broken areas completely.

The most likely data to be corrupt is whatever was being written when the event that caused the corruption happened, so even that restore then rsync plan isn't very likely to avoid trying to read from the corrupt parts of the virtual disk, but it is the best odds you have, other than just restoring from backup and forgetting about this broken virtual disk.
 
Given that there are this many checksum errors in the “data,” I don't think there's any chance that it was just free space.
You should stop using TrueNAS ZFS on ZFS on HW RAID.

If you have a backup, you should restore it.

Code: 
CKSUM
9.15M
18.3M
 
You must log in or register to reply here.
Top Bottom
Back