ZFS IO error on 9.1.4

D

DJI

Active Member
Sep 22, 2020
22
1
43
59
Hi all,

Any advice on this?

Since moving to 9, currently running 9.1.4, we are experiencing Status: io-error on VM's on ZFS.

Pools have been scrubbed, disks pass smart and 0/1 percent wear out, plenty of space and RAM, and it happens on multiple servers randomly.

We have to stop VM and start it again, this is the only error we can see:

Code: 
Jan 23 11:06:23 pm12 zed[1756271]: eid=10789 class=dio_verify_wr pool='data' size=131072 offset=549274644480 priority=1 err=5 flags=0x100200080 bookmark=387:144:0:1574116

Code: 
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS FSTYPE            LABEL                            MODEL
zd0         230:0    0   250G  0 disk                                                                 
|-zd0p1     230:1    0   500M  0 part              ntfs              System Reserved                 
`-zd0p2     230:2    0 199.5G  0 part              ntfs                                               
zd16        230:16   0   200G  0 disk                                                                 
`-zd16p1    230:17   0   200G  0 part              ext4                                               
zd48        230:48   0    32G  0 disk                                                                 
|-zd48p1    230:49   0    31G  0 part              ext4                                               
|-zd48p2    230:50   0     1K  0 part                                                                 
`-zd48p5    230:53   0   975M  0 part              swap                                               
zd80        230:80   0     4M  0 disk                                                                 
zd96        230:96   0   150G  0 disk                                                                 
|-zd96p1    230:97   0   512K  0 part                                                                 
|-zd96p2    230:98   0   146G  0 part              ufs                                               
`-zd96p3    230:99   0     4G  0 part                                                                 
zd112       230:112  0   200G  0 disk                                                                 
|-zd112p1   230:113  0   128M  0 part                                                                 
`-zd112p2   230:114  0 199.9G  0 part              ntfs              MAIL-PO 2021_07_15 13:35 DISK_01
zd128       230:128  0    32G  0 disk                                                                 
|-zd128p1   230:129  0    30G  0 part              ext4                                               
|-zd128p2   230:130  0     1K  0 part                                                                 
`-zd128p5   230:133  0     2G  0 part              swap                                               
zd144       230:144  0    50G  0 disk                                                                 
|-zd144p1   230:145  0    46G  0 part              ext4                                               
|-zd144p2   230:146  0     1K  0 part                                                                 
`-zd144p5   230:149  0     4G  0 part              swap                                               
zd160       230:160  0    32G  0 disk                                                                 
|-zd160p1   230:161  0    31G  0 part              ext4                                               
|-zd160p2   230:162  0     1K  0 part                                                                 
`-zd160p5   230:165  0   975M  0 part              swap                                               
zd176       230:176  0    50G  0 disk                                                                 
|-zd176p1   230:177  0    46G  0 part              ext4                                               
|-zd176p2   230:178  0     1K  0 part                                                                 
`-zd176p5   230:181  0     4G  0 part              swap                                               
zd224       230:224  0    50G  0 disk                                                                 
|-zd224p1   230:225  0   9.3G  0 part              ext4                                               
|-zd224p2   230:226  0     1K  0 part                                                                 
|-zd224p5   230:229  0   9.3G  0 part              swap                                               
`-zd224p6   230:230  0  31.4G  0 part              ext4                                               
nvme3n1     259:0    0 894.3G  0 disk                                                                 MTFDKCC960TGP-1BK1DABYY
|-nvme3n1p1 259:2    0   511M  0 part              linux_raid_member                                 
| `-md1       9:1    0 510.9M  0 raid1 /boot/efi   vfat              EFI_SYSPART                     
|-nvme3n1p2 259:3    0     1G  0 part              linux_raid_member md2                             
| `-md2       9:2    0  1022M  0 raid1 /boot       ext4              boot                             
|-nvme3n1p3 259:4    0    20G  0 part              linux_raid_member md3                             
| `-md3       9:3    0    20G  0 raid1 /           ext4              root                             
|-nvme3n1p4 259:5    0     1G  0 part  [SWAP]      swap              swap-nvme1n1p4                   
`-nvme3n1p5 259:6    0 871.8G  0 part              zfs_member        data                             
nvme2n1     259:1    0 894.3G  0 disk                                                                 MTFDKCC960TGP-1BK1DABYY
|-nvme2n1p1 259:7    0   511M  0 part              linux_raid_member                                 
| `-md1       9:1    0 510.9M  0 raid1 /boot/efi   vfat              EFI_SYSPART                     
|-nvme2n1p2 259:8    0     1G  0 part              linux_raid_member md2                             
| `-md2       9:2    0  1022M  0 raid1 /boot       ext4              boot                             
|-nvme2n1p3 259:9    0    20G  0 part              linux_raid_member md3                             
| `-md3       9:3    0    20G  0 raid1 /           ext4              root                             
|-nvme2n1p4 259:10   0     1G  0 part  [SWAP]      swap              swap-nvme0n1p4                   
|-nvme2n1p5 259:11   0 871.8G  0 part              zfs_member        data                             
`-nvme2n1p6 259:12   0     2M  0 part              iso9660           config-2                         
nvme1n1     259:13   0   1.7T  0 disk                                                                 SAMSUNG MZQL21T9HCJR-00A07
|-nvme1n1p1 259:14   0   1.7T  0 part              zfs_member        DATA                             
`-nvme1n1p9 259:15   0     8M  0 part                                                                 
nvme0n1     259:16   0   1.7T  0 disk                                                                 SAMSUNG MZQL21T9HCJR-00A07
|-nvme0n1p1 259:17   0   1.7T  0 part              zfs_member        DATA                             
`-nvme0n1p9 259:18   0     8M  0 part


Code: 
an 23 09:12:33 pm12 pmxcfs[1569]: [dcdb] notice: data verification successful
Jan 23 10:00:15 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:00:16 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:00:24 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:00:24 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:00:24 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:00:25 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:01:15 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:01:15 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:01:15 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:01:21 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:01:24 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:01:26 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:07:44 pm12 pvestatd[1865]: status update time (6.226 seconds)
Jan 23 10:12:33 pm12 pmxcfs[1569]: [dcdb] notice: data verification successful
Jan 23 10:15:06 pm12 zed[1738320]: eid=10788 class=dio_verify_rd pool='data' size=131072 offset=512825257984 priority=0 err=0 flags=0x100280080 bookmark=387:667:0:1675081
Jan 23 10:15:25 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:30:26 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 10:45:52 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 11:01:01 pm12 systemd[1]: Starting apt-daily.service - Daily apt download activities...
Jan 23 11:01:03 pm12 systemd[1]: apt-daily.service: Deactivated successfully.
Jan 23 11:01:03 pm12 systemd[1]: Finished apt-daily.service - Daily apt download activities.
Jan 23 11:01:03 pm12 systemd[1]: apt-daily.service: Consumed 1.192s CPU time, 241.1M memory peak.
Jan 23 11:01:52 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 11:06:23 pm12 zed[1756271]: eid=10789 class=dio_verify_wr pool='data' size=131072 offset=549274644480 priority=1 err=5 flags=0x100200080 bookmark=387:144:0:1574116
Jan 23 11:12:33 pm12 pmxcfs[1569]: [dcdb] notice: data verification successful
Jan 23 11:38:09 pm12 zed[1767230]: eid=10790 class=dio_verify_rd pool='data' size=131072 offset=323452334080 priority=0 err=0 flags=0x100280080 bookmark=387:667:0:1674798
Jan 23 11:38:09 pm12 zed[1767231]: eid=10791 class=dio_verify_rd pool='data' size=131072 offset=323452465152 priority=0 err=0 flags=0x100280080 bookmark=387:667:0:1674799
Jan 23 11:38:09 pm12 zed[1767234]: eid=10792 class=dio_verify_rd pool='data' size=131072 offset=302719049728 priority=0 err=0 flags=0x100280080 bookmark=387:667:0:1674585
Jan 23 11:38:09 pm12 zed[1767238]: eid=10793 class=dio_verify_rd pool='data' size=131072 offset=318564712448 priority=0 err=0 flags=0x100280080 bookmark=387:667:0:1762356
Jan 23 11:44:42 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 11:44:46 pm12 pmxcfs[1569]: [status] notice: received log
Jan 23 11:44:51 pm12 pmxcfs[1569]: [status] notice: received log
 

Attachments

  • 2026-01-23 11_45_17-Greenshot.png
    2026-01-23 11_45_17-Greenshot.png
    4.1 KB · Views: 5
  • Like
Reactions: michalk
Code: 
  pool: data
 state: ONLINE
  scan: scrub repaired 0B in 00:05:25 with 0 errors on Thu Jan 22 16:26:36 2026
config:

        NAME                                                 STATE     READ WRITE CKSUM   DIO
        data                                                 ONLINE       0     0     0     0
          mirror-0                                           ONLINE       0     0     0 10.5K
            nvme-eui.000000000000000100a075254f178769-part5  ONLINE       0     0     0     0
            nvme-eui.000000000000000100a075254f178767-part5  ONLINE       0     0     0     0

errors: No known data errors
 
There's already a thread about this
T

Thread 'Proxmox 9 - IO error ZFS'

Hi everyone,

We’ve been deploying several new Proxmox 9 nodes using ZFS as the primary storage, and we’re encountering issues where virtual machines become I/O locked.

When it happens, the VMs are paused with an I/O error. We’re aware this can occur when a host runs out of disk space, but in our case there is plenty of free storage available.
We’ve seen this behavior across multiple hosts, different clusters, and different hardware platforms.

Furthermore, we’ve been running ZFS on Proxmox 8 without any issues, but since these problems started with our Proxmox 9 installations, we’re...
 
@Impact Apologies if this is not the correct process, I was trying to separate things.

Should I delete it/and or add to the existing?
 
I'm not a moderator but since it's basically the same issue I'd stick to the one thread for now.
 
You must log in or register to reply here.
Top Bottom
Back