Is this a ZFS related problem?

C

cucumberjuice66

New Member
Mar 15, 2025
16
0
1
Hello fellow Proxmox enjoyers,

I have been running 3 Proxmox servers for a while now but having issues with IO pressure stalls and IO delay on two of my servers running ZFS.
Almost every tutorial recommends using ZFS. I assume most home labbers run ZFS on consumer grade SSD's which is not recommended.
The most recent server I setup uses LVM only and it seems like the IO pressure stall are better but I still saw spikes of roughly 20%.

But one of the servers even became completely unresponsive to the point I had to hard reset. I can pretty much recreate the scenario by running a Ubuntu Server VM with a Docker container where a lot of file transfers and download activity is going on i.e. disk activity
After the reboot I checked my syslog and saw these error messages:
Code: 
task zvol_tq-2:1325994 blocked for more than 122 seconds.
So I assume it has to do with the ZFS file system being overwhelmed. My Proxmox installation is on very cheap 120gig Intenso SSDs in ZFS with no DRAM cache.
But my actual Ubuntu Server VM is running on a Sandisk SSD Plus 480gig model. If the ZFS file system is the issue I would assume only the VM dies and not my whole system.
I also found this on github which seems related to what I experienced https://github.com/openzfs/zfs/issues/18200
Can anyone explain to me what I can do to mitigate these issues? If I run LVM I would like some redundancy aka raid but it does not seem as trivial

Any help is appreciated.
 
Last edited:
Your problem is almost certainly the Intenso drives, not ZFS itself. Those are DRAM-less budget SSDs and ZFS absolutely hammers them, once the small SLC cache fills up during sustained writes, latency goes through the roof, the pool stalls, and you get exactly those "task blocked for 122 seconds" errors. And since your Proxmox root is on that same pool, the whole host locks up, not just the VM.

Quick way to confirm: run zpool iostat -v 1 and iostat -x 1 while reproducing the load. If await times are in the hundreds of ms and util is pinned at 100%, that's your answer.

Real fix is better drives, anything with DRAM cache, or ideally used enterprise SSDs off eBay (Intel S3610, Samsung PM883 etc), they're cheap and make ZFS behave completely differently. Band-aids if you want to keep the Intensos: atime=off, cap the ARC, maybe lower zfs_dirty_data_max.

For LVM with raid: mdadm RAID1 underneath LVM works fine with Proxmox, it's just not in the installer GUI, you have to set it up via the Debian installer or by hand. But honestly ZFS mirror on decent drives is less hassle in the long run.
 
  • Like
Reactions: cucumberjuice66, Johannes S and Kingneutron
Nemesiz said:
Hi, looking at given issue link try to turn off swap
Click to expand...
Hi where would i do that?

Impact said:
Please share
Bash: 
lsblk -o+FSTYPE,LABEL,MODEL
cat /etc/pve/storage.cfg
qm config PROBLEMATICVMIDHERE
swapon
zarcsummary -s arc
I'd also look into this.
Click to expand...

@Impact swapon command does not give me an ouput. Unfortunately I already deleted the VM that had the issues. But the Windows 10 VM also has issues where it locks up for short periods. I knew that link but it's little tough to troubleshoot.

Code: 
sdb        8:16   0 111.8G  0 disk                                    INTENSO SSD
├─sdb1     8:17   0  1007K  0 part                                  
├─sdb2     8:18   0     1G  0 part             vfat                  
└─sdb3     8:19   0   110G  0 part             zfs_member rpool      
sdc        8:32   0 111.8G  0 disk                                    INTENSO SSD
├─sdc1     8:33   0  1007K  0 part                                  
├─sdc2     8:34   0     1G  0 part             vfat                  
└─sdc3     8:35   0   110G  0 part             zfs_member rpool      
sdd        8:48   0 447.1G  0 disk                                    SSD PLUS 480GB
├─sdd1     8:49   0 447.1G  0 part             zfs_member vmdata    
└─sdd9     8:57   0     8M  0 part

Code: 
root@proxmox:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,import,backup,iso,images
        shared 0

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

zfspool: vmdata
        pool vmdata
        content rootdir,images
        mountpoint /vmdata
        nodes proxmox

Code: 
agent: 1
allow-ksm: 0
args: -device ivshmem-plain,memdev=ivshmem,bus=pcie.0 -object memory-backend-file,id=ivshmem,share=on,mem-path=/dev/shm/looking-glass,size=32M
balloon: 0
bios: ovmf
boot: order=scsi0;ide0
cores: 2
cpu: host,hidden=1
efidisk0: vmdata:vm-100-disk-0,efitype=4m,ms-cert=2023k,pre-enrolled-keys=1,size=1M
hostpci0: 0000:01:00,pcie=1,romfile=970.rom,x-vga=1
ide0: none,media=cdrom
machine: pc-q35-10.1
memory: 16384
meta: creation-qemu=10.1.2,ctime=1771871216
name: Win10
net0: virtio=BC:24:11:08:3F:84,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: vmdata:vm-100-disk-1,backup=0,iothread=1,size=200G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=b508958b-f669-49d7-8674-37e537402517
sockets: 1
usb4: host=1-5.4
usb5: host=1-5.3
vmgenid: 8c23e16c-f0d8-4b97-8e84-74de2b4c5bfb

Code: 
------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Aug 21 11:46:53 2026
Linux 7.0.14-8-pve                                            2.4.3-pve1
Machine: proxmox (x86_64)                                      2.4.3-pve1

ARC status:
        Total memory size:                                      31.3 GiB
        Min target size:                               3.1 %  1001.1 MiB
        Max target size:                               10.0 %    3.1 GiB
        Target size (adaptive):                       12.6 %  1001.1 MiB
        Current size:                                  12.6 %  404.8 MiB
        Free memory size:                                       23.8 GiB
        Available memory size:                                  22.7 GiB

ARC structural breakdown (current size):                       404.8 MiB
        Compressed size:                               81.7 %  330.7 MiB
        Overhead size:                                 14.1 %   57.2 MiB
        Bonus size:                                     0.5 %    2.0 MiB
        Dnode size:                                     1.5 %    6.1 MiB
        Dbuf size:                                      0.7 %    3.0 MiB
        Header size:                                    1.2 %    4.7 MiB
        L2 header size:                                 0.0 %    0 Bytes
        ABD chunk waste size:                           0.3 %    1.1 MiB

ARC types breakdown (compressed + overhead):                   387.9 MiB
        Data size:                                     86.7 %  336.4 MiB
        Metadata size:                                 13.3 %   51.5 MiB

ARC states breakdown (compressed + overhead):                  387.9 MiB
        Anonymous data size:                          < 0.1 %  136.0 KiB
        Anonymous metadata size:                        0.1 %  257.5 KiB
        MFU data target:                               37.5 %  145.5 MiB
        MFU data size:                                 16.9 %   65.5 MiB
        MFU evictable data size:                       13.7 %   53.3 MiB
        MFU ghost data size:                                     0 Bytes
        MFU metadata target:                           12.5 %   48.5 MiB
        MFU metadata size:                              3.6 %   13.8 MiB
        MFU evictable metadata size:                    0.9 %    3.6 MiB
        MFU ghost metadata size:                                 0 Bytes
        MRU data target:                               37.5 %  145.5 MiB
        MRU data size:                                 69.8 %  270.8 MiB
        MRU evictable data size:                       65.5 %  253.9 MiB
        MRU ghost data size:                                     0 Bytes
        MRU metadata target:                           12.5 %   48.5 MiB
        MRU metadata size:                              9.6 %   37.4 MiB
        MRU evictable metadata size:                    1.7 %    6.4 MiB
        MRU ghost metadata size:                                 0 Bytes
        Uncached data size:                             0.0 %    0 Bytes
        Uncached metadata size:                         0.0 %    0 Bytes

ARC hash breakdown:
        Elements:                                                  20.3k
        Collisions:                                                   58
        Chain max:                                                     1
        Chains:                                                       42

ARC misc:
        Uncompressed size:                            177.7 %  587.6 MiB
        Memory throttles:                                              0
        Memory direct reclaims:                                        0
        Memory indirect reclaims:                                      0
        Deleted:                                                      33
        Mutex misses:                                                  0
        Eviction skips:                                              801
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 552.5 KiB
        L2 eligible MFU evictions:                      0.0 %    0 Bytes
        L2 eligible MRU evictions:                    100.0 %  552.5 KiB
        L2 ineligible evictions:                                 4.0 KiB

Alban24x7 said:
Your problem is almost certainly the Intenso drives, not ZFS itself. Those are DRAM-less budget SSDs and ZFS absolutely hammers them, once the small SLC cache fills up during sustained writes, latency goes through the roof, the pool stalls, and you get exactly those "task blocked for 122 seconds" errors. And since your Proxmox root is on that same pool, the whole host locks up, not just the VM.

Quick way to confirm: run zpool iostat -v 1 and iostat -x 1 while reproducing the load. If await times are in the hundreds of ms and util is pinned at 100%, that's your answer.

Real fix is better drives, anything with DRAM cache, or ideally used enterprise SSDs off eBay (Intel S3610, Samsung PM883 etc), they're cheap and make ZFS behave completely differently. Band-aids if you want to keep the Intensos: atime=off, cap the ARC, maybe lower zfs_dirty_data_max.

For LVM with raid: mdadm RAID1 underneath LVM works fine with Proxmox, it's just not in the installer GUI, you have to set it up via the Debian installer or by hand. But honestly ZFS mirror on decent drives is less hassle in the long run
Click to expand...
@Alban24x7 I know I mostly wrote about the Intenso drives on one server but I also experienced the issue on my second server with two CrucialMX500 that have DRAM cache. I see pretty high spikes of iodelay and iopressure stalling. See screenshots
spikes.png
iodelay.png
But if my VMs don't even sit on the Intenso drives but instead on the SSD Plus 480GB with DRAM cache why would the Intenso drives get hammered more than the actual drive that hosts the VM? I need to create a scenario again where I can trigger the issue. I will come back to the thread then.

Also with the current SSD crisis the prices I could find are not really great. Would it be enough to get just two small (200-400GB) enterprise SSDs but keep my VMs on consumer grade SSDs?
 
I just turned on my windows 10 VM and did a big file copy from my passed through HDD (D: drive) to my SSD (C: drive) (SSD Plus 480GB). I also attached the output of zpool iostat -v 1.
wa% during the file copy jumped to 26.4 and it hovers around 0,5 - 13,5% on average.

Code: 
top - 12:23:34 up 45 min,  3 users,  load average: 1.06, 0.60, 0.25
Tasks: 273 total,   6 running, 267 sleeping,   0 stopped,   0 zombie
%Cpu(s): 24.6 us, 29.6 sy,  0.0 ni, 19.0 id, 26.4 wa,  0.0 hi,  0.4 si,  0.0 st
MiB Mem :  32034.9 total,  10500.5 free,  21681.1 used,    293.6 buff/cache     
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  10353.8 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                               
   9198 root      20   0   17.2g  16.1g  39188 R 135.2  51.4   2:41.36 kvm                                                                                   
   9950 root       1 -19       0      0      0 R   3.7   0.0   0:00.16 z_wr_iss                                                                               
    878 root       1 -19       0      0      0 R   3.3   0.0   0:01.20 z_wr_iss                                                                               
   1914 root       1 -19       0      0      0 D   3.3   0.0   0:01.17 z_wr_iss                                                                               
    225 root      20   0       0      0      0 S   1.7   0.0   0:00.15 arc_evict                                                                             
    879 root       0 -20       0      0      0 R   1.3   0.0   0:00.41 z_wr_int                                                                               
    228 root      39  19       0      0      0 S   1.0   0.0   0:00.45 dbuf_evict                                                                             
    895 root      39  19       0      0      0 S   1.0   0.0   0:00.34 dp_sync_taskq                                                                         
   9319 root       0 -20       0      0      0 R   1.0   0.0   0:00.28 zvol_tq-0                                                                             
   9951 root       0 -20       0      0      0 D   1.0   0.0   0:00.05 z_wr_int                                                                               
   9959 root       0 -20       0      0      0 D   1.0   0.0   0:00.05 z_wr_int                                                                               
   9985 root       0 -20       0      0      0 S   1.0   0.0   0:00.05 zvol_tq-0                                                                             
    222 root       0 -20       0      0      0 S   0.7   0.0   0:00.38 zvol_tq-0                                                                             
   9489 root       0 -20       0      0      0 S   0.7   0.0   0:00.13 zvol_tq-0                                                                             
    877 root       0 -20       0      0      0 S   0.3   0.0   0:00.79 z_rd_int                                                                               
   9303 root      20   0   10440   4592   2404 R   0.3   0.0   0:00.31 top                                                                                   
   9349 root       0 -20       0      0      0 S   0.3   0.0   0:00.28 zvol_tq-0                                                                             
   9984 root       0 -20       0      0      0 S   0.3   0.0   0:00.04 zvol_tq-0                                                                             
      1 root      20   0   24476  11972   7432 S   0.0   0.0   0:01.11 systemd                                                                               
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.02 kthreadd                                                                               
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_release                                                                 
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_gp                                                                       
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-sync_wq                                                                     
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-kvfree_rcu_recla
 

Attachments

You must log in or register to reply here.
Top Bottom
Back