[ZFS] SSD High I/O (Standalone Installation)

E

elmemis

Renowned Member
Jan 23, 2012
43
0
71
Hi,

I have a Proxmox standalone installation and have high I/O when working in any instance running in raid ssd.

Code: 
proxmox-ve: 6.1-2 (running kernel: 5.0.21-1-pve)
pve-manager: 6.1-3 (running version: 6.1-3/37248ce6)
pve-kernel-5.3: 6.0-12
pve-kernel-helper: 6.0-12
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ifupdown2: residual config
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-14
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-2
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

Code: 
~# zpool status
  pool: nvme-zfs
state: ONLINE
  scan: scrub repaired 0B in 0 days 00:09:01 with 0 errors on Sun Dec  8 00:33:02 2019
config:

    NAME                                                STATE     READ WRITE CKSUM
    nvme-zfs                                            ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        nvme-SAMSUNG_MZQLB3T8HALS-000AZ_S3VJNF0K700531  ONLINE       0     0     0
        nvme-eui.33564a304b7005300025384600000001       ONLINE       0     0     0

errors: No known data errors

  pool: rpool
state: ONLINE
  scan: scrub repaired 0B in 0 days 00:02:11 with 0 errors on Sun Dec  8 00:26:13 2019
config:

    NAME                                             STATE     READ WRITE CKSUM
    rpool                                            ONLINE       0     0     0
      mirror-0                                       ONLINE       0     0     0
        ata-HP_SSD_S700_500GB_HBSA39194101315-part3  ONLINE       0     0     0
        ata-HP_SSD_S700_500GB_HBSA39194102188-part3  ONLINE       0     0     0

errors: No known data errors

  pool: ssd-zfs
state: ONLINE
  scan: scrub repaired 0B in 0 days 00:15:51 with 0 errors on Sun Dec  8 00:39:54 2019
config:

    NAME        STATE     READ WRITE CKSUM
    ssd-zfs     ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        sda     ONLINE       0     0     0
        sdb     ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0

errors: No known data errors

Code: 
~# zfs get all ssd-zfs
NAME     PROPERTY              VALUE                  SOURCE
ssd-zfs  type                  filesystem             -
ssd-zfs  creation              Sat Aug 31 15:04 2019  -
ssd-zfs  used                  696G                   -
ssd-zfs  available             4.43T                  -
ssd-zfs  referenced            232K                   -
ssd-zfs  compressratio         1.16x                  -
ssd-zfs  mounted               yes                    -
ssd-zfs  quota                 none                   default
ssd-zfs  reservation           none                   default
ssd-zfs  recordsize            128K                   default
ssd-zfs  mountpoint            /ssd-zfs               default
ssd-zfs  sharenfs              off                    default
ssd-zfs  checksum              on                     default
ssd-zfs  compression           on                     local
ssd-zfs  atime                 on                     default
ssd-zfs  devices               on                     default
ssd-zfs  exec                  on                     default
ssd-zfs  setuid                on                     default
ssd-zfs  readonly              off                    default
ssd-zfs  zoned                 off                    default
ssd-zfs  snapdir               hidden                 default
ssd-zfs  aclinherit            restricted             default
ssd-zfs  createtxg             1                      -
ssd-zfs  canmount              on                     default
ssd-zfs  xattr                 on                     default
ssd-zfs  copies                1                      default
ssd-zfs  version               5                      -
ssd-zfs  utf8only              off                    -
ssd-zfs  normalization         none                   -
ssd-zfs  casesensitivity       sensitive              -
ssd-zfs  vscan                 off                    default
ssd-zfs  nbmand                off                    default
ssd-zfs  sharesmb              off                    default
ssd-zfs  refquota              none                   default
ssd-zfs  refreservation        none                   default
ssd-zfs  guid                  7483453056672396458    -
ssd-zfs  primarycache          all                    default
ssd-zfs  secondarycache        all                    default
ssd-zfs  usedbysnapshots       0B                     -
ssd-zfs  usedbydataset         232K                   -
ssd-zfs  usedbychildren        696G                   -
ssd-zfs  usedbyrefreservation  0B                     -
ssd-zfs  logbias               latency                default
ssd-zfs  objsetid              54                     -
ssd-zfs  dedup                 off                    default
ssd-zfs  mlslabel              none                   default
ssd-zfs  sync                  standard               default
ssd-zfs  dnodesize             legacy                 default
ssd-zfs  refcompressratio      1.00x                  -
ssd-zfs  written               232K                   -
ssd-zfs  logicalused           605G                   -
ssd-zfs  logicalreferenced     71K                    -
ssd-zfs  volmode               default                default
ssd-zfs  filesystem_limit      none                   default
ssd-zfs  snapshot_limit        none                   default
ssd-zfs  filesystem_count      none                   default
ssd-zfs  snapshot_count        none                   default
ssd-zfs  snapdev               hidden                 default
ssd-zfs  acltype               off                    default
ssd-zfs  context               none                   default
ssd-zfs  fscontext             none                   default
ssd-zfs  defcontext            none                   default
ssd-zfs  rootcontext           none                   default
ssd-zfs  relatime              off                    default
ssd-zfs  redundant_metadata    all                    default
ssd-zfs  overlay               off                    default
ssd-zfs  encryption            off                    default
ssd-zfs  keylocation           none                   default
ssd-zfs  keyformat             none                   default
ssd-zfs  pbkdf2iters           0                      default
ssd-zfs  special_small_blocks  0                      default

Code: 
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Seagate BarraCud
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9C37C950-C19E-4145-9011-49BF38D7424C

Device          Start        End    Sectors  Size Type
/dev/sda1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sda9  3907012608 3907028991      16384    8M Solaris reserved 1


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Seagate BarraCud
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 522CA783-B889-A944-95C7-69A0F6723C27

Device          Start        End    Sectors  Size Type
/dev/sdb1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdb9  3907012608 3907028991      16384    8M Solaris reserved 1


Disk /dev/sdc: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Seagate BarraCud
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F8EA8BCC-32B3-2249-8612-179F28A1C508

Device          Start        End    Sectors  Size Type
/dev/sdc1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdc9  3907012608 3907028991      16384    8M Solaris reserved 1


Disk /dev/sdd: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Seagate BarraCud
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 8DBEE967-6382-2744-B32C-8954C83A7F2E

Device          Start        End    Sectors  Size Type
/dev/sdd1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdd9  3907012608 3907028991      16384    8M Solaris reserved 1

When I'm working in any VM/lxc running in the raid ssd, the VM freeze for high I/O. It only happens in the SSD raid. It freezes for a few seconds or minutes, after then it works normal for a while.

1577332872176.png

1577333169071.png

Server Specs (CPU, Memory total/usage):
1577333432009.png
 
zfs list ssd-zfs
Code: 
ssd-zfs                      696G  4.43T      232K  /ssd-zfs
ssd-zfs/subvol-100-disk-0   1.08G  28.9G     1.08G  /ssd-zfs/subvol-100-disk-0
ssd-zfs/subvol-102-disk-0    931M  7.09G      931M  /ssd-zfs/subvol-102-disk-0
ssd-zfs/subvol-105-disk-0   3.22G  16.8G     3.22G  /ssd-zfs/subvol-105-disk-0
ssd-zfs/subvol-106-disk-0   7.17G  42.8G     7.17G  /ssd-zfs/subvol-106-disk-0
ssd-zfs/subvol-114-disk-0    795M  15.2G      795M  /ssd-zfs/subvol-114-disk-0
ssd-zfs/subvol-135-disk-0   1.07G  98.9G     1.07G  /ssd-zfs/subvol-135-disk-0
ssd-zfs/subvol-202-disk-0   8.96G  16.0G     8.96G  /ssd-zfs/subvol-202-disk-0
ssd-zfs/subvol-203-disk-0   11.4G  13.6G     11.4G  /ssd-zfs/subvol-203-disk-0
ssd-zfs/subvol-205-disk-0    112G  38.2G      112G  /ssd-zfs/subvol-205-disk-0
ssd-zfs/subvol-207-disk-0    896M  19.1G      896M  /ssd-zfs/subvol-207-disk-0
ssd-zfs/vm-101-disk-0       22.4G  4.43T     22.4G  -
ssd-zfs/vm-103-disk-0       2.20G  4.43T     2.20G  -
ssd-zfs/vm-104-disk-0       13.2G  4.43T     13.2G  -
ssd-zfs/vm-107-disk-0       59.3G  4.43T     59.3G  -
ssd-zfs/vm-108-disk-0       23.6G  4.43T     23.6G  -
ssd-zfs/vm-109-disk-0       14.5G  4.43T     14.5G  -
ssd-zfs/vm-112-disk-0       7.65G  4.43T     7.65G  -
ssd-zfs/vm-112-disk-1        293G  4.43T      293G  -
ssd-zfs/vm-112-disk-2       81.4K  4.43T     81.4K  -
ssd-zfs/vm-112-disk-3       81.4K  4.43T     81.4K  -
ssd-zfs/vm-112-disk-4       5.82G  4.43T     5.82G  -
ssd-zfs/vm-113-disk-0       4.12G  4.43T     4.12G  -
ssd-zfs/vm-119-disk-0       22.9G  4.43T     22.9G  -
ssd-zfs/vm-120-disk-0       2.89G  4.43T     2.89G  -
ssd-zfs/vm-124-disk-0       5.77G  4.43T     5.77G  -
ssd-zfs/vm-128-disk-0       19.0G  4.43T     19.0G  -
ssd-zfs/vm-129-disk-0       4.55G  4.43T     4.55G  -
ssd-zfs/vm-130-disk-0       13.0G  4.43T     13.0G  -
ssd-zfs/vm-131-disk-0       12.1G  4.43T     12.1G  -
ssd-zfs/vm-132-disk-0       1.79G  4.43T     1.79G  -
ssd-zfs/vm-133-disk-0       5.86G  4.43T     5.86G  -
ssd-zfs/vm-134-disk-0       2.37G  4.43T     2.37G  -
ssd-zfs/vm-206-disk-0       69.0M  4.43T     69.0M  -
ssd-zfs/vm-208-disk-0       3.09G  4.43T     3.09G  -
ssd-zfs/vm-209-disk-0       3.16G  4.43T     3.16G  -
ssd-zfs/vm-210-disk-0       3.17G  4.43T     3.17G  -
ssd-zfs/vm-211-disk-0       2.91G  4.43T     2.91G  -
 
Code: 
ZFS Subsystem Report                            Wed Dec 25 23:15:04 2019
Linux 5.0.21-1-pve                                            0.8.1-pve2
Machine: pve-us (x86_64)                                      0.8.1-pve2

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    97.2 %   31.1 GiB
        Target size (adaptive):                        97.3 %   31.2 GiB
        Min size (hard limit):                         12.5 %    4.0 GiB
        Max size (high water):                            8:1   32.0 GiB
        Most Frequently Used (MFU) cache size:         63.9 %   18.2 GiB
        Most Recently Used (MRU) cache size:           36.1 %   10.3 GiB
        Metadata cache size (hard limit):              75.0 %   24.0 GiB
        Metadata cache size (current):                 17.0 %    4.1 GiB
        Dnode cache size (hard limit):                 10.0 %    2.4 GiB
        Dnode cache size (current):                    34.4 %  845.1 MiB

ARC hash breakdown:
        Elements max:                                               8.4M
        Elements current:                              49.7 %       4.2M
        Collisions:                                               504.1M
        Chain max:                                                     6
        Chains:                                                   239.8k

ARC misc:
        Deleted:                                                    2.0G
        Mutex misses:                                             175.9k
        Eviction skips:                                            45.2M

ARC total accesses (hits + misses):                                54.1G
        Cache hit ratio:                               96.5 %      52.3G
        Cache miss ratio:                               3.5 %       1.9G
        Actual hit ratio (MFU + MRU hits):             96.5 %      52.2G
        Data demand efficiency:                        99.5 %      33.0G
        Data prefetch efficiency:                       2.4 %       1.7G

Cache hits by cache type:
        Most frequently used (MFU):                    67.1 %      35.1G
        Most recently used (MRU):                      32.9 %      17.2G
        Most frequently used (MFU) ghost:             < 0.1 %       9.7M
        Most recently used (MRU) ghost:                 0.1 %      36.5M

Cache hits by data type:
        Demand data:                                   62.8 %      32.8G
        Demand prefetch data:                           0.1 %      42.3M
        Demand metadata:                               37.1 %      19.4G
        Demand prefetch metadata:                     < 0.1 %       4.2M

Cache misses by data type:
        Demand data:                                    8.5 %     158.8M
        Demand prefetch data:                          91.0 %       1.7G
        Demand metadata:                                0.4 %       6.6M
        Demand prefetch metadata:                       0.2 %       3.4M

DMU prefetch efficiency:                                           15.8G
        Hit ratio:                                      6.7 %       1.1G
        Miss ratio:                                    93.3 %      14.7G

L2ARC not detected, skipping section

Solaris Porting Layer (SPL):
        spl_hostid                                                     0
        spl_hostid_path                                      /etc/hostid
        spl_kmem_alloc_max                                       1048576
        spl_kmem_alloc_warn                                        65536
        spl_kmem_cache_expire                                          2
        spl_kmem_cache_kmem_limit                                   2048
        spl_kmem_cache_kmem_threads                                    4
        spl_kmem_cache_magazine_size                                   0
        spl_kmem_cache_max_size                                       32
        spl_kmem_cache_obj_per_slab                                    8
        spl_kmem_cache_obj_per_slab_min                                1
        spl_kmem_cache_reclaim                                         0
        spl_kmem_cache_slab_limit                                  16384
        spl_max_show_tasks                                           512
        spl_panic_halt                                                 0
        spl_taskq_kick                                                 0
        spl_taskq_thread_bind                                          0
        spl_taskq_thread_dynamic                                       1
        spl_taskq_thread_priority                                      1
        spl_taskq_thread_sequential                                    4
 
Code: 
Tunables:
dbuf_cache_hiwater_pct                                        10
dbuf_cache_lowater_pct                                        10
dbuf_cache_max_bytes                                  1073741824
dbuf_cache_shift                                               5
dbuf_metadata_cache_max_bytes                          536870912
dbuf_metadata_cache_shift                                      6
dmu_object_alloc_chunk_shift                                   7
ignore_hole_birth                                              1
l2arc_feed_again                                               1
l2arc_feed_min_ms                                            200
l2arc_feed_secs                                                1
l2arc_headroom                                                 2
l2arc_headroom_boost                                         200
l2arc_noprefetch                                               1
l2arc_norw                                                     0
l2arc_write_boost                                        8388608
l2arc_write_max                                          8388608
metaslab_aliquot                                          524288
metaslab_bias_enabled                                          1
metaslab_debug_load                                            0
metaslab_debug_unload                                          0
metaslab_force_ganging                                  16777217
metaslab_fragmentation_factor_enabled                          1
metaslab_lba_weighting_enabled                                 1
metaslab_preload_enabled                                       1
send_holes_without_birth_time                                  1
spa_asize_inflation                                           24
spa_config_path                             /etc/zfs/zpool.cache
spa_load_print_vdev_tree                                       0
spa_load_verify_data                                           1
spa_load_verify_maxinflight                                10000
spa_load_verify_metadata                                       1
spa_slop_shift                                                 5
vdev_removal_max_span                                      32768
vdev_validate_skip                                             0
zfetch_array_rd_sz                                       1048576
zfetch_max_distance                                      8388608
zfetch_max_streams                                             8
zfetch_min_sec_reap                                            2
 
Code: 
zfs_abd_scatter_enabled                                        1
zfs_abd_scatter_max_order                                     10
zfs_abd_scatter_min_size                                    1536
zfs_admin_snapshot                                             0
zfs_arc_average_blocksize                                   8192
zfs_arc_dnode_limit                                            0
zfs_arc_dnode_limit_percent                                   10
zfs_arc_dnode_reduce_percent                                  10
zfs_arc_grow_retry                                             0
zfs_arc_lotsfree_percent                                      10
zfs_arc_max                                          34359738368
zfs_arc_meta_adjust_restarts                                4096
zfs_arc_meta_limit                                             0
zfs_arc_meta_limit_percent                                    75
zfs_arc_meta_min                                               0
zfs_arc_meta_prune                                         10000
zfs_arc_meta_strategy                                          1
zfs_arc_min                                           4294967296
zfs_arc_min_prefetch_ms                                        0
zfs_arc_min_prescient_prefetch_ms                              0
zfs_arc_p_dampener_disable                                     1
zfs_arc_p_min_shift                                            0
zfs_arc_pc_percent                                             0
zfs_arc_shrink_shift                                           0
zfs_arc_sys_free                                               0
zfs_async_block_max_blocks                                100000
zfs_autoimport_disable                                         1
zfs_checksum_events_per_second                                20
zfs_commit_timeout_pct                                         5
zfs_compressed_arc_enabled                                     1
zfs_condense_indirect_commit_entry_delay_ms                    0
zfs_condense_indirect_vdevs_enable                             1
zfs_condense_max_obsolete_bytes                       1073741824
zfs_condense_min_mapping_bytes                            131072
zfs_dbgmsg_enable                                              1
zfs_dbgmsg_maxsize                                       4194304
zfs_dbuf_state_index                                           0
zfs_ddt_data_is_special                                        1
zfs_deadman_checktime_ms                                   60000
zfs_deadman_enabled                                            1
zfs_deadman_failmode                                        wait
zfs_deadman_synctime_ms                                   600000
zfs_deadman_ziotime_ms                                    300000
zfs_dedup_prefetch                                             0
zfs_delay_min_dirty_percent                                   60
zfs_delay_scale                                           500000
zfs_delete_blocks                                          20480
zfs_dirty_data_max                                    4294967296
zfs_dirty_data_max_max                                4294967296
zfs_dirty_data_max_max_percent                                25
zfs_dirty_data_max_percent                                    10
zfs_dirty_data_sync_percent                                   20
zfs_disable_ivset_guid_check                                   0
zfs_dmu_offset_next_sync                                       0
zfs_expire_snapshot                                          300
zfs_flags                                                      0
zfs_free_bpobj_enabled                                         1
zfs_free_leak_on_eio                                           0
zfs_free_min_time_ms                                        1000
zfs_immediate_write_sz                                     32768
zfs_initialize_value                        16045690984833335022
zfs_key_max_salt_uses                                  400000000
zfs_lua_max_instrlimit                                 100000000
zfs_lua_max_memlimit                                   104857600
zfs_max_missing_tvds                                           0
zfs_max_recordsize                                       1048576
zfs_metaslab_fragmentation_threshold                          70
zfs_metaslab_segment_weight_enabled                            1
zfs_metaslab_switch_threshold                                  2
zfs_mg_fragmentation_threshold                                85
zfs_mg_noalloc_threshold                                       0
zfs_multihost_fail_intervals                                  10
zfs_multihost_history                                          0
zfs_multihost_import_intervals                                20
zfs_multihost_interval                                      1000
zfs_multilist_num_sublists                                     0
zfs_no_scrub_io                                                0
zfs_no_scrub_prefetch                                          0
zfs_nocacheflush                                               0
zfs_nopwrite_enabled                                           1
zfs_object_mutex_size                                         64
zfs_obsolete_min_time_ms                                     500
zfs_override_estimate_recordsize                               0
zfs_pd_bytes_max                                        52428800
zfs_per_txg_dirty_frees_percent                                5
zfs_prefetch_disable                                           0
zfs_read_chunk_size                                      1048576
zfs_read_history                                               0
zfs_read_history_hits                                          0
zfs_reconstruct_indirect_combinations_max                   4096
zfs_recover                                                    0
zfs_recv_queue_length                                   16777216
zfs_removal_ignore_errors                                      0
zfs_removal_suspend_progress                                   0
zfs_remove_max_segment                                  16777216
zfs_resilver_disable_defer                                     0
zfs_resilver_min_time_ms                                    3000
zfs_scan_checkpoint_intval                                  7200
zfs_scan_fill_weight                                           3
zfs_scan_ignore_errors                                         0
zfs_scan_issue_strategy                                        0
zfs_scan_legacy                                                0
zfs_scan_max_ext_gap                                     2097152
zfs_scan_mem_lim_fact                                         20
zfs_scan_mem_lim_soft_fact                                    20
zfs_scan_strict_mem_lim                                        0
zfs_scan_suspend_progress                                      0
zfs_scan_vdev_limit                                      4194304
zfs_scrub_min_time_ms                                       1000
zfs_send_corrupt_data                                          0
zfs_send_queue_length                                   16777216
zfs_send_unmodified_spill_blocks                               1
zfs_slow_io_events_per_second                                 20
zfs_spa_discard_memory_limit                            16777216
zfs_special_class_metadata_reserve_pct                        25
zfs_sync_pass_deferred_free                                    2
zfs_sync_pass_dont_compress                                    5
zfs_sync_pass_rewrite                                          2
zfs_sync_taskq_batch_pct                                      75
zfs_trim_extent_bytes_max                              134217728
zfs_trim_extent_bytes_min                                  32768
zfs_trim_metaslab_skip                                         0
zfs_trim_queue_limit                                          10
zfs_trim_txg_batch                                            32
zfs_txg_history                                              100
zfs_txg_timeout                                                5
zfs_unlink_suspend_progress                                    0
zfs_user_indirect_is_special                                   1
 
Code: 
zfs_vdev_aggregate_trim                                        0
zfs_vdev_aggregation_limit                               1048576
zfs_vdev_aggregation_limit_non_rotating                   131072
zfs_vdev_async_read_max_active                                 3
zfs_vdev_async_read_min_active                                 1
zfs_vdev_async_write_active_max_dirty_percent                 60
zfs_vdev_async_write_active_min_dirty_percent                 30
zfs_vdev_async_write_max_active                               10
zfs_vdev_async_write_min_active                                2
zfs_vdev_cache_bshift                                         16
zfs_vdev_cache_max                                         16384
zfs_vdev_cache_size                                            0
zfs_vdev_default_ms_count                                    200
zfs_vdev_initializing_max_active                               1
zfs_vdev_initializing_min_active                               1
zfs_vdev_max_active                                         1000
zfs_vdev_min_ms_count                                         16
zfs_vdev_mirror_non_rotating_inc                               0
zfs_vdev_mirror_non_rotating_seek_inc                          1
zfs_vdev_mirror_rotating_inc                                   0
zfs_vdev_mirror_rotating_seek_inc                              5
zfs_vdev_mirror_rotating_seek_offset                     1048576
zfs_vdev_ms_count_limit                                   131072
zfs_vdev_queue_depth_pct                                    1000
zfs_vdev_raidz_implcycle [fastest] original scalar sse2 ssse3 avx2
zfs_vdev_read_gap_limit                                    32768
zfs_vdev_removal_max_active                                    2
zfs_vdev_removal_min_active                                    1
zfs_vdev_scheduler                                          noop
zfs_vdev_scrub_max_active                                      2
zfs_vdev_scrub_min_active                                      1
zfs_vdev_sync_read_max_active                                 10
zfs_vdev_sync_read_min_active                                 10
zfs_vdev_sync_write_max_active                                10
zfs_vdev_sync_write_min_active                                10
zfs_vdev_trim_max_active                                       2
zfs_vdev_trim_min_active                                       1
zfs_vdev_write_gap_limit                                    4096
zfs_zevent_cols                                               80
zfs_zevent_console                                             0
zfs_zevent_len_max                                           640
zfs_zil_clean_taskq_maxalloc                             1048576
zfs_zil_clean_taskq_minalloc                                1024
zfs_zil_clean_taskq_nthr_pct                                 100
zil_nocacheflush                                               0
zil_replay_disable                                             0
zil_slog_bulk                                             786432
zio_deadman_log_all                                            0
zio_dva_throttle_enabled                                       1
zio_requeue_io_start_cut_in_line                               1
zio_slow_io_ms                                             30000
zio_taskq_batch_pct                                           75
zvol_inhibit_dev                                               0
zvol_major                                                   230
zvol_max_discard_blocks                                    16384
zvol_prefetch_bytes                                       131072
zvol_request_sync                                              0
zvol_threads                                                  32
zvol_volmode                                                   1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         1.1G
Commit requests:                                          110.1M
Flushes to stable storage:                                110.1M
Transactions to SLOG storage pool:            0 Bytes          0
Transactions to non-SLOG storage pool:        6.8 TiB     155.8M
 
Hi Sorry for stating up this old thread but i just wanted to ask did you ever get this figured out? i have a very similar issue for the last 6 months or so on my standalone system. i tried a bunch of stuff including zfs performance tweaks, new raid card (running it mode) , btrfs file system does the same for me. when its ext4 it has no issues. its every 30 minutes or so the i/o goes up and up then down. great to hear if you got your issue fiixed.
 
Are you using consumer SSD?
If yes then they maube can't keep up with the IO. SSD writes to DRAM-/SLC-cache so usable performance and no IO delay. Then the cache gets full, performance drop as you now got the real horrible TLC or even worse QLC performance and IO delay goes up. As the IO delay goes up the host will send less data to the SSD. The SSD priorizes the cached data over the new data and first moves the old data from DRAM-cache to SLC-cache and then from SLC-cache to TLC/QLC NAND. Then the cache got free space again, performance will go up and IO delay will drop. This then repeats over and over.

Another explanation could be your workload. ZFS and btrfs are copy-on-write filesystems with way more overhead than ext4. And they also use more sync writes which are slower than async writes and terrible slow (down to HDD performance) when not using enterprise SSD with built-in power-loss protection. Maybe some of your VMs/LXCs are doing some heavy IO every 30 minutes and this gets write amplified by ZFS/btrfs to a level that your SSDs can't handle.
 
  • Like
Reactions: neiion
Thanks for the detailed reply, yes its running consumer ssd drives. i will need to upgrade ssds i think. thanks again for the detailed explanation on the issue.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back