No ARC usage

C

carles89

Renowned Member
Proxmox Subscriber
May 27, 2015
105
18
83
Hi,

I've a server with these specifications:
Intel(R) Xeon(R) CPU E31220
16GB ECC Ram
2 x 1TB SATA drives (WD10EFRX-68FYTN0) with ZFS Mirror.

It runs incredibly slow. I'm aware SATA disks are slow and we should install a SSD cache device, but I noticed arcsize is small. It seems ZFS it's not using it:

Code: 
root@srv2:~# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
14:04:10     0     0      0     0    0     0    0     0    0   500M  526M

The server has two Windows 2016 VMs (AD and FS), one with 2GB RAM and the other with 3GB, so we're using only 5GB from the total available of 16GB:

upload_2018-6-14_14-11-25.png

Swapiness was already set to 10.

Any thougts?

Thank you all!
 

Attachments

  • upload_2018-6-14_14-11-57.png
    upload_2018-6-14_14-11-57.png
    25.4 KB · Views: 14
  • Like
Reactions: carles89
ARC is a dynamic high complex algorithm.
For systems with low memory, it is known that the calculated value (target arc size) is often assumed to be too small.
ZFS is an enterprise-filesystem what is made for huge machines with lots of memory, so low memory systems are more or less edge case for ZFS.
16 GB is not low memory but you have to substrate the used memory and also calculate the memory pressure what can come from QEMU.
 
  • Like
Reactions: carles89
Yours hdd have only 5400 rpm, so it is normal that your server to be slow. Also your arc usege is very strange. I never see only 0(even when my zfs arc was at a maximum of 0.5 GB) . Maybe you have run this after few moments after server boot. Try to run the arc stat after few hours after your server was started.

What show your zpool status -v and zpool list?
 
Hi guletz,

I didn't noticed disks were 5400rpm... It explains a lot.

System uptime is 163 days now.

Here are zpool status -v and zpool list:

Code: 
root@srv2:~# zpool status -v
  pool: rpool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 27h8m with 0 errors on Mon Jun 11 03:32:33 2018
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0

errors: No known data errors

Code: 
root@srv2:~# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool   928G   502G   426G         -    45%    54%  1.00x  ONLINE  -

Thank you!
 
Hi,

Here it is. Thank you!

Code: 
root@srv2:~# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Jun 18 11:38:40 2018
ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                2.43b
        Mutex Misses:                           1.10m
        Evict Skips:                            1.10m

ARC Size:                               8.93%   714.90  MiB
        Target Size: (Adaptive)         9.36%   749.17  MiB
        Min Size (Hard Limit):          6.25%   500.37  MiB
        Max Size (High Water):          16:1    7.82    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       55.54%  416.06  MiB
        Frequently Used Cache Size:     44.46%  333.11  MiB

ARC Hash Breakdown:
        Elements Max:                           2.05m
        Elements Current:               5.86%   120.17k
        Collisions:                             222.57m
        Chain Max:                              10
        Chains:                                 2.86k

ARC Total accesses:                                     4.70b
        Cache Hit Ratio:                42.37%  1.99b
        Cache Miss Ratio:               57.63%  2.71b
        Actual Hit Ratio:               38.21%  1.80b

        Data Demand Efficiency:         81.28%  2.11b
        Data Prefetch Efficiency:       6.44%   2.37b

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           86.57%  1.72b
          Most Frequently Used:         3.61%   71.95m
          Most Recently Used Ghost:     11.88%  236.63m
          Most Frequently Used Ghost:   0.27%   5.32m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  86.14%  1.72b
          Prefetch Data:                7.65%   152.32m
          Demand Metadata:              2.95%   58.82m
          Prefetch Metadata:            3.26%   64.89m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  14.59%  395.12m
          Prefetch Data:                81.76%  2.21b
          Demand Metadata:              1.75%   47.51m
          Prefetch Metadata:            1.90%   51.35m


DMU Prefetch Efficiency:                                        1.62b
        Hit Ratio:                      27.93%  451.51m
        Miss Ratio:                     72.07%  1.17b



ZFS Tunable:
        zvol_volmode                                      1
        l2arc_headroom                                    2
        dbuf_cache_max_shift                              5
        zfs_free_leak_on_eio                              0
        zfs_free_max_blocks                               100000
        zfs_read_chunk_size                               1048576
        metaslab_preload_enabled                          1
        zfs_dedup_prefetch                                0
        zfs_txg_history                                   0
        zfs_scrub_delay                                   4
        zfs_vdev_async_read_max_active                    3
        zfs_read_history                                  0
        zfs_arc_sys_free                                  0
        l2arc_write_max                                   8388608
        zil_slog_bulk                                     786432
        zfs_dbuf_state_index                              0
        zfs_sync_taskq_batch_pct                          75
        metaslab_debug_unload                             0
        zvol_inhibit_dev                                  0
        zfs_abd_scatter_enabled                           1
        zfs_arc_pc_percent                                0
        zfetch_max_streams                                8
        zfs_recover                                       0
        metaslab_fragmentation_factor_enabled             1
        zfs_deadman_checktime_ms                          5000
        zfs_sync_pass_rewrite                             2
        zfs_object_mutex_size                             64
        zfs_arc_min_prefetch_lifespan                     0
        zfs_arc_meta_prune                                10000
        zfs_read_history_hits                             0
        zfetch_max_distance                               8388608
        l2arc_norw                                        0
        zfs_dirty_data_max_percent                        10
        zfs_per_txg_dirty_frees_percent                   30
        zfs_arc_meta_min                                  0
        metaslabs_per_vdev                                200
        zfs_arc_meta_adjust_restarts                      4096
        spa_load_verify_maxinflight                       10000
        spa_load_verify_metadata                          1
        zfs_multihost_history                             0
        zfs_send_corrupt_data                             0
        zfs_delay_min_dirty_percent                       60
        zfs_vdev_sync_read_max_active                     10
        zfs_dbgmsg_enable                                 0
        zfs_metaslab_segment_weight_enabled               1
        zio_requeue_io_start_cut_in_line                  1
        l2arc_headroom_boost                              200
        zfs_zevent_cols                                   80
        zfs_dmu_offset_next_sync                          0
        spa_config_path                                   /etc/zfs/zpool.cache
        zfs_vdev_cache_size                               0
        dbuf_cache_hiwater_pct                            10
        zfs_multihost_interval                            1000
        zfs_multihost_fail_intervals                      5
        zio_dva_throttle_enabled                          1
        zfs_vdev_sync_write_min_active                    10
        zfs_vdev_scrub_max_active                         2
        ignore_hole_birth                                 1
        zvol_major                                        230
        zil_replay_disable                                0
        zfs_dirty_data_max_max_percent                    25
        zfs_expire_snapshot                               300
        zfs_sync_pass_deferred_free                       2
        spa_asize_inflation                               24
        dmu_object_alloc_chunk_shift                      7
        zfs_vdev_mirror_rotating_seek_offset              1048576
        l2arc_feed_secs                                   1
        zfs_autoimport_disable                            1
        zfs_arc_p_aggressive_disable                      1
        zfs_zevent_len_max                                64
        zfs_arc_meta_limit_percent                        75
        l2arc_noprefetch                                  1
        zfs_vdev_raidz_impl                               [fastest] original scalar sse2 ssse3
        zfs_arc_meta_limit                                0
        zfs_flags                                         0
        zfs_dirty_data_max_max                            4197377024
        zfs_arc_average_blocksize                         8192
        zfs_vdev_cache_bshift                             16
        zfs_vdev_async_read_min_active                    1
        zfs_arc_dnode_reduce_percent                      10
        zfs_free_bpobj_enabled                            1
        zfs_arc_grow_retry                                0
        zfs_vdev_mirror_rotating_inc                      0
        l2arc_feed_again                                  1
        zfs_vdev_mirror_non_rotating_inc                  0
        zfs_arc_lotsfree_percent                          10
        zfs_zevent_console                                0
        zvol_prefetch_bytes                               131072
        zfs_free_min_time_ms                              1000
        zfs_arc_dnode_limit_percent                       10
        zio_taskq_batch_pct                               75
        dbuf_cache_max_bytes                              104857600
        spa_load_verify_data                              1
        zfs_delete_blocks                                 20480
        zfs_vdev_mirror_non_rotating_seek_inc             1
        zfs_multihost_import_intervals                    10
        zfs_dirty_data_max                                1678950809
        zfs_vdev_async_write_max_active                   10
        zfs_dbgmsg_maxsize                                4194304
        zfs_nocacheflush                                  0
        zfetch_array_rd_sz                                1048576
        zfs_arc_meta_strategy                             1
        zfs_dirty_data_sync                               67108864
        zvol_max_discard_blocks                           16384
        zvol_threads                                      32
        zfs_vdev_async_write_active_max_dirty_percent     60
        zfs_arc_p_dampener_disable                        1
        zfs_txg_timeout                                   5
        metaslab_aliquot                                  524288
        zfs_mdcomp_disable                                0
        zfs_vdev_sync_read_min_active                     10
        zfs_arc_dnode_limit                               0
        dbuf_cache_lowater_pct                            10
        zfs_abd_scatter_max_order                         10
        metaslab_debug_load                               0
        zfs_vdev_aggregation_limit                        131072
        metaslab_lba_weighting_enabled                    1
        zfs_vdev_scheduler                                noop
        zfs_vdev_scrub_min_active                         1
        zfs_no_scrub_io                                   0
        zfs_vdev_cache_max                                16384
        zfs_scan_idle                                     50
        zfs_arc_shrink_shift                              0
        spa_slop_shift                                    5
        zfs_vdev_mirror_rotating_seek_inc                 5
        zfs_deadman_synctime_ms                           1000000
        send_holes_without_birth_time                     1
        metaslab_bias_enabled                             1
        zvol_request_sync                                 0
        zfs_admin_snapshot                                1
        zfs_no_scrub_prefetch                             0
        zfs_metaslab_fragmentation_threshold              70
        zfs_max_recordsize                                1048576
        zfs_arc_min                                       0
        zfs_nopwrite_enabled                              1
        zfs_arc_p_min_shift                               0
        zfs_multilist_num_sublists                        0
        zfs_vdev_queue_depth_pct                          1000
        zfs_mg_fragmentation_threshold                    85
        l2arc_write_boost                                 8388608
        zfs_prefetch_disable                              0
        l2arc_feed_min_ms                                 200
        zio_delay_max                                     30000
        zfs_vdev_write_gap_limit                          4096
        zfs_pd_bytes_max                                  52428800
        zfs_scan_min_time_ms                              1000
        zfs_resilver_min_time_ms                          3000
        zfs_delay_scale                                   500000
        zfs_vdev_async_write_active_min_dirty_percent     30
        zfs_vdev_sync_write_max_active                    10
        zfs_mg_noalloc_threshold                          0
        zfs_deadman_enabled                               1
        zfs_resilver_delay                                2
        zfs_metaslab_switch_threshold                     2
        zfs_arc_max                                       0
        zfs_top_maxinflight                               32
        zfetch_min_sec_reap                               2
        zfs_immediate_write_sz                            32768
        zfs_vdev_async_write_min_active                   2
        zfs_sync_pass_dont_compress                       5
        zfs_vdev_read_gap_limit                           32768
        zfs_compressed_arc_enabled                        1
        zfs_vdev_max_active
 
Your ARC usage show very strange low statistics! Try to backup all you have, and after that try to upgrade your pool(with all your VM stoped):

zpool upgrade -a
 
guletz said:
Your ARC usage show very strange low statistics! Try to backup all you have, and after that try to upgrade your pool(with all your VM stoped):

zpool upgrade -a
Click to expand...

I'll try that on next maintenance window. Thanks!

wolfgang said:
Hi,

on Systems with small memory, you should set the arc size to a fixed value like 6GB.

I mean set options zfs zfs_arc_min, options zfs zfs_arc_max to the same size.
see
https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Install_on_a_high_performance_system
Click to expand...

After doing that, it seems to increase arc usage but it finally drops to low values:

Code: 
root@srv2-global:~# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
16:14:32     0     0      0     0    0     0    0     0    0   647M  710M

If I ran update-initramfs -u, should I reboot the host?

Thank you all
 
Hi all,

After rebooting the server (with zfs_arc_min and zfs_arc_max updated) values are much better for these slow disks.

Arc size is growing just now to fit min size.

Thanks for your help!

Code: 
arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Jun 25 17:01:59 2018
ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                19
        Mutex Misses:                           0
        Evict Skips:                            1

ARC Size:                               19.57%  1.57    GiB
        Target Size: (Adaptive)         100.00% 8.00    GiB
        Min Size (Hard Limit):          100.00% 8.00    GiB
        Max Size (High Water):          1:1     8.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       84.23%  1.25    GiB
        Frequently Used Cache Size:     15.77%  239.49  MiB

ARC Hash Breakdown:
        Elements Max:                           226.10k
        Elements Current:               100.00% 226.10k
        Collisions:                             14.24k
        Chain Max:                              3
        Chains:                                 11.15k

ARC Total accesses:                                     750.59k
        Cache Hit Ratio:                71.86%  539.39k
        Cache Miss Ratio:               28.14%  211.20k
        Actual Hit Ratio:               70.84%  531.75k

        Data Demand Efficiency:         57.08%  245.62k
        Data Prefetch Efficiency:       6.30%   92.01k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             1.42%   7.64k
          Most Recently Used:           31.68%  170.90k
          Most Frequently Used:         66.90%  360.85k
          Most Recently Used Ghost:     0.00%   0
          Most Frequently Used Ghost:   0.00%   0

        CACHE HITS BY DATA TYPE:
          Demand Data:                  25.99%  140.20k
          Prefetch Data:                1.07%   5.80k
          Demand Metadata:              72.52%  391.17k
          Prefetch Metadata:            0.41%   2.22k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  49.91%  105.42k
          Prefetch Data:                40.82%  86.21k
          Demand Metadata:              7.20%   15.20k
          Prefetch Metadata:            2.07%   4.37k


DMU Prefetch Efficiency:                                        226.62k
        Hit Ratio:                      8.49%   19.25k
        Miss Ratio:                     91.51%  207.37k



ZFS Tunables:
        dbuf_cache_hiwater_pct                            10
        dbuf_cache_lowater_pct                            10
        dbuf_cache_max_bytes                              104857600
        dbuf_cache_max_shift                              5
        dmu_object_alloc_chunk_shift                      7
        ignore_hole_birth                                 1
        l2arc_feed_again                                  1
        l2arc_feed_min_ms                                 200
        l2arc_feed_secs                                   1
        l2arc_headroom                                    2
        l2arc_headroom_boost                              200
        l2arc_noprefetch                                  1
        l2arc_norw                                        0
        l2arc_write_boost                                 8388608
        l2arc_write_max                                   8388608
        metaslab_aliquot                                  524288
        metaslab_bias_enabled                             1
        metaslab_debug_load                               0
        metaslab_debug_unload                             0
        metaslab_fragmentation_factor_enabled             1
        metaslab_lba_weighting_enabled                    1
        metaslab_preload_enabled                          1
        metaslabs_per_vdev                                200
        send_holes_without_birth_time                     1
        spa_asize_inflation                               24
        spa_config_path                                   /etc/zfs/zpool.cache
        spa_load_verify_data                              1
        spa_load_verify_maxinflight                       10000
        spa_load_verify_metadata                          1
        spa_slop_shift                                    5
        zfetch_array_rd_sz                                1048576
        zfetch_max_distance                               8388608
        zfetch_max_streams                                8
        zfetch_min_sec_reap                               2
        zfs_abd_scatter_enabled                           1
        zfs_abd_scatter_max_order                         10
        zfs_admin_snapshot                                1
        zfs_arc_average_blocksize                         8192
        zfs_arc_dnode_limit                               0
        zfs_arc_dnode_limit_percent                       10
        zfs_arc_dnode_reduce_percent                      10
        zfs_arc_grow_retry                                0
        zfs_arc_lotsfree_percent                          10
        zfs_arc_max                                       8589934592
        zfs_arc_meta_adjust_restarts                      4096
        zfs_arc_meta_limit                                0
        zfs_arc_meta_limit_percent                        75
        zfs_arc_meta_min                                  0
        zfs_arc_meta_prune                                10000
        zfs_arc_meta_strategy                             1
        zfs_arc_min                                       8589934592
        zfs_arc_min_prefetch_lifespan                     0
        zfs_arc_p_dampener_disable                        1
        zfs_arc_p_min_shift                               0
        zfs_arc_pc_percent                                0
        zfs_arc_shrink_shift                              0
        zfs_arc_sys_free                                  0
        zfs_autoimport_disable                            1
        zfs_checksums_per_second                          20
        zfs_compressed_arc_enabled                        1
        zfs_dbgmsg_enable                                 0
        zfs_dbgmsg_maxsize                                4194304
        zfs_dbuf_state_index                              0
        zfs_deadman_checktime_ms                          5000
        zfs_deadman_enabled                               1
        zfs_deadman_synctime_ms                           1000000
        zfs_dedup_prefetch                                0
        zfs_delay_min_dirty_percent                       60
        zfs_delay_scale                                   500000
        zfs_delays_per_second                             20
        zfs_delete_blocks                                 20480
        zfs_dirty_data_max                                1678617804
        zfs_dirty_data_max_max                            4196544512
        zfs_dirty_data_max_max_percent                    25
        zfs_dirty_data_max_percent                        10
        zfs_dirty_data_sync                               67108864
        zfs_dmu_offset_next_sync                          0
        zfs_expire_snapshot                               300
        zfs_flags                                         0
        zfs_free_bpobj_enabled                            1
        zfs_free_leak_on_eio                              0
        zfs_free_max_blocks                               100000
        zfs_free_min_time_ms                              1000
        zfs_immediate_write_sz                            32768
        zfs_max_recordsize                                1048576
        zfs_mdcomp_disable                                0
        zfs_metaslab_fragmentation_threshold              70
        zfs_metaslab_segment_weight_enabled               1
        zfs_metaslab_switch_threshold                     2
        zfs_mg_fragmentation_threshold                    85
        zfs_mg_noalloc_threshold                          0
        zfs_multihost_fail_intervals                      5
        zfs_multihost_history                             0
        zfs_multihost_import_intervals                    10
        zfs_multihost_interval                            1000
        zfs_multilist_num_sublists                        0
        zfs_no_scrub_io                                   0
        zfs_no_scrub_prefetch                             0
        zfs_nocacheflush                                  0
        zfs_nopwrite_enabled                              1
        zfs_object_mutex_size                             64
        zfs_pd_bytes_max                                  52428800
        zfs_per_txg_dirty_frees_percent                   30
        zfs_prefetch_disable                              0
        zfs_read_chunk_size                               1048576
        zfs_read_history                                  0
        zfs_read_history_hits                             0
        zfs_recover                                       0
        zfs_recv_queue_length                             16777216
        zfs_resilver_delay                                2
        zfs_resilver_min_time_ms                          3000
        zfs_scan_idle                                     50
        zfs_scan_ignore_errors                            0
        zfs_scan_min_time_ms                              1000
        zfs_scrub_delay                                   4
        zfs_send_corrupt_data                             0
        zfs_send_queue_length                             16777216
        zfs_sync_pass_deferred_free                       2
        zfs_sync_pass_dont_compress                       5
        zfs_sync_pass_rewrite                             2
        zfs_sync_taskq_batch_pct                          75
        zfs_top_maxinflight                               32
        zfs_txg_history                                   0
        zfs_txg_timeout                                   5
        zfs_vdev_aggregation_limit                        131072
        zfs_vdev_async_read_max_active                    3
        zfs_vdev_async_read_min_active                    1
        zfs_vdev_async_write_active_max_dirty_percent     60
        zfs_vdev_async_write_active_min_dirty_percent     30
        zfs_vdev_async_write_max_active                   10
        zfs_vdev_async_write_min_active                   2
        zfs_vdev_cache_bshift                             16
        zfs_vdev_cache_max                                16384
        zfs_vdev_cache_size                               0
        zfs_vdev_max_active                               1000
        zfs_vdev_mirror_non_rotating_inc                  0
        zfs_vdev_mirror_non_rotating_seek_inc             1
        zfs_vdev_mirror_rotating_inc                      0
        zfs_vdev_mirror_rotating_seek_inc                 5
        zfs_vdev_mirror_rotating_seek_offset              1048576
        zfs_vdev_queue_depth_pct                          1000
        zfs_vdev_raidz_impl                               [fastest] original scalar sse2 ssse3
        zfs_vdev_read_gap_limit                           32768
        zfs_vdev_scheduler                                noop
        zfs_vdev_scrub_max_active                         2
        zfs_vdev_scrub_min_active                         1
        zfs_vdev_sync_read_max_active                     10
        zfs_vdev_sync_read_min_active                     10
        zfs_vdev_sync_write_max_active                    10
        zfs_vdev_sync_write_min_active                    10
        zfs_vdev_write_gap_limit                          4096
        zfs_zevent_cols                                   80
        zfs_zevent_console                                0
        zfs_zevent_len_max                                64
        zil_replay_disable                                0
        zil_slog_bulk                                     786432
        zio_delay_max                                     30000
        zio_dva_throttle_enabled                          1
        zio_requeue_io_start_cut_in_line                  1
        zio_taskq_batch_pct                               75
        zvol_inhibit_dev                                  0
        zvol_major                                        230
        zvol_max_discard_blocks                           16384
        zvol_prefetch_bytes                               131072
        zvol_request_sync                                 0
        zvol_threads                                      32
        zvol_volmode                                      1
 
You must log in or register to reply here.
Top Bottom