VZDump "kills" ARC cache...?

Pavel Hruška

Member
May 1, 2018
75
8
8
45
Hello, I'm wondering why when vzdump starts (once overy week), that it suddenly empties ZFS ARC cache. If you take a look at the graphs you see increase in CPU utilization (vzdump starts using pigz) and on the same time there is steady memory usage drop, which corresponds to lowering ARC memory usage...

upload_2018-11-3_8-59-39.png

Yesterday ARC cache memory usage was at 10GB, now it is on low limit.

Anyone can help me explaining this?

Thank you!

arc_summary dump:

Code:
ZFS Subsystem Report                            Sat Nov 03 09:02:03 2018
ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                1.32G
        Mutex Misses:                           252.99k
        Evict Skips:                            445.25M

ARC Size:                               20.02%  2.00    GiB
        Target Size: (Adaptive)         20.39%  2.04    GiB
        Min Size (Hard Limit):          20.00%  2.00    GiB
        Max Size (High Water):          5:1     10.00   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       7.81%   118.28  MiB
        Frequently Used Cache Size:     92.19%  1.36    GiB

ARC Hash Breakdown:
        Elements Max:                           10.23M
        Elements Current:               49.28%  5.04M
        Collisions:                             1.13G
        Chain Max:                              11
        Chains:                                 1.03M

ARC Total accesses:                                     5.02G
        Cache Hit Ratio:                69.15%  3.47G
        Cache Miss Ratio:               30.85%  1.55G
        Actual Hit Ratio:               68.05%  3.42G

        Data Demand Efficiency:         46.68%  1.57G
        Data Prefetch Efficiency:       7.11%   762.18M

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             1.36%   47.26M
          Most Recently Used:           31.41%  1.09G
          Most Frequently Used:         67.01%  2.33G
          Most Recently Used Ghost:     0.14%   4.96M
          Most Frequently Used Ghost:   0.08%   2.74M

        CACHE HITS BY DATA TYPE:
          Demand Data:                  21.07%  731.20M
          Prefetch Data:                1.56%   54.22M
          Demand Metadata:              77.21%  2.68G
          Prefetch Metadata:            0.16%   5.56M

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  53.95%  835.36M
          Prefetch Data:                45.72%  707.96M
          Demand Metadata:              0.24%   3.67M
          Prefetch Metadata:            0.10%   1.55M

L2 ARC Summary: (HEALTHY)
        Low Memory Aborts:                      177
        Free on Write:                          7.09M
        R/W Clashes:                            184
        Bad Checksums:                          0
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         39.60   GiB
        Compressed:                     63.36%  25.09   GiB
        Header Size:                    1.08%   437.96  MiB

L2 ARC Evicts:
        Lock Retries:                           5.54k
        Upon Reading:                           0

L2 ARC Breakdown:                               1.55G
        Hit Ratio:                      0.40%   6.26M
        Miss Ratio:                     99.60%  1.54G
        Feeds:                                  941.12k

L2 ARC Writes:
        Writes Sent:                    100.00% 709.02k

DMU Prefetch Efficiency:                                        221.34M
        Hit Ratio:                      44.23%  97.90M
        Miss Ratio:                     55.77%  123.45M



ZFS Tunables:
        dbuf_cache_hiwater_pct                            10
        dbuf_cache_lowater_pct                            10
        dbuf_cache_max_bytes                              104857600
        dbuf_cache_max_shift                              5
        dmu_object_alloc_chunk_shift                      7
        ignore_hole_birth                                 1
        l2arc_feed_again                                  1
        l2arc_feed_min_ms                                 200
        l2arc_feed_secs                                   1
        l2arc_headroom                                    2
        l2arc_headroom_boost                              200
        l2arc_noprefetch                                  0
        l2arc_norw                                        0
        l2arc_write_boost                                 33554432
        l2arc_write_max                                   16777216
        metaslab_aliquot                                  524288
        metaslab_bias_enabled                             1
        metaslab_debug_load                               0
        metaslab_debug_unload                             0
        metaslab_fragmentation_factor_enabled             1
        metaslab_lba_weighting_enabled                    1
        metaslab_preload_enabled                          1
        metaslabs_per_vdev                                200
        send_holes_without_birth_time                     1
        spa_asize_inflation                               24
        spa_config_path                                   /etc/zfs/zpool.cache
        spa_load_verify_data                              1
        spa_load_verify_maxinflight                       10000
        spa_load_verify_metadata                          1
        spa_slop_shift                                    5
        zfetch_array_rd_sz                                1048576
        zfetch_max_distance                               8388608
        zfetch_max_streams                                8
        zfetch_min_sec_reap                               2
        zfs_abd_scatter_enabled                           1
        zfs_abd_scatter_max_order                         10
        zfs_admin_snapshot                                1
        zfs_arc_average_blocksize                         8192
        zfs_arc_dnode_limit                               0
        zfs_arc_dnode_limit_percent                       10
        zfs_arc_dnode_reduce_percent                      10
        zfs_arc_grow_retry                                0
        zfs_arc_lotsfree_percent                          10
        zfs_arc_max                                       10737418240
        zfs_arc_meta_adjust_restarts                      4096
        zfs_arc_meta_limit                                0
        zfs_arc_meta_limit_percent                        75
        zfs_arc_meta_min                                  0
        zfs_arc_meta_prune                                10000
        zfs_arc_meta_strategy                             1
        zfs_arc_min                                       2147483648
        zfs_arc_min_prefetch_lifespan                     0
        zfs_arc_p_dampener_disable                        1
        zfs_arc_p_min_shift                               0
        zfs_arc_pc_percent                                0
        zfs_arc_shrink_shift                              0
        zfs_arc_sys_free                                  0
        zfs_autoimport_disable                            1
        zfs_checksums_per_second                          20
        zfs_compressed_arc_enabled                        1
        zfs_dbgmsg_enable                                 0
        zfs_dbgmsg_maxsize                                4194304
        zfs_dbuf_state_index                              0
        zfs_deadman_checktime_ms                          5000
        zfs_deadman_enabled                               1
        zfs_deadman_synctime_ms                           1000000
        zfs_dedup_prefetch                                0
        zfs_delay_min_dirty_percent                       60
        zfs_delay_scale                                   500000
        zfs_delays_per_second                             20
        zfs_delete_blocks                                 20480
        zfs_dirty_data_max                                4294967296
        zfs_dirty_data_max_max                            4294967296
        zfs_dirty_data_max_max_percent                    25
        zfs_dirty_data_max_percent                        10
        zfs_dirty_data_sync                               67108864
        zfs_dmu_offset_next_sync                          0
        zfs_expire_snapshot                               300
        zfs_flags                                         0
        zfs_free_bpobj_enabled                            1
        zfs_free_leak_on_eio                              0
        zfs_free_max_blocks                               100000
        zfs_free_min_time_ms                              1000
        zfs_immediate_write_sz                            32768
        zfs_max_recordsize                                1048576
        zfs_mdcomp_disable                                0
        zfs_metaslab_fragmentation_threshold              70
        zfs_metaslab_segment_weight_enabled               1
        zfs_metaslab_switch_threshold                     2
        zfs_mg_fragmentation_threshold                    85
        zfs_mg_noalloc_threshold                          0
        zfs_multihost_fail_intervals                      5
        zfs_multihost_history                             0
        zfs_multihost_import_intervals                    10
        zfs_multihost_interval                            1000
        zfs_multilist_num_sublists                        0
        zfs_no_scrub_io                                   0
        zfs_no_scrub_prefetch                             0
        zfs_nocacheflush                                  0
        zfs_nopwrite_enabled                              1
        zfs_object_mutex_size                             64
        zfs_pd_bytes_max                                  52428800
        zfs_per_txg_dirty_frees_percent                   30
        zfs_prefetch_disable                              0
        zfs_read_chunk_size                               1048576
        zfs_read_history                                  0
        zfs_read_history_hits                             0
        zfs_recover                                       0
        zfs_recv_queue_length                             16777216
        zfs_resilver_delay                                2
        zfs_resilver_min_time_ms                          3000
        zfs_scan_idle                                     50
        zfs_scan_ignore_errors                            0
        zfs_scan_min_time_ms                              1000
        zfs_scrub_delay                                   4
        zfs_send_corrupt_data                             0
        zfs_send_queue_length                             16777216
        zfs_sync_pass_deferred_free                       2
        zfs_sync_pass_dont_compress                       5
        zfs_sync_pass_rewrite                             2
        zfs_sync_taskq_batch_pct                          75
        zfs_top_maxinflight                               32
        zfs_txg_history                                   0
        zfs_txg_timeout                                   5
        zfs_vdev_aggregation_limit                        131072
        zfs_vdev_async_read_max_active                    3
        zfs_vdev_async_read_min_active                    1
        zfs_vdev_async_write_active_max_dirty_percent     60
        zfs_vdev_async_write_active_min_dirty_percent     30
        zfs_vdev_async_write_max_active                   10
        zfs_vdev_async_write_min_active                   2
        zfs_vdev_cache_bshift                             16
        zfs_vdev_cache_max                                16384
        zfs_vdev_cache_size                               0
        zfs_vdev_max_active                               1000
        zfs_vdev_mirror_non_rotating_inc                  0
        zfs_vdev_mirror_non_rotating_seek_inc             1
        zfs_vdev_mirror_rotating_inc                      0
        zfs_vdev_mirror_rotating_seek_inc                 5
        zfs_vdev_mirror_rotating_seek_offset              1048576
        zfs_vdev_queue_depth_pct                          1000
        zfs_vdev_raidz_impl                               [fastest] original scalar sse2 ssse3 avx2
        zfs_vdev_read_gap_limit                           32768
        zfs_vdev_scheduler                                noop
        zfs_vdev_scrub_max_active                         2
        zfs_vdev_scrub_min_active                         1
        zfs_vdev_sync_read_max_active                     10
        zfs_vdev_sync_read_min_active                     10
        zfs_vdev_sync_write_max_active                    10
        zfs_vdev_sync_write_min_active                    10
        zfs_vdev_write_gap_limit                          4096
        zfs_zevent_cols                                   80
        zfs_zevent_console                                0
        zfs_zevent_len_max                                128
        zil_replay_disable                                0
        zil_slog_bulk                                     786432
        zio_delay_max                                     30000
        zio_dva_throttle_enabled                          1
        zio_requeue_io_start_cut_in_line                  1
        zio_taskq_batch_pct                               75
        zvol_inhibit_dev                                  0
        zvol_major                                        230
        zvol_max_discard_blocks                           16384
        zvol_prefetch_bytes                               131072
        zvol_request_sync                                 0
        zvol_threads                                      32
        zvol_volmode                                      1
 
Last edited:
I see, but memory usage dropped (see the graphs) when backup started... Well I am assuming that those graphs show host's overall memory usage. As the ARC cache warmed up next day the memory usage returned back to the previous numbers...
 
When you use SMB, NFS, or any other network/protocol, your cache memory free -hw starts to grow up, at some point is going to compete with ARC cache, which will eventually cause your a drop. You can fix this with ...

Code:
# Insert this line in /etc/modprobe.d/zfs.conf
options zfs zfs_arc_pc_percent=100

Check this link, to know more about this parameter
 
When you use SMB, NFS, or any other network/protocol, your cache memory free -hw starts to grow up, at some point is going to compete with ARC cache, which will eventually cause your a drop. You can fix this with ...

Code:
# Insert this line in /etc/modprobe.d/zfs.conf
options zfs zfs_arc_pc_percent=100

Check this link, to know more about this parameter
Good to know, I will give it a try, thank you for your feedback!
 
It's always good to monitor your ARC over time too. Every modern metric server should be able to collect those numbers.
You than can compare the numbers from "outside" to ZFS.

Here is an example:

1569497462572.png
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!