how do I improve my zfs random 4k performance?

proxmoxrks

Member
Apr 14, 2020
32
1
8
49
Hi,
I installed proxmox using ZFS RAID0 on my nvme.


Code:
root@pve:~# zpool get all
NAME   PROPERTY                       VALUE                          SOURCE
rpool  size                           464G                           -
rpool  capacity                       0%                             -
rpool  altroot                        -                              default
rpool  health                         ONLINE                         -
rpool  guid                           12978240975727830286           -
rpool  version                        -                              default
rpool  bootfs                         rpool/ROOT/pve-1               local
rpool  delegation                     on                             default
rpool  autoreplace                    off                            default
rpool  cachefile                      -                              default
rpool  failmode                       wait                           default
rpool  listsnapshots                  off                            default
rpool  autoexpand                     off                            default
rpool  dedupditto                     0                              default
rpool  dedupratio                     1.00x                          -
rpool  free                           463G                           -
rpool  allocated                      945M                           -
rpool  readonly                       off                            -
rpool  ashift                         12                             local
rpool  comment                        -                              default
rpool  expandsize                     -                              -
rpool  freeing                        0                              -
rpool  fragmentation                  0%                             -
rpool  leaked                         0                              -
rpool  multihost                      off                            default
rpool  checkpoint                     -                              -
rpool  load_guid                      17530097993645509932           -
rpool  autotrim                       off                            default
rpool  feature@async_destroy          enabled                        local
rpool  feature@empty_bpobj            active                         local
rpool  feature@lz4_compress           active                         local
rpool  feature@multi_vdev_crash_dump  enabled                        local
rpool  feature@spacemap_histogram     active                         local
rpool  feature@enabled_txg            active                         local
rpool  feature@hole_birth             active                         local
rpool  feature@extensible_dataset     active                         local
rpool  feature@embedded_data          active                         local
rpool  feature@bookmarks              enabled                        local
rpool  feature@filesystem_limits      enabled                        local
rpool  feature@large_blocks           enabled                        local
rpool  feature@large_dnode            enabled                        local
rpool  feature@sha512                 enabled                        local
rpool  feature@skein                  enabled                        local
rpool  feature@edonr                  enabled                        local
rpool  feature@userobj_accounting     active                         local
rpool  feature@encryption             enabled                        local
rpool  feature@project_quota          active                         local
rpool  feature@device_removal         enabled                        local
rpool  feature@obsolete_counts        enabled                        local
rpool  feature@zpool_checkpoint       enabled                        local
rpool  feature@spacemap_v2            active                         local
rpool  feature@allocation_classes     enabled                        local
rpool  feature@resilver_defer         enabled                        local
rpool  feature@bookmark_v2            enabled                        local

Code:
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
#              Yet-Another-Bench-Script              #
#                     v2020-09-21                    #
# https://github.com/masonr/yet-another-bench-script #
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #

Sun 18 Oct 2020 06:55:48 AM EDT


fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 59.53 MB/s   (14.8k) | 731.85 MB/s  (11.4k)
Write      | 59.65 MB/s   (14.9k) | 735.70 MB/s  (11.4k)
Total      | 119.18 MB/s  (29.7k) | 1.46 GB/s    (22.9k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 1.16 GB/s     (2.2k) | 1.21 GB/s     (1.1k)
Write      | 1.22 GB/s     (2.3k) | 1.30 GB/s     (1.2k)
Total      | 2.39 GB/s     (4.6k) | 2.52 GB/s     (2.4k)

When I am benchmarking with EXT4, I could get 1.41 GB/s (354.9k). When using ZFS, it dropped to 119.18 MB/s (29.7k). How do I tune it to improve it? Thanks.

Code:
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 709.00 MB/s (177.2k) | 913.78 MB/s  (14.2k)
Write      | 710.87 MB/s (177.7k) | 918.59 MB/s  (14.3k)
Total      | 1.41 GB/s   (354.9k) | 1.83 GB/s    (28.6k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 1.01 GB/s     (1.9k) | 1.14 GB/s     (1.1k)
Write      | 1.06 GB/s     (2.0k) | 1.21 GB/s     (1.1k)
Total      | 2.07 GB/s     (4.0k) | 2.35 GB/s     (2.3k)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!