performance difference zfs

alexandere

Member
Jul 20, 2021
7
0
6
41
I'm trying to understand a performance difference between between 2 zfs filesystems on the same disk. One is filesystem made by creating a container through proxmox. e.a. /rpool/data/subvol-***-disk-* and the other is the local filesystem e.a. /rpool/ROOT/pve-1. This only happens on non sync workloads, sync workloads give roughly the same results.

I tested both from the host os, not from inside a container. It seems that the container filesystem spends more time flushing to disk?

I guess there must a logic explanation, but I can't seem te find either here on the forum or google.

container filesystem:
Code:
READ: bw=246MiB/s (258MB/s), 59.8MiB/s-62.8MiB/s (62.7MB/s-65.8MB/s), io=7410MiB (7770MB), run=30104-30104msec
  WRITE: bw=61.9MiB/s (64.9MB/s), 14.0MiB/s-15.8MiB/s (15.7MB/s-16.5MB/s), io=1862MiB (1953MB), run=30104-30104msec

local filesystem:
Code:
READ: bw=524MiB/s (549MB/s), 128MiB/s-133MiB/s (135MB/s-140MB/s), io=15.3GiB (16.5GB), run=30001-30001msec
  WRITE: bw=132MiB/s (138MB/s), 32.3MiB/s-33.4MiB/s (33.9MB/s-35.0MB/s), io=3949MiB (4141MB), run=30001-30001msec

fio config:
Code:
 rw=rw
 rwmixread=80
 size=500M
 directory=[either the local or container filesystem]
 fadvise_hint=0
 blocksize=8k
 direct=1
 numjobs=4
 nrfiles=1
 runtime=30
 ioengine=libaio
 time_based
 
Last edited:
No information on ashift, volblocksize, recordsize, compression type.

But the performance inside the guest is what matters the most so thats where your tests should be run. Zvol's are most suited for that task. Datasets better for local access.
 
zvols have fixed blocksize, whereas datasets/filesystems have variable blocksize up to 128k (default), so it makes a huge difference in real-world-performance, if you store e.g. a 10 MB logfile on it. The logfile can be compressed heavily with a large blocksize, where it cannot on a zvol with e.g. 8K blocksize and 4K ashift. Storing on a filesystem will be much faster, because you need less space that is written faster. Therefore @chrcoluk asked about it.
 
Yep we need this information from the OP as how this is all configured can play a massive part on the performance.
 
Last edited:
No information on ashift, volblocksize, recordsize, compression type.
The first 2 should not matter.
Both are using the same rpool so ashift is identical.
Both are datasets so volblocksize isn't used.

Recordsize and compression of cause would be useful to know.
 
Thank you for the replies.

It's one pool with different performance on different datasets. I have posted the properties of the datasets and pool.

Code:
NAME   PROPERTY                       VALUE                          SOURCE
rpool  size                           928G                           -
rpool  capacity                       48%                            -
rpool  altroot                        -                              default
rpool  health                         ONLINE                         -
rpool  guid                           13579433081099798997           -
rpool  version                        -                              default
rpool  bootfs                         rpool/ROOT/pve-1               local
rpool  delegation                     on                             default
rpool  autoreplace                    off                            default
rpool  cachefile                      -                              default
rpool  failmode                       wait                           default
rpool  listsnapshots                  off                            default
rpool  autoexpand                     off                            default
rpool  dedupratio                     1.00x                          -
rpool  free                           478G                           -
rpool  allocated                      450G                           -
rpool  readonly                       off                            -
rpool  ashift                         12                             local
rpool  comment                        -                              default
rpool  expandsize                     -                              -
rpool  freeing                        0                              -
rpool  fragmentation                  7%                             -
rpool  leaked                         0                              -
rpool  multihost                      off                            default
rpool  checkpoint                     -                              -
rpool  load_guid                      2091712894548477348            -
rpool  autotrim                       off                            default
rpool  compatibility                  off                            default
rpool  feature@async_destroy          enabled                        local
rpool  feature@empty_bpobj            active                         local
rpool  feature@lz4_compress           active                         local
rpool  feature@multi_vdev_crash_dump  enabled                        local
rpool  feature@spacemap_histogram     active                         local
rpool  feature@enabled_txg            active                         local
rpool  feature@hole_birth             active                         local
rpool  feature@extensible_dataset     active                         local
rpool  feature@embedded_data          active                         local
rpool  feature@bookmarks              enabled                        local
rpool  feature@filesystem_limits      enabled                        local
rpool  feature@large_blocks           enabled                        local
rpool  feature@large_dnode            enabled                        local
rpool  feature@sha512                 enabled                        local
rpool  feature@skein                  enabled                        local
rpool  feature@edonr                  enabled                        local
rpool  feature@userobj_accounting     active                         local
rpool  feature@encryption             enabled                        local
rpool  feature@project_quota          active                         local
rpool  feature@device_removal         enabled                        local
rpool  feature@obsolete_counts        enabled                        local
rpool  feature@zpool_checkpoint       enabled                        local
rpool  feature@spacemap_v2            active                         local
rpool  feature@allocation_classes     enabled                        local
rpool  feature@resilver_defer         enabled                        local
rpool  feature@bookmark_v2            enabled                        local
rpool  feature@redaction_bookmarks    enabled                        local
rpool  feature@redacted_datasets      enabled                        local
rpool  feature@bookmark_written       enabled                        local
rpool  feature@log_spacemap           active                         local
rpool  feature@livelist               enabled                        local
rpool  feature@device_rebuild         enabled                        local
rpool  feature@zstd_compress          active                         local
rpool  feature@draid                  enabled                        local

Code:
rpool/ROOT/pve-1                                    type                            filesystem                      -
rpool/ROOT/pve-1                                    creation                        Mon Mar  8 10:01 2021           -
rpool/ROOT/pve-1                                    used                            4.53G                           -
rpool/ROOT/pve-1                                    available                       450G                            -
rpool/ROOT/pve-1                                    referenced                      4.53G                           -
rpool/ROOT/pve-1                                    compressratio                   1.69x                           -
rpool/ROOT/pve-1                                    mounted                         yes                             -
rpool/ROOT/pve-1                                    quota                           none                            default
rpool/ROOT/pve-1                                    reservation                     none                            default
rpool/ROOT/pve-1                                    recordsize                      128K                            default
rpool/ROOT/pve-1                                    mountpoint                      /                               local
rpool/ROOT/pve-1                                    sharenfs                        off                             default
rpool/ROOT/pve-1                                    checksum                        on                              default
rpool/ROOT/pve-1                                    compression                     lz4                             inherited from rpool
rpool/ROOT/pve-1                                    atime                           off                             inherited from rpool
rpool/ROOT/pve-1                                    devices                         on                              default
rpool/ROOT/pve-1                                    exec                            on                              default
rpool/ROOT/pve-1                                    setuid                          on                              default
rpool/ROOT/pve-1                                    readonly                        off                             default
rpool/ROOT/pve-1                                    zoned                           off                             default
rpool/ROOT/pve-1                                    snapdir                         hidden                          default
rpool/ROOT/pve-1                                    aclmode                         discard                         default
rpool/ROOT/pve-1                                    aclinherit                      restricted                      default
rpool/ROOT/pve-1                                    createtxg                       9                               -
rpool/ROOT/pve-1                                    canmount                        on                              default
rpool/ROOT/pve-1                                    xattr                           on                              default
rpool/ROOT/pve-1                                    copies                          1                               default
rpool/ROOT/pve-1                                    version                         5                               -
rpool/ROOT/pve-1                                    utf8only                        off                             -
rpool/ROOT/pve-1                                    normalization                   none                            -
rpool/ROOT/pve-1                                    casesensitivity                 sensitive                       -
rpool/ROOT/pve-1                                    vscan                           off                             default
rpool/ROOT/pve-1                                    nbmand                          off                             default
rpool/ROOT/pve-1                                    sharesmb                        off                             default
rpool/ROOT/pve-1                                    refquota                        none                            default
rpool/ROOT/pve-1                                    refreservation                  none                            default
rpool/ROOT/pve-1                                    guid                            8969698150623594166             -
rpool/ROOT/pve-1                                    primarycache                    all                             default
rpool/ROOT/pve-1                                    secondarycache                  all                             default
rpool/ROOT/pve-1                                    usedbysnapshots                 0B                              -
rpool/ROOT/pve-1                                    usedbydataset                   4.53G                           -
rpool/ROOT/pve-1                                    usedbychildren                  0B                              -
rpool/ROOT/pve-1                                    usedbyrefreservation            0B                              -
rpool/ROOT/pve-1                                    logbias                         latency                         default
rpool/ROOT/pve-1                                    objsetid                        135                             -
rpool/ROOT/pve-1                                    dedup                           off                             default
rpool/ROOT/pve-1                                    mlslabel                        none                            default
rpool/ROOT/pve-1                                    sync                            standard                        inherited from rpool
rpool/ROOT/pve-1                                    dnodesize                       legacy                          default
rpool/ROOT/pve-1                                    refcompressratio                1.69x                           -
rpool/ROOT/pve-1                                    written                         4.53G                           -
rpool/ROOT/pve-1                                    logicalused                     7.46G                           -
rpool/ROOT/pve-1                                    logicalreferenced               7.46G                           -
rpool/ROOT/pve-1                                    volmode                         default                         default
rpool/ROOT/pve-1                                    filesystem_limit                none                            default
rpool/ROOT/pve-1                                    snapshot_limit                  none                            default
rpool/ROOT/pve-1                                    filesystem_count                none                            default
rpool/ROOT/pve-1                                    snapshot_count                  none                            default
rpool/ROOT/pve-1                                    snapdev                         hidden                          default
rpool/ROOT/pve-1                                    acltype                         off                             default
rpool/ROOT/pve-1                                    context                         none                            default
rpool/ROOT/pve-1                                    fscontext                       none                            default
rpool/ROOT/pve-1                                    defcontext                      none                            default
rpool/ROOT/pve-1                                    rootcontext                     none                            default
rpool/ROOT/pve-1                                    relatime                        on                              temporary
rpool/ROOT/pve-1                                    redundant_metadata              all                             default
rpool/ROOT/pve-1                                    overlay                         on                              default
rpool/ROOT/pve-1                                    encryption                      off                             default
rpool/ROOT/pve-1                                    keylocation                     none                            default
rpool/ROOT/pve-1                                    keyformat                       none                            default
rpool/ROOT/pve-1                                    pbkdf2iters                     0                               default
rpool/ROOT/pve-1                                    special_small_blocks            0                               default
rpool/ROOT/pve-1                                    com.sun:auto-snapshot           false                           inherited from rpool
 
Last edited:
Code:
rpool/data/subvol-106-disk-0                        creation                        Mon Apr  4 13:16 2022           -
rpool/data/subvol-106-disk-0                        used                            5.02G                           -
rpool/data/subvol-106-disk-0                        available                       2.98G                           -
rpool/data/subvol-106-disk-0                        referenced                      5.02G                           -
rpool/data/subvol-106-disk-0                        compressratio                   1.33x                           -
rpool/data/subvol-106-disk-0                        mounted                         yes                             -
rpool/data/subvol-106-disk-0                        quota                           none                            default
rpool/data/subvol-106-disk-0                        reservation                     none                            default
rpool/data/subvol-106-disk-0                        recordsize                      128K                            default
rpool/data/subvol-106-disk-0                        mountpoint                      /rpool/data/subvol-106-disk-0   default
rpool/data/subvol-106-disk-0                        sharenfs                        off                             default
rpool/data/subvol-106-disk-0                        checksum                        on                              default
rpool/data/subvol-106-disk-0                        compression                     lz4                             inherited from rpool
rpool/data/subvol-106-disk-0                        atime                           off                             inherited from rpool
rpool/data/subvol-106-disk-0                        devices                         on                              default
rpool/data/subvol-106-disk-0                        exec                            on                              default
rpool/data/subvol-106-disk-0                        setuid                          on                              default
rpool/data/subvol-106-disk-0                        readonly                        off                             default
rpool/data/subvol-106-disk-0                        zoned                           off                             default
rpool/data/subvol-106-disk-0                        snapdir                         hidden                          default
rpool/data/subvol-106-disk-0                        aclmode                         discard                         default
rpool/data/subvol-106-disk-0                        aclinherit                      restricted                      default
rpool/data/subvol-106-disk-0                        createtxg                       5645285                         -
rpool/data/subvol-106-disk-0                        canmount                        on                              default
rpool/data/subvol-106-disk-0                        xattr                           sa                              local
rpool/data/subvol-106-disk-0                        copies                          1                               default
rpool/data/subvol-106-disk-0                        version                         5                               -
rpool/data/subvol-106-disk-0                        utf8only                        off                             -
rpool/data/subvol-106-disk-0                        normalization                   none                            -
rpool/data/subvol-106-disk-0                        casesensitivity                 sensitive                       -
rpool/data/subvol-106-disk-0                        vscan                           off                             default
rpool/data/subvol-106-disk-0                        nbmand                          off                             default
rpool/data/subvol-106-disk-0                        sharesmb                        off                             default
rpool/data/subvol-106-disk-0                        refquota                        8G                              local
rpool/data/subvol-106-disk-0                        refreservation                  none                            default
rpool/data/subvol-106-disk-0                        guid                            16519904382576469181            -
rpool/data/subvol-106-disk-0                        primarycache                    all                             default
rpool/data/subvol-106-disk-0                        secondarycache                  all                             default
rpool/data/subvol-106-disk-0                        usedbysnapshots                 0B                              -
rpool/data/subvol-106-disk-0                        usedbydataset                   5.02G                           -
rpool/data/subvol-106-disk-0                        usedbychildren                  0B                              -
rpool/data/subvol-106-disk-0                        usedbyrefreservation            0B                              -
rpool/data/subvol-106-disk-0                        logbias                         latency                         default
rpool/data/subvol-106-disk-0                        objsetid                        89968                           -
rpool/data/subvol-106-disk-0                        dedup                           off                             default
rpool/data/subvol-106-disk-0                        mlslabel                        none                            default
rpool/data/subvol-106-disk-0                        sync                            standard                        inherited from rpool
rpool/data/subvol-106-disk-0                        dnodesize                       legacy                          default
rpool/data/subvol-106-disk-0                        refcompressratio                1.33x                           -
rpool/data/subvol-106-disk-0                        written                         5.02G                           -
rpool/data/subvol-106-disk-0                        logicalused                     6.65G                           -
rpool/data/subvol-106-disk-0                        logicalreferenced               6.65G                           -
rpool/data/subvol-106-disk-0                        volmode                         default                         default
rpool/data/subvol-106-disk-0                        filesystem_limit                none                            default
rpool/data/subvol-106-disk-0                        snapshot_limit                  none                            default
rpool/data/subvol-106-disk-0                        filesystem_count                none                            default
rpool/data/subvol-106-disk-0                        snapshot_count                  none                            default
rpool/data/subvol-106-disk-0                        snapdev                         hidden                          default
rpool/data/subvol-106-disk-0                        acltype                         posix                           local
rpool/data/subvol-106-disk-0                        context                         none                            default
rpool/data/subvol-106-disk-0                        fscontext                       none                            default
rpool/data/subvol-106-disk-0                        defcontext                      none                            default
rpool/data/subvol-106-disk-0                        rootcontext                     none                            default
rpool/data/subvol-106-disk-0                        relatime                        off                             default
rpool/data/subvol-106-disk-0                        redundant_metadata              all                             default
rpool/data/subvol-106-disk-0                        overlay                         on                              default
rpool/data/subvol-106-disk-0                        encryption                      off                             default
rpool/data/subvol-106-disk-0                        keylocation                     none                            default
rpool/data/subvol-106-disk-0                        keyformat                       none                            default
rpool/data/subvol-106-disk-0                        pbkdf2iters                     0                               default
rpool/data/subvol-106-disk-0                        special_small_blocks            0                               default
rpool/data/subvol-106-disk-0                        com.sun:auto-snapshot           false                           inherited from rpool