Help tuning PBS with SSD special mirror

Raito00

Member
Sep 10, 2019
24
1
23
43
Hi!

I plan to add special mirror to my PBS with 3x 1TB M.2

So this is way to do?

1. zpool add backups special mirror <device1_m.2> <device2_m2> <device3_m2>

2. zfs set special_small_blocks=8K backups

/# zdb -Lbbbs backups

Code:
8.79T completed (3259MB/s) estimated time remaining: 306093hr 44min 07sec       
        bp count:              52620622
        ganged count:                 0
        bp logical:       6870475302400      avg: 130566
        bp physical:      6398673716224      avg: 121600     compression:   1.07
        bp allocated:     9660833992704      avg: 183594     compression:   0.71
        bp deduped:                   0    ref>1:      0   deduplication:   1.00
        Normal class:     9654074339328     used: 26.84%
        Embedded log class          12288     used:  0.00%

        additional, non-pointer bps of type 0:        141
         number of (compressed) bytes:  number of bps
                         25:     26 **************************
                         26:      0
                         27:     11 ***********
                         28:      3 ***
                         29:      0
                         30:      0
                         31:      1 *
                         32:      0
                         33:      0
                         34:      0
                         35:      0
                         36:      0
                         37:      0
                         38:      0
                         39:      4 ****
                         40:      0
                         41:      0
                         42:      0
                         43:      0
                         44:      1 *
                         45:      0
                         46:      0
                         47:      0
                         48:      0
                         49:      1 *
                         50:      0
                         51:      0
                         52:      0
                         53:      1 *
                         54:      0
                         55:      0
                         56:      0
                         57:      4 ****
                         58:      1 *
                         59:      0
                         60:      0
                         61:      0
                         62:      0
                         63:      0
                         64:      1 *
                         65:      0
                         66:      3 ***
                         67:      0
                         68:      0
                         69:      0
                         70:      0
                         71:      0
                         72:      1 *
                         73:      0
                         74:      1 *
                         75:      0
                         76:      1 *
                         77:      0
                         78:      0
                         79:      0
                         80:      0
                         81:      1 *
                         82:      0
                         83:      0
                         84:      2 **
                         85:      1 *
                         86:      1 *
                         87:      1 *
                         88:      0
                         89:     11 ***********
                         90:      0
                         91:      1 *
                         92:      0
                         93:      0
                         94:      0
                         95:      1 *
                         96:      2 **
                         97:      2 **
                         98:      1 *
                         99:      0
                        100:      0
                        101:      1 *
                        102:      1 *
                        103:      0
                        104:      6 ******
                        105:      3 ***
                        106:      5 *****
                        107:      2 **
                        108:     10 **********
                        109:      5 *****
                        110:      4 ****
                        111:      3 ***
                        112:     17 *****************
        Dittoed blocks on same vdev: 2633201
        Dittoed blocks in same metaslab: 6

Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     -      -       -       -       -       -        -  unallocated
     2    32K      8K     72K     36K    4.00     0.00  object directory
     1   128K      4K     36K     36K   32.00     0.00      L1 object array
    33  16.5K   16.5K   1.16M     36K    1.00     0.00      L0 object array
    34   144K   20.5K   1.20M     36K    7.05     0.00  object array
     1    16K      4K     36K     36K    4.00     0.00  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     -      -       -       -       -       -        -  bpobj
     -      -       -       -       -       -        -  bpobj header
     -      -       -       -       -       -        -  SPA space map header
   140  2.19M    560K   4.92M     36K    4.00     0.00      L1 SPA space map
 2.38K   305M   71.1M    366M    153K    4.29     0.00      L0 SPA space map
 2.52K   307M   71.6M    371M    147K    4.29     0.00  SPA space map
     1   128K    128K    192K    192K    1.00     0.00  ZIL intent log
     1   128K      4K     24K     24K   32.00     0.00      L5 DMU dnode
     1   128K      4K     24K     24K   32.00     0.00      L4 DMU dnode
     1   128K      4K     24K     24K   32.00     0.00      L3 DMU dnode
     1   128K      8K     48K     48K   16.00     0.00      L2 DMU dnode
    81  10.1M   3.59M   11.0M    139K    2.82     0.00      L1 DMU dnode
 75.6K  1.18G    302M   1.78G   24.1K    4.00     0.02      L0 DMU dnode
 75.7K  1.19G    306M   1.79G   24.2K    3.99     0.02  DMU dnode
     2     8K      8K     60K     30K    1.00     0.00  DMU objset
     -      -       -       -       -       -        -  DSL directory
     -      -       -       -       -       -        -  DSL directory child map
     -      -       -       -       -       -        -  DSL dataset snap map
     5  33.5K      8K     72K   14.4K    4.19     0.00  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
 2.25M   288G   9.00G   54.0G   24.0K   31.96     0.60      L1 ZFS plain file
 47.7M  5.96T   5.81T   8.73T    187K    1.03    99.31      L0 ZFS plain file
 49.9M  6.24T   5.82T   8.78T    180K    1.07    99.91  ZFS plain file
 64.0K  8.00G    256M   1.50G   24.0K   32.00     0.02      L1 ZFS directory
  129K  2.01G    772M   4.53G   36.0K    2.66     0.05      L0 ZFS directory
  193K  10.0G   1.00G   6.03G   32.0K    9.97     0.07  ZFS directory
     1    512     512     24K     24K    1.00     0.00  ZFS master node
     -      -       -       -       -       -        -  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     1   128K      8K     72K     72K   16.00     0.00  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     -      -       -       -       -       -        -  DSL dataset next clones
     -      -       -       -       -       -        -  scan work queue
     -      -       -       -       -       -        -  ZFS user/group/project used
     -      -       -       -       -       -        -  ZFS user/group/project quota
     -      -       -       -       -       -        -  snapshot refcount tags
     -      -       -       -       -       -        -  DDT ZAP algorithm
     -      -       -       -       -       -        -  DDT statistics
     -      -       -       -       -       -        -  System attributes
     -      -       -       -       -       -        -  SA master node
     1  1.50K   1.50K     24K     24K    1.00     0.00  SA attr registration
     2    32K      8K     48K     24K    4.00     0.00  SA attr layouts
     -      -       -       -       -       -        -  scan translations
     -      -       -       -       -       -        -  deduplicated block
     -      -       -       -       -       -        -  DSL deadlist map
     -      -       -       -       -       -        -  DSL deadlist map hdr
     -      -       -       -       -       -        -  DSL dir clones
     -      -       -       -       -       -        -  bpobj subobj
     -      -       -       -       -       -        -  deferred free
     -      -       -       -       -       -        -  dedup ditto
    16   210K     26K    324K   20.2K    8.06     0.00  other
     1   128K      4K     24K     24K   32.00     0.00      L5 Total
     1   128K      4K     24K     24K   32.00     0.00      L4 Total
     1   128K      4K     24K     24K   32.00     0.00      L3 Total
     1   128K      8K     48K     48K   16.00     0.00      L2 Total
 2.31M   296G   9.25G   55.5G   24.0K   31.95     0.62      L1 Total
 47.9M  5.96T   5.81T   8.73T    187K    1.03    99.38      L0 Total
 50.2M  6.25T   5.82T   8.79T    179K    1.07   100.00  Total

Block Size Histogram

  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    512:    456   228K   228K    456   228K   228K      0      0      0
     1K:    308   356K   584K    308   356K   584K      0      0      0
     2K:    205   514K  1.07M    205   514K  1.07M      0      0      0
     4K:  2.82M  11.3G  11.3G    789  4.52M  5.59M      0      0      0
     8K:   186K  1.69G  13.0G  2.78K  33.3M  38.9M   384K  4.50G  4.50G
    16K:   235K  5.04G  18.0G   211K  3.36G  3.40G  2.62M  62.8G  67.3G
    32K:   468K  21.0G  39.0G  9.39K   419M  3.81G   359K  16.8G  84.1G
    64K:   820K  71.2G   110G  16.2K  1.59G  5.40G   654K  61.5G   146G
   128K:  45.7M  5.71T  5.82T  49.9M  6.24T  6.25T  46.2M  8.64T  8.79T
   256K:      0      0  5.82T      0      0  6.25T    437   160M  8.79T
   512K:      0      0  5.82T      0      0  6.25T      0      0  8.79T
     1M:      0      0  5.82T      0      0  6.25T      0      0  8.79T
     2M:      0      0  5.82T      0      0  6.25T      0      0  8.79T
     4M:      0      0  5.82T      0      0  6.25T      0      0  8.79T
     8M:      0      0  5.82T      0      0  6.25T      0      0  8.79T
    16M:      0      0  5.82T      0      0  6.25T      0      0  8.79T

                            capacity   operations   bandwidth  ---- errors ----
description                used avail  read write  read write  read write cksum
backups                   8.78T 24.0T   856     0 3.41M     0     0     0     0
  raidz2                  8.78T 24.0T   856     0 3.41M     0     0     0     0
    /dev/sda1                           143     0  592K     0     0     0     0
    /dev/sdb1                           144     0  586K     0     0     0     0
    /dev/sdc1                           140     0  569K     0     0     0     0
    /dev/sdd1                           143     0  591K     0     0     0     0
    /dev/sde1                           143     0  585K     0     0     0     0
    /dev/sdf1                           140     0  570K     0     0     0     0
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!