ZFS bad performance

[showiproute]

Hello everyone,

I am fighting for a good IO performance on my server.
Anytime when I am doing some random stuff (e.g. copying within a VM) my whole server is like freezing.

Currently I am restoring a VM on one SSD with a specific ZFS pool. Parallel my webserver which uses a different ZFS pool on a totally different SSD freezes and is unable to respond.

Proxmox says that my IO delay is somewhere around 70 - 80 %.

zpool iostat -v

says:

                                                          capacity     operations     bandwidth
pool                                                    alloc   free   read  write   read  write
------------------------------------------------------  -----  -----  -----  -----  -----  -----
SSD_500GB                                                140G   324G    164    615  2.03M  32.9M
  ata-Samsung_SSD_850_EVO_500GB_S3R3NF0JA52227V          140G   324G    164    615  2.03M  32.9M
------------------------------------------------------  -----  -----  -----  -----  -----  -----
SSD_NVMe                                                 140G   748G    392     48  3.97M  1.31M
  nvme-Force_MP510_194982380001288848EA                  140G   748G    392     48  3.97M  1.31M
------------------------------------------------------  -----  -----  -----  -----  -----  -----
Storage_10TB                                             974G  8.14T      5      0  53.2K  14.9K
  ata-ST10000VN0008-2JJ101_ZL00NQWQ                      974G  8.14T      1      0  11.9K  13.7K
logs                                                        -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part1                         12K  31.5G      3      0  41.3K  1.24K
cache                                                       -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part6                       8.18M  63.0G      0      0    136  8.33K
------------------------------------------------------  -----  -----  -----  -----  -----  -----
Storage_12TB                                            2.94T  7.97T      0      0  3.61K  6.03K
  ata-HGST_HUH721212ALE600_8DG62YJX                     2.94T  7.97T      0      0  3.15K  5.10K
logs                                                        -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part2                         12K  31.5G      0      0    474    956
cache                                                       -      -      -      -      -      -
  nvme-Samsung_SSD_970_PRO_512GB_S5JYNS0N801547H-part7   454K  63.0G      0      0    136    480
------------------------------------------------------  -----  -----  -----  -----  -----  -----
Storage_14TB                                            7.17T  5.55T      0      0  4.07K  6.58K
  ata-WDC_WUH721414ALE6L4_Z2JJD6BT                      7.17T  5.55T      0      0  3.61K  5.64K
logs                                                        -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part3                        140K  31.5G      0      0    473    955
cache                                                       -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part8                        386K  63.0G      0      0    136    411
------------------------------------------------------  -----  -----  -----  -----  -----  -----
Storage_6TB                                              929G  4.55T      0      0  2.11K  3.94K
  ata-ST6000VN0041-2EL11C_ZA15SVNM                       929G  4.55T      0      0  1.65K     3K
logs                                                        -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part4                         12K  31.5G      0      0    475    958
cache                                                       -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part9                        302K  63.0G      0      0    136    332
------------------------------------------------------  -----  -----  -----  -----  -----  -----
Storage_8TB                                             2.38T  4.89T      0      0  4.76K  6.77K
  ata-WDC_WD80EFZX-68UW8N0_VKHN516X                     2.38T  4.89T      0      0  4.29K  5.83K
logs                                                        -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part5                         24K  31.5G      0      0    474    957
cache                                                       -      -      -      -      -      -
  nvme-eui.0025385801411b9b-part10                      1.12M  63.0G      0      0    136  1.15K
------------------------------------------------------  -----  -----  -----  -----  -----  -----

I have spend the last couple of days trying to debug/fix those issues but did not find any solution.

My server config would be:
8 x Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz
128 GB RAM (currently while restoring backup ~ 50% are being used)


Does anyone have any idea what to do ?

 

[H4R0]

Running l2arc and slog on the same nvme is very bad for performance.

Fyi your slog is not redundant, if it fails the temporary buffer is gone and that consumer nvme will die fast as slog.

Remove the slog it will just hurt performance or use something like intel optane with high mixed io for it.


You can set a rate limit for vm's to prevent io stalls.

Please post "zpool status" and "cat /etc/modprobe.d/zfs.conf"

Did you enable deduplication on any vdev by any chance ?

 

[showiproute]

I know that my slog is not redudant - I just added this consumer NVMe to see if it brings any enhancement. It didn't so far.

My zpool status would be:

root@proxmox1:~# zpool status
  pool: SSD_500GB
 state: ONLINE
  scan: scrub repaired 0B in 0 days 00:04:34 with 0 errors on Sun Nov  8 00:28:36 2020
config:

        NAME                                             STATE     READ WRITE CKSUM
        SSD_500GB                                        ONLINE       0     0     0
          ata-Samsung_SSD_850_EVO_500GB_S3R3NF0JA52227V  ONLINE       0     0     0

errors: No known data errors

  pool: SSD_NVMe
 state: ONLINE
  scan: scrub repaired 0B in 0 days 00:08:06 with 0 errors on Sun Nov  8 00:32:09 2020
config:

        NAME                                     STATE     READ WRITE CKSUM
        SSD_NVMe                                 ONLINE       0     0     0
          nvme-Force_MP510_194982380001288848EA  ONLINE       0     0     0

errors: No known data errors

  pool: Storage_10TB
 state: ONLINE
  scan: none requested
config:

        NAME                                 STATE     READ WRITE CKSUM
        Storage_10TB                         ONLINE       0     0     0
          ata-ST10000VN0008-2JJ101_ZL00NQWQ  ONLINE       0     0     0
        logs
          nvme-eui.0025385801411b9b-part1    ONLINE       0     0     0
        cache
          nvme-eui.0025385801411b9b-part6    ONLINE       0     0     0

errors: No known data errors

  pool: Storage_12TB
 state: ONLINE
  scan: none requested
config:

        NAME                                                    STATE     READ WRITE CKSUM
        Storage_12TB                                            ONLINE       0     0     0
          ata-HGST_HUH721212ALE600_8DG62YJX                     ONLINE       0     0     0
        logs
          nvme-eui.0025385801411b9b-part2                       ONLINE       0     0     0
        cache
          nvme-Samsung_SSD_970_PRO_512GB_S5JYNS0N801547H-part7  ONLINE       0     0     0

errors: No known data errors

  pool: Storage_14TB
 state: ONLINE
  scan: none requested
config:

        NAME                                STATE     READ WRITE CKSUM
        Storage_14TB                        ONLINE       0     0     0
          ata-WDC_WUH721414ALE6L4_Z2JJD6BT  ONLINE       0     0     0
        logs
          nvme-eui.0025385801411b9b-part3   ONLINE       0     0     0
        cache
          nvme-eui.0025385801411b9b-part8   ONLINE       0     0     0

errors: No known data errors

  pool: Storage_6TB
 state: ONLINE
  scan: none requested
config:

        NAME                                STATE     READ WRITE CKSUM
        Storage_6TB                         ONLINE       0     0     0
          ata-ST6000VN0041-2EL11C_ZA15SVNM  ONLINE       0     0     0
        logs
          nvme-eui.0025385801411b9b-part4   ONLINE       0     0     0
        cache
          nvme-eui.0025385801411b9b-part9   ONLINE       0     0     0

errors: No known data errors

  pool: Storage_8TB
 state: ONLINE
  scan: none requested
config:

        NAME                                 STATE     READ WRITE CKSUM
        Storage_8TB                          ONLINE       0     0     0
          ata-WDC_WD80EFZX-68UW8N0_VKHN516X  ONLINE       0     0     0
        logs
          nvme-eui.0025385801411b9b-part5    ONLINE       0     0     0
        cache
          nvme-eui.0025385801411b9b-part10   ONLINE       0     0     0

errors: No known data errors

/etc/modprobe.d/zfs.conf is not existing.

Deduplication is off on all drives.

 

[area51tazz]

I'd like to follow this as my

etc/modprobe.d/zfs.conf

doesn't exist either, and when my ZFS pool scrubs, it kills my guests. Just curious if we create zfs.conf there if it'll have an effect.

 

[Dunuin]

You only need the "/etc/modprobe.d/zfs.conf" if you want to overwrite the default configuration. Like limiting the ARC size so ZFS isn't using 50% of your total RAM for caching.