[SOLVED] Bug? zpool auto-trim=on leads to high iowait on SSD pool

[rholighaus]

Hi,

Our system has the following setup. It boots from a USB stick which holds the /root filesystem to prevent ZFS issues when booting as it doesn't support UEFI boot yet.
The pool rpool consists of Samsung pm1633a enterprise SSDs in an external SAS enclose and ususally perform well.

After issuing the trim command, we have decided to use the auto-trim feature, only to realise it really drivers up the iowait to an average of 6-8%. At around 09:00 this morning I issued a zpool set auto-trim=off rpool and the iowait dropped back to almost nowhere - like before the auto-trim feature was enabled. Looks like auto-trim causes high iowait.

Is this a know bug?

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 0 days 01:09:40 with 0 errors on Sun May 10 01:33:43 2020
config:

        NAME             STATE     READ WRITE CKSUM
        rpool            ONLINE       0     0     0
          mirror-0       ONLINE       0     0     0
            E1S24-part2  ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 09:48:56 AM CEST)
            E1S25-part2  ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 09:48:57 AM CEST)
          mirror-1       ONLINE       0     0     0
            E1S20        ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 10:06:42 AM CEST)
            E1S21        ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 10:06:42 AM CEST)
          mirror-2       ONLINE       0     0     0
            E1S22        ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 10:01:44 AM CEST)
            E1S23        ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 10:01:45 AM CEST)
          mirror-3       ONLINE       0     0     0
            E1S19        ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 11:25:31 AM CEST)
            E1S18        ONLINE       0     0     0  (100% trimmed, completed at Wed 03 Jun 2020 11:25:30 AM CEST)

errors: No known data errors

pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

 

[mira]

The manpage mentions the following:

autotrim=on|off
             When set to on space which has been recently freed, and is no longer allocated by the pool, will be periodically trimmed.  This allows block device vdevs which support BLKDISCARD, such as SSDs,
             or file vdevs on which the underlying file system supports hole-punching, to reclaim unused blocks.  The default setting for this property is off.

             Automatic TRIM does not immediately reclaim blocks after a free. Instead, it will optimistically delay allowing smaller ranges to be aggregated in to a few larger ones.  These can then be issued
             more efficiently to the storage.

             Be aware that automatic trimming of recently freed data blocks can put significant stress on the underlying storage devices.  This will vary depending of how well the specific device handles
             these commands.  For lower end devices it is often possible to achieve most of the benefits of automatic trimming by running an on-demand (manual) TRIM periodically using the zpool trim command.

So this is expected behaviour. It would probably be best to run the trim command on a timer.

 

[rholighaus]

Thank you, I'll mark this as SOLVED. And sorry for not reading the manual deeper...