ZFS on host node high CPU and system load

[tjk]

In our prod setup, we run Truenas devices, which are based on ZFS and present that to our clusters over NFS and it works well, great performance, no complaints.

I am testing a few nodes that will have local storage, 8x1TB SSD in ZFS raid10 and vm's stored as zvols.

However any amount of IO inside any of the guest VM's drives the host CPU and load up to crazy levels, like 20-50 load and 500% CPU, tons of zvol processes and zfs_wr_int processes.

Is this a known issue with ZFS on the host node and just something you have to live with?

Node is AMD R9 5950X, 128G Ram, and 8xMicron 5200 Pro SSD's @ Sata 6G in a R10 setup.

See attached screen cap - I am running a single VM on the node running FIO doing 50/50 read/write inside the guest.

 

[Dunuin]

Did you set the pools block size from the default 8K to 16K (Datacenter -> Storage -> YourZFSPool -> Edit -> Block size) so that newly created zvols will be created with a volblocksize of 16K instead of 8K?
.

 

[tjk]

I tested with 8k and 16k block size, and moved the VM off and back on, same results.

 

[tjk]

Is anyone running ZFS on the host node using zvols with a bunch of VM's doing lots of IO and not having cpu load issues?

If so, pls share your setup. Crazy I can drive the host node load up to 30+ with a single VM and some IO inside that VM.

 

[virtus223]

These SSD´s are made for intensive read operations. Did you change your zfs cache settings? can you post also the zpool configuration, and zfs get all?

 

[tjk]

I set arc max to 8g and min to 4g. I get they are read intensive but to still have zvol take up that sort of cpu load on the host node is nuts.

root@pve4:~# zpool status
  pool: rpool
 state: ONLINE
config:

    NAME                                       STATE     READ WRITE CKSUM
    rpool                                      ONLINE       0     0     0
      mirror-0                                 ONLINE       0     0     0
        nvme-CT2000P5PSSD8_2141322EE6CB-part3  ONLINE       0     0     0
        nvme-CT2000P5PSSD8_2141322EFB6F-part3  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
config:

    NAME                                            STATE     READ WRITE CKSUM
    tank                                            ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        ata-Micron_5100_MTFDDAK960TCB_1728215B340D  ONLINE       0     0     0
        ata-Micron_5100_MTFDDAK960TCB_17151D76999F  ONLINE       0     0     0
      mirror-1                                      ONLINE       0     0     0
        ata-Micron_5100_MTFDDAK960TCB_17151D765BC1  ONLINE       0     0     0
        ata-Micron_5100_MTFDDAK960TCB_1728215C362E  ONLINE       0     0     0
      mirror-2                                      ONLINE       0     0     0
        ata-Micron_5100_MTFDDAK960TCB_1726215A71DC  ONLINE       0     0     0
        ata-Micron_5100_MTFDDAK960TCB_1728215C390D  ONLINE       0     0     0

errors: No known data errors
root@pve4:~#

zfs get all is too big to post (forum software complained), what parts do you want to see?

 

[virtus223]

so far i could only see 6 disks?

paste the zfs get all output here:
https://paste.debian.net/

 

[tjk]

Correct 6 drives, sorry.

https://pastebin.com/sqHDyVj9

I've also tested with different volblocksizes, from 4k, 8k, and 128k. There seems to be no agreement on the best volblocksize, at least searching through the threads here.

 

[tomee]

I have a similar hw-setup and the same symphoms, have you ever found a solution?

 

[tjk]

I have a similar hw-setup and the same symphoms, have you ever found a solution?

No, same thing on a new AMD EPYC with 8x NVME drives.

I don't see the load on TrueNAS when I present that as NFS to the nodes and run the same testing.

 

[RolandK]

@tjk , have a look at this thread https://forum.proxmox.com/threads/high-io-wait-during-backups-after-upgrading-to-proxmox-7.113790/