Slow ZFS encryption inside TrueNAS Scale

[mathiasdev]

I'm running TrueNAS SCALE ElectricEel 24.10 inside Proxmox VE 8.3.3 and the disk access is extremely slow when using encryption on datasets. I understand that encryption will hit the disk performance, but the speed is so low that it's not usable and I believe something might be wrong in my settings.

In TrueNAS I have one pool of two mirrored 8 TB enterprise drives connected via a LSI 9211-8I SAS card using passthrough.

Inside my Pool:

Dataset 1: no encryption - read/write speed >280 MB/s (only limited by my 2.5G network)
Dataset 2: encryption AES-256-GCM - read/write speed ~30 MB (sometimes dropping way below)

My VM is using following settings:

Memory: 16 GB
Processors: 4(1 sockets, 4 cores) [x86-64-v2-AES, flags=+aes]
BIOS: OVMF (UEFI)
Machine: q35
SCSI Controller: VirtIO SCSI single
PCI Device (hostpci0): mapping=SAS-controller

My best guess is that the q35 is not actually using hardware acceleration for AES-NI instructions.

Any thoughts?

 

[IronWisp]

What CPU does the machine have? Older xeons can cause bottlenecks while encrypting data
Are you using disk cache?

 

[mathiasdev]

I'm using an Intel i5 12400 (6C, 12T) and no additional disk cache. I also have another server running TrueNAS Scale on bare metal with the same encryption enabled which does not seem to affect the performance much at all.

 

[jshl]

It looks like you found my other thread about this, but I'm going to reply here because your title is a better description of the issue... this appears to be an issue with Truenas (possibly other guest OSes too!) but not what I originally thought, which was an issue with Proxmox's CPU types or flags.

My workaround is to simulate a specific CPU, not a generic x86-64* variant. Since you're never going to live-migrate a VM with a PCIe card and drives attached to it, you should probably just use the Host CPU type. For a cluster with different hardware types where you do want to migrate VMs, you would want to pick a specific CPU generation supported by all your hosts (i.e. equal to the oldest CPU in your cluster), that way you can still do migrations without the VM seeing a hardware change and potentially having issues.