VM to VM SMB performance: What do I need to configure?

[phil1c]

Ok, the longer version of what I'm trying to do is here. The shorter version is as follows:

I am trying to condense a homelab to a single "box". I want to use Proxmox as the hypervisor which will control three critical VMs:

1. FreeNAS VM: pretty GUI and lots of controls for a ZFS-based NAS. Two controllers are passed to the VM to direct-control disks. Will share pool via SMB. NOT FOR VM STORAGE/USE. This is only for media/personal documents/etc. VMs will access this but will NOT run off of the pool directly.
   1. Inb4 "Why not use Proxmox to host the pool?!". Because I like GUIs. I do NOT want to manage the pool or users or samba or anything else via a CLI every time I need to make a change or tweak.
2. Ubuntu 20.04 VM: VM for running dockers and eventually a couple other bits and bobs.
3. WIn10 VM: Plex server and convenient remote device, if needed. Has GPU passthrough.

I have all three set up, GPU passthrough, LSI HBA and onboard SAS controller and all. However, two big issues, I think related to the FN VM:

1. Performing large-file throughput testing on the pool from within the FN VM, as the bulk of file movement will be large (85% >10gb, 20%+ >50GB), simulating with dd using /dev/zero as the source (compression is OFF on the dataset used for testing) get's me slower than expected and confusingly flipped results:
   1. Reads are around 500MB/s, writes are 850MB/s. I'm using a 12 disk pool with 2x 6 disk RaidZ2 VDEVs filled with shucked 12TB easystores. They are not SMR and based on individual-drive testing prior to building the pool, they perform about 225MB/s read/200MB/s write.
2. Sharing the zfs dataset with the Ubuntu VM, and reading an existing file fresh from the disks (as in not previously accessed so not cached in ARC nor in the Ubu cache) I'm getting ~200MB/s read. Renaming that file and writing back to the pool I see ~500MB/s writes, which is the theoretical limit of the read of my sata SSD. Renaming the duplicate on the pool and copying back again to ubuntu get's me 400MB/s which is theoretical write limit of the SSD.

My two problems are:

1. Why are my reads from within FN itself so slow compared to writes?
2. Why are my reads from the pool when shared via SMB to ubuntu even slower?

The abridged "I dont think this" list in response to #2 is:

• It's not the "network" as iperf is >=10Gbps.
• I don't think it's smbd as from ARC to VM is maxing out my SSD.
• It's not disk access as raw speeds reading from disk from within FN are far faster (190MB/s vs 500MB/s) though still not what expected.
• It's not the SSD as, again, ARC to SSD maxes out the SSD write speed.
• It's not hardware as importing the pool into ZFS on Proxmox and sharing reaches 800MB/s write/850MB/s read.
• Throwing more cores at each VM did nothing to change the performance.

I have done more testing than referenced here (see the FreeNas forum post I linked above for more) and I have done even more today, most interestingly using ESXI (I don't want to use ESXI, nor stir up a debate about the two, I just wanted to see what happened) and it performed better but not "theoretical max" either.

Are there.... things (? sorry, I'm coming from unraid after 5 years and while fighting this and my previous attempted setup up with two machines has taught me a lot over the last few months, I'm not familiar with what exactly to use to describe "tunables") that I can modify or tweak related to Proxmox that can get better VM performance?

I can provide any confs or outputs requested, and, again, there is more detail in my other post, but repeating that post in it's entirety seemed wasteful and this is a bit more tailored to Proxmox, not FreeNas.

 

[phil1c]

Bump