How to use write caching to sustain 10GbE transfers

[crm]

Hi, I use a container as SMB server connected with a 10gbe NIC. Underlying storage is a striped zpool consisting of 2x1TB HDDs added as mountpoint/subvol. Via async io and jumbo frames I get about 1GB/s from my workstation to SMB but just for a few seconds. After that speed is down to the pools HDD performance.

Is there any way to tweak this so I can sustain full speed of the NIC longer? Redundancy isn't an issue and I'm willing to lose unfinished transfers in case of power losses etc. If necessary I could commit about 100GB of RAM (maybe as ramdisk if it would work?). Read performance isn't an issue as well. Preferably I'd like to get this done without additional hardware. If absolutely necessary I could commit a Corsair Force MP510 960 GB NVME SSD. My usual filesize I'd like to push over SMB is about 50GB.

 

[LnxBil]

Just for understanding:
You have a big machine with at least 100 GB-RAM, 10 GBE and you're using 2x 1 TB spinners for storage? Wow ... looks like you tried to save on the wrong end. Even with 6 disk spinners in RAID0 it is hard to constantly saturate 10 GBE.

Now back to your problem:
How big is your ARC? Can you monitor it while transferring? As far as I understand SLOG, there is no benefit for streamed writes to disk.

 

[crm]

Even with 6 disk spinners in RAID0 it is hard to constantly saturate 10 GBE.

Thats prob the reason I did't buy 6 spinners and constantly delivering 1GB/s isn't a requirement. ARC is 64GB of which 10 are used.

How about ZIL in ramdisk?

 

[LnxBil]

How about ZIL in ramdisk?

ZIL is best for random synchronous writes. Stream writes (copying large files) bypass also the ARC, because it would poison the whole cache.

I think the best solution is to have an intermediate storage on NVME or RAM and then copy back to ZFS. IMHO there is no solution to the ZFS write speed that will not be fixed by more spindels.