ZFS Strange behave

[f!R3W4LL]

Hello Together

Recently switched from TrueNas Core to OMV and both systems are managed under Proxmox.

My system:
Asrock X570D4U-2l2t motherboard
64GB ECC Ram of which 16GB to OMV
4x8TB Ironwolf HDDs
and a 20GB LVM SSD from Proxmox for the system.
The hard disks are individually forwarded to the VM via the console, i.e. via Passtrought.

Now yesterday everything was nicely configured with all permissions and data placed on it. Now I notice that the transfer/copy behave very strange.

For example, if I move or copy a movie of 4GB from one folder to another, then it starts at 500mbs and a short time later it drops to almost 100mbs and jumps briefly to 300mbs and then down again. Always in this pattern, up and down until the file is moved or copied.

I find it strange because with Truenas I had this problem not or partially. So it went up and down from time to time. But the transfer speeds were definitely better and were over 300-500mbs. Even from a NVME SSD similar speeds are registered. But the ram is only at 50% capacity.

Now I'm wondering, have any of you also encountered a similar problem already? I find OMV much more intuitive and easy to use than Truenas, especially it is not so extremely packed with additional features which I don't need. The UI I personally find better especially from the scale but OMV 6 also looks very promising. Only I'm not sure yet if OMV6 works stable or not

happy to hear from you

Regards

 

[Dunuin]

That sounds like normal caching behavior. First ZFS inside your VM will cache all written data in its RAM using the ARC, then you maybe also got caching in the hosts RAM because of the cache mode of the VM. And then your HDDs got a internal RAM cache and only after these are full you see the real speed of your disks. So the 100-200 MB/s are the performance of your HDDs and all above is just caching.
So thats totally normal. It will start with great speed until the cache is full. Then the performance will drop to a low because the cache needs to be written to disk first before the cache can store data again. As soon as there is some free space in the cache it will write to the cache again and the performance will climb up to a medium speed. Then it will drop again and toggle between a low and a medium speed until the write operation has finished. By default ZFS only writes all 5 seconds to the disk so its always an up and down if the data you want to write is bigger than your cache.

By the way, that is low RAM for a ZFS. According to the rule of thumb of 4GB + 1GB RAM per 1TB of raw storage your ARC needs 36GB RAM. By default the ARC is 50% of your RAM. So if you got a 16GB OMV VM your ARC should only be 8GB. With TrueNAS up to 100% of your RAM will be used for the ARC. So if that TrueNAS VM also got 16GB you now got a smaller ARC so you see a worse performance.

You could run a arc_summary inside your OMV VM to see how big your ARC actually is.

And also keep in mind that you don't use a real physical passthrough if you passthrough individual disks. You are still working with virtual disks but just that the data of that virtual disk is stored on physical disks. So you still get that virtualization overhead you wouldn't get if you would PCI passthrough a HBA.

 

[f!R3W4LL]

Hi Dunuin

Many thanks for your Feedback.

But thats is really strange because: Truenas had the same config, same settings simply everything the same beside the OS so i dot understand.

In truenas i had speeds over 300mbs some time under!

I have read somewhere that the right config of OMV and ZFS should be done with Peoxmox and passtrought to the vm and so one and in OMV create a new pool with Ext4? Now what is the best way to do that? Modt performance but safe for my datas?