ZFS - shifting ARC to L2ARC

[Mudgee Host]

Hi,

With the shortage of DDR5 I'm needing to optimise ram for a new Proxmox ve build. I'm not going to have 32GB of ram for the ZFS ARC as required by the storage (24TB). Im not confident to use BTSFS raid1.

I'd like to exploit L2ARC to its max. I can get a 500Gb m2 ssd the same price as for 8gb of ram. thats a lot more cache for the same price (albeit slower).

* what virtual disk formats / options should I look at that are L2ARC friendly (ie write caching)?
* how do I size L2ARC?
* how small can I get ARC before I get performance issues, how small can it go before it becomes unstable?

the box will have two storage pools, ssd based for system disks, hdd + L2ARC for large data

 

[Kingneutron]

You don't "format" an L2arc device, you just add it to the pool and ZFS takes care of it.

https://search.brave.com/search?q=zpool+add+l2arc+device

https://search.brave.com/search?q=h...ersation=08b32fddee760743dfb5fb8c77e708498d43

https://klarasystems.com/articles/openzfs-all-about-l2arc/

You can GPT partition the l2arc physical device to limit the partition size for the cache (you don't have to give it the whole drive.)

I would start by giving L2arc 16-32GB and monitor your arcstat / arc_summary output for about a week.

Note that you cannot mirror L2arc, they are disposable - but the data survives a reboot. You can "stack" L2arc if you need to give it more, just add another partition on the l2arc device and add it to the pool as cache.

You'd have to experiment with low sizes of l2arc, but higher sizes with limited system RAM have been known to cause instability.

 

[Mudgee Host]

You don't "format" an L2arc device, you just add it to the pool and ZFS takes care of it.

sorry I wasn't clear, the question was about the guest's virtual disk. https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache
ie are there any gotchas with guests that can impact zfs ability to cache on the host. drive encryption and the like. Its just blocks according to the host so I wouldn't think so.

You'd have to experiment with low sizes of l2arc, but higher sizes with limited system RAM have been known to cause instability.

hmm ok. I was reading about zfs causing oom
https://github.com/openzfs/zfs/issues/17920

 

[IsThisThingOn]

I'm not going to have 32GB of ram for the ZFS ARC as required by the storage (24TB).

Don't worry, that rule is old and bad and useless.
Proxmox will by default just use max 10% of your RAM. So if you have 32, that is 3,2.

* how do I size L2ARC?

Depends on the use case. L2ARC will just store things that evicted ARC. So if your ARC has a hit ratio of +90%, I would say no L2ARC at all.
Good thing about L2ARC is that you can add it later. Maybe your pool does not even need one.

 

[Mudgee Host]

Proxmox will by default just use max 10% of your RAM. So if you have 32, that is 3,2.

oh ok, thats fine. I could allocate ~6gb

Depends on the use case. L2ARC will just store things that evicted ARC. So if your ARC has a hit ratio of +90%, I would say no L2ARC at all.
Good thing about L2ARC is that you can add it later. Maybe your pool does not even need one.

thanks, I think thats a key message - test and find out
ps for anyone else reading; I found this pretty helpful to explain the inner workings https://www.snia.org/sites/default/...ices_for_OpenZFS_L2ARC_in_the_Era_of_NVMe.pdf

My takeaway, l2arc will only be of benefit in a low memory system for random reads of large blocks since it consumes ram itself that is better utilised as arc

 

[IsThisThingOn]

I would want to emphasize few points:

smaller volblock/record sizese yield more metadata blocks

This is especially relevant for Proxmox, since by default you will only with the small default 16k volblocksize.

Instead of having a huge RAW VM disk, that is a pita to backup and has a 16k volblocksize, you could also create a dataset with 1M record size for data and access that data over VirtioFS from the VMs.

Or you simply put the data into another TrueNAS host that shares its data over NFS with the VMs.

L2ARC is not reboot persistent

That is no longer the case. It is now. At least for TrueNAS, not 100% sure about Proxmox.

 

[Mudgee Host]

you could also create a dataset with 1M record size for data and access that data over VirtioFS from the VMs

VirtioFS looks really promising. thank you I'm going to try that out.