ZFS RAM usage more than 90% of the system

[saphirblanc]

Hello,
We've upgraded our PVE cluster of ndoes from 5.4 to 6.2 few weeks ago, we also upgraded our ZFS pool to the version 0.8.4. We also added an iSCSI storage (FreeNAS) which hosts now most of the storage.

I'm now seeing that the ZFS cache is like eating a little bit too much of RAM (this is certainly some paranoia).

On a 256GB RAM ECC server, it's eating around 100GB of RAM which leads to a usage of RAM more than 90% (including our VM's usage of course).

Curiously, I've never seen this behavior on 5.4 (as much as now). I just wanted to know, should I be worry that my system is using more than 90% of the RAM available in the server ? I know ZFS uses as much as possible, but...

Here is the arc_summary of one host :

------------------------------------------------------------------------
ZFS Subsystem Report                            Tue Aug 11 07:37:55 2020
Linux 5.4.44-1-pve                                            0.8.4-pve1
Machine: athos (x86_64)                                       0.8.4-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    84.4 %  106.2 GiB
        Target size (adaptive):                        84.4 %  106.2 GiB
        Min size (hard limit):                          6.2 %    7.9 GiB
        Max size (high water):                           16:1  125.8 GiB
        Most Frequently Used (MFU) cache size:         57.1 %   55.9 GiB
        Most Recently Used (MRU) cache size:           42.9 %   41.9 GiB
        Metadata cache size (hard limit):              75.0 %   94.3 GiB
        Metadata cache size (current):                 10.8 %   10.2 GiB
        Dnode cache size (hard limit):                 10.0 %    9.4 GiB
        Dnode cache size (current):                     0.5 %   50.8 MiB

ARC hash breakdown:
        Elements max:                                              31.7M
        Elements current:                              83.8 %      26.6M
        Collisions:                                                 1.2G
        Chain max:                                                    11
        Chains:                                                     6.3M

ARC misc:
        Deleted:                                                    2.9G
        Mutex misses:                                             768.1k
        Eviction skips:                                           954.7M

ARC total accesses (hits + misses):                                13.0G
        Cache hit ratio:                               80.7 %      10.5G
        Cache miss ratio:                              19.3 %       2.5G
        Actual hit ratio (MFU + MRU hits):             79.9 %      10.4G
        Data demand efficiency:                        86.4 %       3.9G
        Data prefetch efficiency:                       7.5 %       2.1G

Cache hits by cache type:
        Most frequently used (MFU):                    73.9 %       7.7G
        Most recently used (MRU):                      25.2 %       2.6G
        Most frequently used (MFU) ghost:               0.4 %      45.1M
        Most recently used (MRU) ghost:                 1.1 %     119.8M

Cache hits by data type:
        Demand data:                                   31.8 %       3.3G
        Demand prefetch data:                           1.5 %     160.3M
        Demand metadata:                               66.7 %       7.0G
        Demand prefetch metadata:                     < 0.1 %       3.4M

Cache misses by data type:
        Demand data:                                   21.0 %     525.5M
        Demand prefetch data:                          78.6 %       2.0G
        Demand metadata:                                0.2 %       5.0M
        Demand prefetch metadata:                       0.2 %       3.8M

DMU prefetch efficiency:                                            1.8G
        Hit ratio:                                     20.3 %     364.2M
        Miss ratio:                                    79.7 %       1.4G

L2ARC not detected, skipping section
...

Thanks for your hints!

 

[Stefan_R]

Your ZFS is configured to use as much as half of your RAM (as indicated by the "Max size" line in the output you posted). ZFS will however always yield ARC memory to other processes if needed, so the ARC should never lead to out-of-memory situations.

In general, the behaviour you're seeing is expected and fine.

If you want, you can restrict the caching behaviour manually as well, but consider that this might reduce IO performance.

 

[mrg]

Just to clarify for myself.
I am currently running at 92% memory, with a handful of vms and ct's.

RAM usage 92.76% (28.79 GiB of 31.03 GiB

If I restarted this box today, the memory would reset at maybe 20% at best. That being the case if I understand what your saying is that regardless if its at 92% I could add as many vm's cts as my 32gb system can take? I currently have a mix of ct/vms. only 7 of them.

 

[Stefan_R]

Check your 'arc_summary'. If the space is actually used by ZFS caches, then yes, it should be available to VMs/CTs. Keep in mind however, that ZFS caches the data in RAM for a reason - if possible, you should probably just add more RAM instead of cutting down on caching, otherwise ZFS (or the system in general) might leave some performance behind.

 

[mrg]

Hello Stefan_R thank you for the reply, These are the results, I am new to ZFS and proxmox so any help would be gratefully appreciated.
I had to remove alot of it due to the word count restrictions. if the parts missing you need please let me know.

arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report Mon Dec 21 13:44:17 2020
Linux 5.4.73-1-pve 0.8.5-pve1
Machine: server-2 (x86_64) 0.8.5-pve1

ARC status: HEALTHY
Memory throttle count: 0

ARC size (current): 98.9 % 15.3 GiB
Target size (adaptive): 99.7 % 15.5 GiB
Min size (hard limit): 6.2 % 993.1 MiB
Max size (high water): 16:1 15.5 GiB
Most Frequently Used (MFU) cache size: 92.3 % 13.5 GiB
Most Recently Used (MRU) cache size: 7.7 % 1.1 GiB
Metadata cache size (hard limit): 75.0 % 11.6 GiB
Metadata cache size (current): 8.6 % 1023.0 MiB
Dnode cache size (hard limit): 10.0 % 1.2 GiB
Dnode cache size (current): 11.5 % 136.5 MiB

ARC hash breakdown:
Elements max: 2.3M
Elements current: 75.4 % 1.7M
Collisions: 23.4M
Chain max: 8
Chains: 276.3k

ARC misc:
Deleted: 18.4M
Mutex misses: 1.8k
Eviction skips: 190.1k

ARC total accesses (hits + misses): 832.7M
Cache hit ratio: 95.7 % 796.8M
Cache miss ratio: 4.3 % 35.9M
Actual hit ratio (MFU + MRU hits): 95.5 % 795.6M
Data demand efficiency: 98.4 % 443.1M
Data prefetch efficiency: 15.3 % 33.6M

Cache hits by cache type:
Most frequently used (MFU): 77.3 % 615.6M
Most recently used (MRU): 22.6 % 180.0M
Most frequently used (MFU) ghost: 1.9 % 15.0M
Most recently used (MRU) ghost: 0.3 % 2.6M

Cache hits by data type:
Demand data: 54.7 % 436.0M
Demand prefetch data: 0.6 % 5.2M
Demand metadata: 44.5 % 354.7M
Demand prefetch metadata: 0.1 % 949.2k

Cache misses by data type:
Demand data: 19.7 % 7.1M
Demand prefetch data: 79.3 % 28.5M
Demand metadata: 0.7 % 245.6k
Demand prefetch metadata: 0.3 % 118.1k

DMU prefetch efficiency: 204.2M
Hit ratio: 7.4 % 15.1M
Miss ratio: 92.6 % 189.2M

 

[Stefan_R]

ARC size (current): 98.9 % 15.3 GiB

15.3 GiB of your RAM are currently used by ZFS. The cache being this full is usually a sign of it being in good use, so again, more RAM is always preferrable to limiting ZFS cache by assigning it somewhere else.

Also, you can use [CODE][/CODE] tags to paste longer text and also format it correctly

 

[mrg]

That's good to know thank you Stefan_R

 

[pedrorgil]

We are using Proxmox 6.3.3 with 64GB RAM. We use an external ceph cluster as storage for the VM. We have 10 VMs configured to use 4GB RAM each, and the memory usage in the host in at 90% but in the arc_summary the current ARC size is 0.0%, is this normal?

------------------------------------------------------------------------
ZFS Subsystem Report Tue Mar 16 07:46:22 2021
Linux 5.4.78-2-pve 0.8.5-pve1
Machine: pxnode5-gra (x86_64) 0.8.5-pve1

ARC status: HEALTHY
Memory throttle count: 0

ARC size (current): 0.0 % 0 Bytes
Target size (adaptive): 100.0 % 31.2 GiB
Min size (hard limit): 6.2 % 2.0 GiB
Max size (high water): 16:1 31.2 GiB
Most Frequently Used (MFU) cache size: n/a 0 Bytes
Most Recently Used (MRU) cache size: n/a 0 Bytes
Metadata cache size (hard limit): 75.0 % 23.4 GiB
Metadata cache size (current): 0.0 % 0 Bytes
Dnode cache size (hard limit): 10.0 % 2.3 GiB
Dnode cache size (current): 0.0 % 0 Bytes

 

[Stefan_R]

We are using Proxmox 6.3.3 with 64GB RAM. We use an external ceph cluster as storage for the VM. We have 10 VMs configured to use 4GB RAM each, and the memory usage in the host in at 90% but in the arc_summary the current ARC size is 0.0%, is this normal?

Please don't reply to such old posts, you can always open a new topic for such questions

To answer your question: ARC is a ZFS feature, so if you use Ceph you'd see 0% there because it's not in use. Ceph also has caching though, so that's probably the memory usage you're seeing (if you don't use krbd it's a bit hard to tell, because the cached memory will be attributed to the VMs memory usage directly, as Ceph then runs in user-space).

 

[willybong]

quindi l'ARC non dovrebbe mai portare a situazioni di esaurimento della memoria.

Hey there! Are you sure about that?
For two of my customers, I'm having trouble with ARC cache. In particular, some Windows VM has been stopped because hadn't enough memory.
Infact, using "arc_summary" I note that ZFS spends about 100% of the memory (I have 128GB of memory for the host)
I have some cluster with Proxmox 5 as well, but this problem doesn't appear. The ARC cache is always pretty low.

Thanks in advance

 

[Stefan_R]

Hey there! Are you sure about that?
For two of my customers, I'm having trouble with ARC cache. In particular, some Windows VM has been stopped because hadn't enough memory.
Infact, using "arc_summary" I note that ZFS spends about 100% of the memory (I have 128GB of memory for the host)
I have some cluster with Proxmox 5 as well, but this problem doesn't appear. The ARC cache is always pretty low.

Thanks in advance

It's quite poetic to reply to a thread that ended with
> Please don't reply to such old posts, you can always open a new topic for such questions
after 4 months with another question

And yes, I am sure about it, this is how the ARC cache is designed to function. Unless you have modified the config values, it will also never take up more than 50% of the total memory available. If it were to take 100%, the system would crash long before...