Extreme high memory utilization.

[Urbanovits]

HI,
Latest build running on a solo HP server (not a cluster) and I found extreme high memory utilization. When no VMs running 191+GB used of 384G.

Seems to be that usr/bin/pmxcfs causing this problem, Stopped and disabled pve-ha-lrm.service and pve-ha-crm.service
Not helped.

Running htop did not find any other process doing this.

After reboot everything goes back to normal for a while, then symptom same..



Shared memory



Physical memory

 

[waltar]

What does "free" say ?

 

[Urbanovits]

What does "free" say ?

Side by side

 

[cheiss]

Hi,

do you use ZFS? Is this directly after boot or after running some VMs and then stopping them?

Seems to be that usr/bin/pmxcfs causing this problem,

How did you come to this conclusion? As per your screenshots, pmxcfs uses nearly no memory, as it should.

 

[Impact]

Do you use ZFS? Check arcstat and arc_summary -s arc. There's rarely a reason to use free without -h. Just makes things harder.

 

[news]

run arcstat in a root console, please.

 

[Urbanovits]

Answer to all of you

Probably I went to wrong way with pmxcfs, That is why I'm ask community for help.
Yes ZFS in use, one of my drive managed by RAID (RAID5) and ZFS employed, because of snapshoot requirement.
*I know this ZFS config is not supported, but no other way to have snapshoot option.



ZFS Subsystem Report Wed Nov 13 10:15:23 2024
Linux 6.8.12-3-pve 2.2.6-pve1
Machine: nkilab (x86_64) 2.2.6-pve1

ARC status: HEALTHY
Memory throttle count: 0

ARC size (current): 63.9 % 120.8 GiB
Target size (adaptive): 66.1 % 124.8 GiB
Min size (hard limit): 6.2 % 11.8 GiB
Max size (high water): 16:1 188.9 GiB
Anonymous data size: 0.0 % 0 Bytes
Anonymous metadata size: 0.0 % 0 Bytes
MFU data target: 37.5 % 44.5 GiB
MFU data size: 14.9 % 17.7 GiB
MFU ghost data size: 0 Bytes
MFU metadata target: 12.5 % 14.8 GiB
MFU metadata size: 0.5 % 652.7 MiB
MFU ghost metadata size: 0 Bytes
MRU data target: 37.5 % 44.5 GiB
MRU data size: 83.9 % 99.7 GiB
MRU ghost data size: 0 Bytes
MRU metadata target: 12.5 % 14.8 GiB
MRU metadata size: 0.6 % 691.1 MiB
MRU ghost metadata size: 0 Bytes
Uncached data size: 0.0 % 0 Bytes
Uncached metadata size: 0.0 % 0 Bytes
Bonus size: < 0.1 % 169.4 KiB
Dnode cache target: 10.0 % 18.9 GiB
Dnode cache size: < 0.1 % 1.5 MiB
Dbuf size: < 0.1 % 427.2 KiB
Header size: 1.7 % 2.1 GiB
L2 header size: 0.0 % 0 Bytes
ABD chunk waste size: < 0.1 % 10.0 KiB

ARC hash breakdown:
Elements max: 9.3M
Elements current: 100.0 % 9.3M
Collisions: 1.4M
Chain max: 5
Chains: 590.5k

ARC misc:
Deleted: 16
Mutex misses: 0
Eviction skips: 2
Eviction skips due to L2 writes: 0
L2 cached evictions: 0 Bytes
L2 eligible evictions: 255.5 KiB
L2 eligible MFU evictions: 0.0 % 0 Bytes
L2 eligible MRU evictions: 100.0 % 255.5 KiB
L2 ineligible evictions: 4.0 KiB[/CODE]

 

[cheiss]

ARC size (current): 63.9 % 120.8 GiB
Target size (adaptive): 66.1 % 124.8 GiB
Min size (hard limit): 6.2 % 11.8 GiB
Max size (high water): 16:1 188.9 GiB

Well, ZFS is using ~120 GiB of RAM just for the ARC, plus some more for other things.
That's were the usage comes from, since this is counted as used RAM. There are a lot of threads about this exact topic in this forum.

You can limit the ZFS ARC size if you want, see Limit ZFS ARC memory usage in our admin guide.

 

[Urbanovits]

Thank you so much!
Meanwhile I found ZFS eating up half of host memory, as usual. That is the root cause.

Last question (not found in guide): What would be a proper number ( "X"*1024*1024*1024) or calculation method to limit min/max memory usage of ZFS? Is it per VM or which way?
options zfs zfs_arc_min=X?
options zfs zfs_arc_max=X?

Or is it try and fail?