Mem usage on standalone empty node

[Anrew]

Hello,
can someone help me to find how pve using 7GB RAM w/o running any vm and almost all services stopped.

Here is some stats:

arcstat
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
20:34:16     0       0     0       0     0      0    0   5.6K   3.9G   114G

free -m
               total        used        free      shared  buff/cache   available
Mem:          128419        7455      121656           3         280      120963
Swap:           2047           0        2047

top
top - 20:31:11 up  6:21,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 349 total,   1 running, 348 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 128419.8 total, 121661.7 free,   7450.5 used,    280.1 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used. 120969.3 avail Mem

cat /proc/meminfo
MemTotal:       131501824 kB
MemFree:        124575920 kB
MemAvailable:   123867088 kB
Buffers:           12384 kB
Cached:           251904 kB
SwapCached:            0 kB
Active:           120988 kB
Inactive:         187036 kB
Active(anon):      46972 kB
Inactive(anon):        0 kB
Active(file):      74016 kB
Inactive(file):   187036 kB
Unevictable:        1876 kB
Mlocked:               0 kB
SwapTotal:       2097144 kB
SwapFree:        2097144 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         45636 kB
Mapped:            54384 kB
Shmem:              3236 kB
KReclaimable:      23004 kB
Slab:             165680 kB
SReclaimable:      23004 kB
SUnreclaim:       142676 kB
KernelStack:        6128 kB
PageTables:         2716 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    67848056 kB
Committed_AS:     200464 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      208860 kB
VmallocChunk:          0 kB
Percpu:            13728 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      262944 kB
DirectMap2M:     5625856 kB
DirectMap1G:    128974848 kB

ps -e -o pid,vsz,comm= | sort -n -k 2
      2      0 kthreadd
      3      0 pool_workqueue_release
      4      0 kworker/R-rcu_g
......................................

  13077      0 kworker/u51:0-cqp_cmpl_wq
  13457      0 kworker/u51:2-cqp_cmpl_wq
   3522   2644 sftp-server
   1361   2892 agetty
   1367   2936 agetty
  12486   4716 bash
   1384   5192 proxmox-firewal
    944   5308 qmeventd
   3525   5432 bash
    766   7516 mdadm
    914   7880 rpcbind
    934   7920 dbus-daemon
  14559   8664 ps
   1026   8828 lxc-monitord
   1031  10676 chronyd
    953  11340 smartd
   1099  15432 sshd
   3505  17672 sshd
   3419  17980 sshd
  12484  18116 mc
   1028  18860 chronyd
   3490  19272 systemd
  14560  19492 sort
    968  25048 systemd-logind
    608  27440 systemd-udevd
   1057  28700 unattended-upgr
    568  41204 systemd-journal
   1340  42660 master
  11897  42684 pickup
   1344  42728 qmgr
   6188  48436 tlsmgr
   1456  79184 pvefw-logger
    988 101412 zed
      1 103064 systemd
   3491 103672 (sd-pam)
    937 278156 pve-lxc-syscall
   1378 430836 pmxcfs

As you see all process summ like 1GB, cache is almost not exist, "/proc/sys/vm/drop_caches" do not help, I do not use zfs, only ext4 and mdraid.
I can't understand where 6GB RAM go, what's using it.

Please help to figure it out, as for me it's not normal for newly installed server to use 7.5 GB RAM out of box.

 

[leesteken]

I think you are using ZFS? Then it's probably ARC, which is shown as used instead of buff/cache due to not being part of the Linux kernel due to the open source license.
How much memory did you give to the ZFS ARC? What is shown in the ARC size (current) section of the output of arc_summary?

EDIT: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_limit_memory_usage

EDIT2: 5.6 KiB of ARC does not account for the memory usage. (But it is a common surprise for new users.)

 

[Anrew]

No, I do not use zfs.

arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Nov 13 21:02:55 2024
Linux 6.8.12-4-pve                                            2.2.6-pve1
Machine: ns3242648 (x86_64)                                   2.2.6-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                   < 0.1 %    5.6 KiB

Here is fs types in use:

df -Th
Filesystem              Type      Size  Used Avail Use% Mounted on
udev                    devtmpfs   63G     0   63G   0% /dev
tmpfs                   tmpfs      13G  1.5M   13G   1% /run
/dev/md3                ext4       20G  3.7G   15G  20% /
tmpfs                   tmpfs      63G   34M   63G   1% /dev/shm
tmpfs                   tmpfs     5.0M     0  5.0M   0% /run/lock
efivarfs                efivarfs  128K   29K   95K  23% /sys/firmware/efi/efivars
/dev/md2                ext4      988M  191M  731M  21% /boot
/dev/nvme3n1p1          vfat      511M  5.9M  505M   2% /boot/efi
/dev/fuse               fuse      128M   16K  128M   1% /etc/pve
tmpfs                   tmpfs      13G     0   13G   0% /run/user/0
/dev/mapper/vg-data     ext4      857G   40G  774G   5% /var/lib/vz
/dev/mapper/vgdata-data ext4      7.0T  1.3T  5.4T  20% /var/lib/vzdata

That's why I'm asking, I'm already read a lot of thread trying to find out what's using that mem, tried everything but no luck=(

 

[Anrew]

EDIT: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_limit_memory_usage

EDIT2: 5.6 KiB of ARC does not account for the memory usage. (But it is a common surprise for new users.)

Yeah, actually I did not believe this value just in case and changed max and min arc memory, reboot aaaand nothing changed.

Super weird situation for me, use such much of memory for almost empty linux.